Public journal-independent evaluation of impactful research
The Unjournal is making research better by evaluating what really matters. We aim to make rigorous research more impactful and impactful research more rigorous.
The academic journal system is out-of-date, discourages innovation, and encourages rent-seeking.
The Unjournal is not a journal. We don’t publish research. Instead, we commission (and pay for) open, rigorous expert evaluation of publicly-hosted research. We make it easier for researchers to get feedback and credible ratings of their work, so they can focus on doing better research rather than journal-shopping.
We currently focus on quantitative work that informs global priorities, especially in economics, policy, and social science. We focus on what's practically important to researchers, policy-makers, and the world.
The Unjournal is a registered nonprofit; we don’t charge fees. Visit our knowledge base to learn more.
Research papers and projects
Some of our latest evaluations
See all of our evaluation packages at Unjournal.pubpub.org
Selecting the Most Effective Nudge: Evidence from a Large-Scale Experiment on Immunization
How can a government or NGO find the ‘best way’ to get people to do something like vaccinate or send their kids to school? There’s a lot of ‘new policy tools’ such as nudges, social media/ network interventions, conditional direct payments, and many ways to do and combine these. … A huge ‘policy choice space’. Often the only way to learn about this is a ‘one-shot’ trial simultaneously testing different bundles of policies. This paper proposes and demonstrates a new approach: “treatment variant aggregation (TVA) — to select a policy from a large factorial design, comparing all levels and combinations. This “pools... policy variants that are not meaningfully different and prunes those deemed ineffective”, taking the winners’ curse into account. They apply this to an RCT on interventions (75 combos including reminders, incentives, local ambassadors) to promote immunization in Haryana, India. They report that the most cost-effective combination increases immunizations per dollar by 9.1% relative to status quo.
Why this paper? We’ve heard that this research is “already informing a J-PAL project in India, as well as Suvita.” Considering ‘incentives vs reminders’ is alsorelevant to charities like New Incentives. The TVA approach can inform a range of high-impact contexts, from health, to fighting misinformation, to promoting effective charitable giving.
The evaluations from field and methodology experts were extremely positive overall. However, evaluator 2 expressed some doubts about the novelty of the authors’ (“treatment variation aggregation”) approach and its practical advantages relative to those Bayesian estimators that are more sophisticated than the ones they tested. Evaluator 3 had concerns about the robustness of the assumptions behind the econometric justifications for TVA.
These evaluators strongly encourage the authors to provide further software sharing and guidance. Approaches like TVA (or Bayesian equivalents) may be increasingly used in research and practice. This will happen faster and better if researchers are encouraged to provide these methodological public goods.
Water Treatment and Child Mortality: A Meta-analysis and Cost-effectiveness Analysis
Dirty water kills kids. So treating water to kill germs should help save lives. But how effective is it? It would be nice to know because lots of other things save lives too and resources are scarce.
We have evidence on this, but (thankfully!) kids dont die often so statistical power to detect death in these studies is low. What people can do is test for a signal on the causal path from dirty water → death, like diarrhea,& convert that using an estimate of deaths per diarrhea. But … indirect approaches are built on assumptions that might be wrong.
So a few years ago a rockstar team of social scientists conducted a meta-analysis pooling lots of studies of water treatment on death to gain more statistical power. Given the (pretty bad!) state of reporting results in economics, meta-analysis is so much work. Big kudos. They found that water treatment worked a good deal better than indirect methods suggested. They also did a cost-effectiveness analysis that suggested it's competitive w AMF.
This interested me! (@ryancbriggs.net) and GiveWell [blog]! But I was also skeptical. Thankfully I'm an evaluation manager at The Unjournal, where we do open evaluations of impactful research, so we evaluated it.
I hired … 3 evaluators working in 2 teams to evaluate the paper. … Experts in their fields, drawn from both public health & economics. I'm so grateful for their participation. They found errors & raised … important issues, like missing studies or apparent deviations from analysis plans. They also had suggestions for making the CE more useful.
The authors wrote a careful response & updated the draft. How often do you read this in a response letter?
“Evaluators were right to point out that including this study contradicts what was proposed in the AEA search strategy. We are looking into this further and may exclude this study”
… My read is that the major takeaways of the article are unchanged, but I'm more confident in the results now. This is how science should work. Everyone involved wanted the best answer to this question and it's all open.
(from Ryan Briggs’ BlueSky/X thread)
See all our evaluation packages, including evaluator reports, ratings and predictions, author responses, and manager syntheses at unjournal.pubpub.org.
For the latest public summaries of these packages, see this interface.
Willful Ignorance and Moral Behavior
Consumers' willful ignorance about the consequences of their actions may impede moral behavior. We test this concern in a real-world context based on a laboratory experiment and field data. We organized two evaluations of the paper: “Willful Ignorance and Moral Behavior”. To read these evaluations, please see the link above.
A rich economic literature has examined the human capital impacts of disease-eliminating health interventions, such as the rollout of new vaccines. This literature is based on reduced-form approaches which exploit proxies for disease burden, such as mortality, instead of actual infection counts, which are difficult to measure.
We organized two evaluations of the paper: "Existential Risk and Growth" . The evaluators see the paper as a substantively important model informing a critical question, but have concerns about the degree to which the assumptions and model structure are sufficient for actually answering key questions.
How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World?
For this paper, the evaluators were moderately positive, but offered critique of the non-systematic nature of this review and its departure from “best practices”. To read more, see the link above.
Does Conservation Work in General Equilibrium?
We organized one evaluation of the paper: “Does Conservation Work in General Equilibrium” under our “interim evaluation” policy. To read this evaluation, please see the link above.