The file drawer

Ideally, social science should be an institutionalised reflective equilibrium between theory and empirical work—through which arguments guide empirical inquiry and findings prompt revisions to our assumptions, explanations, and questions (Ashworth et al., 2021)—with scholars contributing according to their comparative advantages. Realising this ideal requires clear theorising and, as the credibility revolution has taught us, attention to the identification challenges associated with testing the empirical implications of theories. It also requires care in thinking about the scope conditions and external validity of findings.1

Contributing to this collective endeavour means individual researchers have to invest time and effort under considerable uncertainty about which ideas will prove fruitful. Indeed, research is often a non-linear process of going down dead ends, trying other avenues, failing, and trying again until one makes progress in answering the research question of interest. This process requires a willingness to be usefully wrong (though being right never hurts, of course)—an aspiration reflected in the title of my Substack, Often wrong, but sometimes useful. Even projects that fall short of their original aims can advance understanding by clarifying the limits of an argument or revealing why a design cannot answer the intended research question.2

These ideals are (all too) often in tension with the incentives of academic publishing and career progression, which can encourage researchers to retrofit questions or theoretical intuitions to familiar methods, explore the ‘garden of forking paths’ somewhat too assiduously, and consign ‘unsuccessful’ projects to the file drawer (Gelman and Loken, 2013; Smaldino and McElreath, 2016; McElreath, 2020). Academic websites are like the tip of the research iceberg, showing polished, successful projects while the far larger submerged part of the research process remains unreported. Unclear theoretical ideas, plausible but ultimately uninformative empirical designs,3 and inconclusive or contradictory findings consequently receive less attention.

This selective reporting creates a form of survivorship bias,4 which has at least three negative effects. First, students see only polished projects that “work out” while regularly “failing” with their own projects. This can be demoralising, with frustration—which is normal in research and often also productive—being mistaken for a lack of ability. It can also discourage them from exploring new avenues, leading to excessive risk aversion in the form of hewing closely to what has already been done.

Second, selective reporting can distort the allocation of research effort. Without transparency about failures, researchers may well repeat avoidable mistakes or pursue research designs that others have already found unfruitful. Third, it inhibits the accumulation of knowledge. When credible findings that provide little support for an argument remain unpublished, researchers may overestimate both the strength of its empirical support and the range of circumstances in which it applies (see footnote 1).

While there is a severe limit to what any single individual can do, this part of my website is meant to list ‘failed’ theoretical ideas and empirical projects that can potentially inform subsequent work. Sharing these attempts is one way I try to resist the temptation to let unsuccessful work disappear in my file drawer, as it were. Doing so at scale, however, requires institutional changes rather than merely individual resolve. Aletheia, which makes both submitted research and reviewers’ feedback publicly available, illustrates what better institutions might look like and how difficult institutional change is.

Critical scrutiny of the incentive structures that social scientists are subject to and suggestions for reform are much needed, particularly in the age of generative artificial intelligence (Munger, 2026). But a single-minded focus on what is wrong risks giving rise to the impression that the entire system is broken. My own experience gives me reasons for a more hopeful view. In many ways, this website is an homage to the generous senior academics who have mentored me—from my supervisors to my co-authors and beyond—and to the ideals of intellectual curiosity, rigour, and generosity that they embody. Finally, the website also serves as a commitment device through which I hope to hold myself accountable to these ideals in my own research, knowing that I will sometimes fall short.

Green transition, brave new feminine world?

2025–26 · Survey experiment

With Henri Gruhl (RWI – Leibniz Institute for Economic Research and Vrije Universiteit Amsterdam), Johannes Brehm (RWI – Leibniz Institute for Economic Research and Hertie School), and Lara Hankeln (University of Oxford)

Motivation

Women consistently express greater support for climate policy than men, yet the mechanisms underlying this difference remain contested. We examine how people who are not personally exposed to job loss respond when climate policy displaces workers from traditionally masculine occupations. Even when relatively few workers lose their jobs, the broader public’s judgements about their losses and the compensation they deserve may influence support for the policy.

The theoretical intuition

We treat occupational status as the degree of prestige an occupation commands, independent of income. We focus on its masculinity dimension—the extent to which an occupation’s prestige derives from its association with masculinity. These associations involve traits such as physical risk, toughness, or manual skill. By focusing on a single dimension, we avoid the interpretative ambiguity of composite measures, in which it can be unclear which component produces an observed political response. Occupations are also central sites of preference formation. Two jobs can have similar overall prestige, wages, and employment security while differing in the masculinity component of occupational status.

When occupational transitions preserve the masculinity component of status, disagreement among people unaffected by job loss chiefly reflects the balance between other-regarding motivations and efficiency concerns. When transitions reduce this component, respondents also disagree over whether the loss is normatively deserving of compensation. We expect men without tertiary education, who tend to have stronger affinities with masculine occupations, to be more supportive of compensation than women without tertiary education. Among the tertiary-educated, we expect lower support for compensation and a narrower gender gap.

The “status-substitution problem” creates a further obstacle. While material harms can be offset with sufficient resources, money is less effective at compensating losses in the masculinity component of occupational status. Compensation may therefore fail to sustain support for climate policy among those who regard these losses as normatively significant, while others may oppose transfers that they construe as sustaining patriarchal structures.

The research design

The pre-analysis plan specifies a survey experiment in Germany’s Social-Ecological Panel, with respondents randomly assigned in equal proportions to two vignettes. Both describe a 45-year-old male coal worker who loses his job through the coal phase-out, receives temporary financial compensation equal to his previous income, and undergoes retraining and placement in a new occupation. In the technical and manual condition, intended to preserve the masculinity component of occupational status, the new occupation involves physically demanding technical and manual tasks and regular use of tools and machinery. In the social condition, intended to reduce this component, the new occupation involves emotionally demanding tasks and direct interaction with people. Both descriptions specify the same income and social standing as before, a secure position in the same region, and a working environment, hours, and team composition similar to the previous job.

We describe broad domains of tasks because specific job titles, such as construction worker or nurse, could introduce stereotypes about migration, wages, prestige, and working conditions. We also avoid explicit references to masculinity or gender norms to reduce concerns about social desirability and respondents inferring the intended answer. Immediately after the vignette, respondents estimate the new job’s income and describe its occupation in their own words. The vignette remains visible during these checks, then disappears before the outcome questions. These measure support for temporary financial compensation, retraining and placement, and climate policy. They also assess the perceived fairness of compensation. Later questions test recall of the new occupation and whether respondents inferred the study’s purpose. The analysis examines differences by respondents’ gender and education.

The result

Few of the preregistered predictions were borne out. Respondents initially noticed the difference between the two occupations, but their later answers often failed to retain it. When missing answers are counted as failures, 74% of respondents later recalled an occupation consistent with the technical and manual vignette, compared with 21% for the social vignette. Some respondents who initially recognised the social occupation later described the new occupation as technical or manual work. Respondents given the social vignette also expected lower earnings, despite the statement that income was unchanged. These problems leave the proposed mechanism inadequately tested.

Lessons for subsequent research

The central lesson is that noticing a vignette’s wording does not establish that respondents believe its premises or retain the intended distinction. I suspect that the transition into social and interpersonal work, with unchanged income, social standing, and working conditions, lacked credibility; the evidence cannot distinguish among all possible explanations. I would therefore consider other experimental approaches, including images, while testing whether respondents accept the stipulated conditions and retain the difference between the occupations. The theory was also too complex for the design, which had low statistical power for its predictions. A subsequent study should examine a simpler argument with sufficient power for comparisons by gender and education.

Materials

OSF project · Thread on the pre-analysis plan · Thread on the results

Is corporatism clean or dirty?

2024–26 · Cross-national panel analysis

Motivation

The effect of corporatism on climate policy—which has received renewed attention as the literature on climate politics has taken an institutional turn—is theoretically disputed and empirically ambiguous. Finnegan (2022) argues that corporatism facilitates stringent climate policy. Because tripartite concertation gives organised business and labour guaranteed access to policymaking (Munk Christiansen, 2020), repeated bargaining allows governments to commit credibly to compensating the losers from climate policy. Mildenberger (2020) argues that corporatism hinders ambitious climate policy because it institutionalises the ‘double representation’ of carbon-intensive interests—through business in parties of the right and unions in parties of the left—which can then block or dilute climate policy. The empirical evidence is mixed and mostly old, with most samples ending in the late 1990s or early 2000s. Against this backdrop, the paper revisits the relationship between corporatism and climate policy theoretically and empirically. I wrote it in 2024 as an essay for a course on the comparative political economy of advanced democracies.

The theoretical intuition

I argue that the two accounts can be integrated once the aggregation of interests in corporatist systems is theorised more carefully. Unlike Mildenberger, I do not assume that carbon-intensive interests always block more stringent policies. They do so only if the value of a veto exceeds the value of the concessions they can extract from the government. Unlike Finnegan, I do not assume that governments compensate the losers from climate policy merely because they can do so credibly. Because compensation is costly in time and fiscal resources, governments pay it only when the pressure to act on climate change is high enough. The main effect of corporatism on the stringency of climate policy is therefore ambiguous. It is positive if credible compensation dominates and negative if double representation dominates. This balance may also explain why the empirical evidence is inconclusive.

Initially, I argued that three moderators determine which effect dominates. First, the bargaining power of carbon-intensive interests grows with their economic weight. The effect of corporatism on stringency should therefore decline as the economic importance of carbon-intensive industry increases. When climate policies are adopted nonetheless, for instance because of international or European obligations, their costs should fall more heavily on consumers. Second, in open economies, concerns about competitiveness should lead capital and labour in carbon-intensive industries to use corporatist structures to shift the costs of climate policy to consumers (Katzenstein, 1985). Third, electoral competition from green parties should make governments less willing to impose costs on consumers and more willing to pay for the consent of carbon-intensive interests. Where green parties compete intensely, corporatism should consequently yield more stringent climate policy and a distribution of costs more favourable to consumers.

The research design

To test these predictions, the analysis extends the work of Finnegan (2022) with new data. The independent variable is corporatism. Its proxies are the time-varying index of Jahn (2016) and a dummy for tripartite concertation from the ICTWSS database on the institutions of labour markets. Tripartite concertation refers to the routine involvement of unions and employers in the decisions of governments on social and economic policy. The dependent variables are the stringency of climate policy, the adoption of climate policies, and the distribution of the costs of climate policy between consumers and producers. Two-way fixed-effects regressions use the data of Finnegan (2022). Three-way fixed-effects specifications, with fixed effects for countries, half-decades, and policy instruments, use the OECD’s Climate Actions and Policies Measurement Framework (CAPMF). The CAPMF measures the adoption and stringency of climate policies by instrument. The specifications interact corporatism with the share of industry in GDP, the openness of the economy, and the probability that green parties are included in coalitions.

The result

Most estimates pointed in the predicted directions. As the share of industry in GDP increased, the association between corporatism and the adoption of climate policies weakened. In the data of Finnegan (2022), greater openness was associated with a shift of costs from producers to consumers. As the probability that green parties are included in coalitions increased, corporatism was associated with more stringent climate policy. The estimates for the main effect of corporatism on stringency were inconsistent across specifications, as expected when two mechanisms offset each other. At the time, I concluded that corporatism facilitates more stringent climate policy where carbon-intensive industry accounts for a small share of GDP and where governments face intense electoral competition from green parties.

The limitations of the theory and the evidence

Two limitations led me to ditch the project. The first concerns the theory. Corporatism plausibly affects the strategies of governments, employers, and unions, but in my argument its effect on climate policy depends on interactions with the share of carbon-intensive industry, the openness of the economy, and electoral competition from green parties. These interactions admit many combinations of conditions. Because the two mechanisms can offset each other, almost any estimate is compatible with the argument. Moreover, the source of the pressure to act on climate change remained vague, though both the willingness of governments to pay compensation and the value of a veto to carbon-intensive insiders depend on the latter. Finally, corporatism was rarely designed for climate policy. Its use in climate policy is a matter of repurposing—a process that neither Finnegan nor Mildenberger theorises and that my argument took for granted.

In 2026, I set out in a thread how I would revise the theory. Electoral activation—the extent to which voters visibly reward climate action and punish inaction—links the two accounts. Activation gives governments an electoral reason to buy the consent of carbon-intensive insiders. It also lowers the value of their veto because insiders gain more from a side payment than from blocking a reform that will come anyway. If the gain of governments from effective policy is smaller than the loss of insiders, diluting the policy is cheaper than any compensation, as in Mildenberger’s account. If the gain is larger, governments prefer paying insiders to diluting the policy. Credible promises of deferred compensation then combine real costs for insiders with compensation, as in Finnegan’s account, while governments buy any remaining consent with exemptions. Credibility matters only for deferred compensation because up-front payments and statutory exemptions are self-enforcing in any political system.

Corporatism then operates through two channels. It raises the price of consent—the concessions that governments must offer—because insiders can block policies more effectively. It also changes the currency of consent—the form in which governments pay—because deferred transfers become credible and governments need fewer exemptions and less free allocation of permits. Because activation raises what governments will pay for consent but lowers what consent is worth to insiders, the effect of corporatism on average stringency is non-monotonic in activation. It is negative at low levels of activation, positive at intermediate levels, and null once obstruction is futile. At high levels of activation, governments buy consent with transfers; at low levels, they buy it with exemptions. In pooled regressions across countries and years, the positive and negative effects of corporatism can offset each other. A null result for the main effect is then unsurprising and uninformative.

Who has a seat at the bargaining table matters as well. Taking my cue from Thelen (2014), I treat the net stake of corporatist insiders in decarbonisation, aggregated at the level at which bargaining takes place, as the relevant variable. Encompassing arenas, such as Scandinavian peak organisations that include services and the public sector, internalise the costs of shielding producers. Narrow arenas pass these costs on to outsiders. German exemptions of energy-intensive firms from the renewables surcharge were financed by households and small and medium-sized firms, which had no seat at the table. In Finland, a corporatist inner circle designed feed-in tariffs that favoured large firms and were financed from the state budget (Husu and Salo, 2022). The narrower the arena relative to those who pay, the more the costs of climate policy fall on outsiders.

A second thread applies this argument to the German coal exit, which is often used as an example of the ‘good’ effects of corporatism on climate policy. The thread, however, argues that this example could also be construed as evidence of the opposite. Rising carbon prices in the EU emissions trading system would have pushed many coal plants out of the market in any case. Unless incumbents could have kept these prices low for about two decades, the negotiated exit mainly delayed closures, compensated incumbents who would otherwise have faced market pressure, and shifted the costs to voters.

While the flaws in my original theory could be fixed, the second limitation, the empirical one, was the main reason for abandoning the project. The central problem was low statistical power. Corporatism changes slowly and is measured once per country and year. After controlling for fixed effects for countries and for years or half-decades, little variation in corporatism remains to identify the interactions on which the theory depends. Comparisons between countries cannot compensate for this lack of variation. Relatively few rich democracies have corporatist institutions. These institutions developed mainly in small open economies (Katzenstein, 1985). Their effects on climate policy are consequently hard to disentangle from the effects of the conditions that produced them.

The measures of corporatism exacerbate the problem of low statistical power. They raise three distinct concerns. The first is conceptual validity, the extent to which a measure captures the concept that the theory specifies. Jahn’s index reflects changes in wage bargaining, which may be unrelated to concertation over climate policy. The dummy for tripartite concertation is closer to the concept but exhibits even less temporal variation within countries. The second is the interpretability of the scale. An index that aggregates several dimensions of corporatism has no natural unit. It is unclear what a change of one unit in Jahn’s index would mean for the bargaining between governments, employers, and unions. The third is measurement error. Random error in the index of corporatism biases its estimated effects towards zero, which understates them. Country fixed effects amplify this attenuation because they net out much of the true variation in corporatism while leaving the error in place. Error in both the index and the moderators makes their product, on which the interactions rely, noisier still.

Qualitative approaches like Mildenberger’s might then be the way to go. I do not yet have a clearly formed view of their promises and pitfalls. I have concerns about both measurement and identification, of which the second seem to me more serious. On measurement, elite interviews usually rely on convenience samples, which I distrust even though I have used them in my own research. I am also sceptical about the conditions under which the answers of interviewees can be treated as sincere. Archival evidence, such as the minutes of corporatist negotiations, offers an alternative. These concerns about measurement are real. They are not specific to corporatism, however, and can be addressed. The more difficult question concerns identification. It is unclear which variation would correspond to the theoretical quantities of interest, such as electoral activation or the breadth of the bargaining arena, and which real-world qualitative comparison would approximate that variation best.

Lessons for subsequent research

My tentative lessons from this project are twofold. The first concerns the study of institutions and involves four related problems, namely endogeneity, unfalsifiability, the definition of the treatment, and the transformation of institutions over time.

The first problem is obviously endogeneity. Institutions are endogenous to the conditions under which they emerge and endure (Przeworski, 2004). If those conditions also affect the outcome of interest, the association between an institution and the outcome mixes the effect of the institution with the effect of the conditions. In the extreme case, an institution merely transmits the effects of the conditions that produced it.

Beramendi and Rueda (2014) argue that such endogeneity is highly plausible. On their account, centralised wage bargaining emerged in early twentieth-century Europe where low inequality had eased the distributive conflicts that coordination entails. The low inequality usually attributed to coordination was thus its primary determinant a few decades earlier. A cross-national association between an institution and an outcome can therefore reflect the selection of the institution by conditions closely related to the outcome.

It is pretty straightforward to see why the study of corporatism is plagued by both problems. Corporatist institutions developed mainly in small open economies, where exposure to world markets gave capital and labour reasons to coordinate (Katzenstein, 1985). Openness also affects climate policy directly. It raises concerns about competitiveness and speeds the diffusion of clean technologies and policies. Corporatist and pluralist democracies differ, in addition, in the economic importance of manufacturing, which affects the resistance of producers to climate policy. Few large, closed economies have corporatist institutions, whereas few small, open economies lack them. With so few comparable cases, an estimated effect of corporatism may reflect openness or industrial structure.

The selection problem to which Beramendi and Rueda point may arise as well. If the conditions that made concertation feasible, such as manageable distributive conflicts between capital and labour, also ease agreement on climate policy, the association between corporatism and climate policy would reflect those conditions. Country fixed effects net out such stable differences between countries. They also net out most of the variation in corporatism, a problem to which the second lesson returns.

The second problem is unfalsifiability, to which Beramendi and Rueda also point. A theory becomes unfalsifiable when it contains more explanatory factors than there are independent cases to test it. Each factor, and each interaction between factors, can absorb patterns in the data until the theory accommodates every result. The temptation to select confirming evidence grows accordingly. The theory of corporatism set out above illustrates the problem. It combines two offsetting mechanisms with three moderators. The revised theory in the thread adds electoral activation and the breadth of the bargaining arena. Yet only a minority of rich democracies have corporatist institutions. The number of explanatory factors thus approaches the number of cases that could discriminate between them.

The third problem concerns the definition of the treatment. A causal claim about an institution presupposes a well-defined treatment, in the sense that one can specify what a change in the institution would consist of. A concept that bundles several components does not meet this requirement. An index that aggregates the components can then change for reasons unrelated to the mechanism under study. Estimates based on such an index average over components with different effects. Corporatism has long been such a concept. Even in the older literature in comparative political economy, the term designates both a structure of interest representation, with centralised and monopolistic associations, and a process of policymaking, often called concertation—and the two may no longer go together (Baccaro, 2003; see also Molina and Rhodes, 2002). Measures of corporatism differ accordingly (Siaroff, 1999; Kenworthy, 2003).

The debate on corporatism and climate policy has, mostly implicitly, imported these challenges. The mechanism of credible compensation in Finnegan (2022) operates through concertation, whereas the mechanism of double representation in Mildenberger (2020) operates through the privileged access of carbon-intensive interests to policymaking. The two accounts may therefore concern different treatments, which an integrated theory has to reconcile before it can weigh one mechanism against the other. The measures in my analysis pose the same problem. Jahn’s index varies from year to year, but mainly with wage bargaining. The dummy for tripartite concertation is closer to the mechanisms but rarely changes. The variation that statistical power requires is thus the variation with the least validity.

The fourth problem is that institutions can change their function while keeping their form. Historical institutionalists call this conversion (Streeck and Thelen, 2005). A measure that tracks the form of an institution then assigns the same score to arrangements that serve different ends. Comparisons over time then conflate the two—a serious problem for designs with country fixed effects, which rely entirely on such comparisons.

Baccaro and Howell (2017) argue that industrial relations in Western Europe have followed a common neoliberal trajectory since the 1970s. Even where corporatist institutions persist, as in Sweden, they have been converted to serve liberalisation. Many in comparative political economy consequently regard corporatism as dead or, at the very least, in its death throes. It is striking that research on climate politics turned to corporatism at this time. Most climate policies were adopted from the late 2000s onwards, when corporatism had already declined. The corporatist institutions that persisted into this period may no longer have worked as the theories of the 1970s and 1980s assumed, yet a country-year score implies that they are the same treatment. The decline also impinges on the relevance of the debate. Since relatively few rich democracies retain corporatist institutions, even a convincing answer to the question of whether corporatism helps or hinders climate policy would explain climate policy in only a few countries.

The second lesson concerns statistical power, the probability that a study detects an effect of a given size if the effect exists. Research on institutions has paid close attention to identification, yet a prior question has received far less attention, namely whether the political world contains enough variation to estimate the effects of institutions at all. Power depends on the size of the effect, the number of independent units, and the variation in the treatment that remains after the design has removed confounding variation. Designs that improve identification often reduce power. Country fixed effects, for instance, discard all variation between countries and leave only changes over time. Underpowered studies miss many true effects. When they do find significant results, the estimates are likely to exaggerate the size of the effect and may even have the wrong sign. Gelman and Carlin (2014) call these Type M (magnitude) and Type S (sign) errors.

Doucette (2025) shows how severe the problem is for cross-national data. Using simulations based on the most extensive data on democracy and income, he finds that democracy would have to make countries around 16 per cent richer for a standard two-way fixed-effects design to attain 80 per cent power. With data for only 75 countries, the required effect rises to around 24 per cent. Dynamic effects and the newer difference-in-differences estimators for treatments that units adopt at different times raise the requirements further. Interaction effects are almost never adequately powered, even when the difference in the effect between groups amounts to 200 per cent. Doucette expects similar problems for other institutional causes.

In some respects, corporatism is a harder case than democracy. Studies of corporatism draw on a much smaller set of countries than studies of democracy. The measures of corporatism that vary from year to year also capture the relevant concept poorly, as the third problem above explains. Because corporatism changes slowly, successive years in the same country add little independent information. The effective number of cases is closer to the number of countries than to the number of country-years. The theory, moreover, rests on interactions whose offsetting mechanisms reduce the average effect that a design could detect.

Applying these points to my own analysis shows that a design with so little power is unlikely to detect even a moderately large main effect. The significant interactions are no more reassuring because significant estimates from underpowered designs tend to exaggerate the size of an effect and may have the wrong sign. For institutions such as corporatism, the political world may simply not contain enough variation to estimate their effects.

Materials

GitHub repository · Thread on corporatism and climate policy · Thread on the German coal phase-out

Footnotes

  1. Questions of external validity have immediate practical relevance in my work on the IPCC chapter, where we have to synthesise the literature and assess how far its findings can be generalised. This requires specifying both the quantities that individual studies estimate and the questions that the synthesis seeks to answer. The MIDA framework of Blair et al. (2023)—model, inquiry, data strategy, and answer strategy—distinguishes the question of interest, including the units to which it refers, from the procedures used to collect and analyse evidence.

    Because syntheses typically concern settings other than those studied, external validity requires that the mechanism producing an effect in the studied setting also operates in the settings of interest. Gailmard (2021) examines what could justify such a claim. A mechanism credibly identified in one case can be inferred to operate in a second case only if similarity in the observable characteristics of the two cases is taken to imply that the same mechanism operates in both. That step requires the same assumption as identification via selection on observables in a single case, namely that the relationship between observable characteristics and outcomes is known well enough to infer an outcome that has not been observed. Researchers who reject this assumption for causal inference in a single case cannot consistently accept it for generalisation across cases.

    Put simply, external validity requires a theory that specifies which characteristics of a case determine whether the mechanism operates in it. Such theory is rarely made explicit in the synthesis of evidence and in what is often called evidence-based policy advice, where the results of well-identified studies are applied to settings that resemble the original study in observable respects.

    There has been growing interest in how to think carefully about external validity, though the literature remains nascent and its insights have not yet percolated through the profession. Three strands of this literature strike me as particularly useful. The first examines how much a single study reveals about a population beyond its sample. The second examines when estimates from different studies can be combined. The third examines why treatment effects vary across studies even when their designs are harmonised.

    Little and Pepinsky (2021) provide a Bayesian framework for the first question, which relates to the trade-off between internal and external validity. Any estimate equals the true effect plus the bias of the design. Once researchers specify prior beliefs about both, Bayes’s rule determines how much an estimate reveals about the effect (see Gelman et al., 2013, for a general treatment of Bayesian inference); the more of the estimate’s variation they attribute to bias, the less they learn.

    Restricting a study to a subpopulation in which the design is credible, such as compliers or units near a discontinuity threshold, removes the bias but introduces a different unknown, namely the difference between the effect in the subpopulation and the effect in the population of interest. The estimate from the whole population is biased, whereas the estimate from the subpopulation answers a different question. Little and Pepinsky make the same point about experiments, describing “a trade-off between studying the actual phenomenon of interest without the benefit of experimental control and experimental designs that are unbiased but possibly too artificial to be useful” (p. 613).

    Which estimate is more informative about the effect in the population depends on which of the two quantities researchers are more uncertain about. The estimate from the subpopulation is more informative if the effect in the subpopulation is likely to track the effect in the population closely. The estimate from the whole population is more informative if researchers are confident about how large its bias is. The more uncertain researchers are about that bias, the less closely the effect in the subpopulation needs to track the effect in the population for the estimate from the subpopulation to be preferable. In either case, the answer depends on prior beliefs about quantities that the study itself does not estimate. Little and Pepinsky show how to update on a belief about the similarity between the subpopulation and the population “however derived” (p. 613), yet forming such beliefs requires a model of the underlying data-generating process, which yields priors about the sign and magnitude of the bias and about heterogeneity across units.

    Slough and Tyson (2023, 2024) address the second question. Studies estimate the same target if and only if they draw the same contrast, measure outcomes in the same way, and examine a mechanism that has external validity across their settings. Without such harmonisation, a meta-analysis may fail to find consistent evidence for a mechanism that operates in every setting, or find apparent consistency where the mechanism is absent. Meta-analysis presupposes these assumptions and cannot establish them.

    Two further contributions address the third question. Izzo, Dewan, and Wolton (2025) show that even harmonised studies in the same context may estimate different quantities because the average treatment effect depends on the information available to individuals, which varies with realised circumstances even when the structural characteristics of the environment remain constant. Only effects conditional on a sufficient statistic for that information are invariant across circumstances. Tosh et al. (2025) show that many variables can have large effects on the same outcome only if they are strongly dependent on one another or interact strongly; interactions make the effect of any one factor depend on the levels of the other factors in a particular study. Both results imply that a synthesis has to specify the conditions under which an effect is expected to hold (see also Findley, Denly, and Kikuta, 2026). ↩ ↩

  2. Spirling and Stewart (2025) provide an epistemological rationale for this claim. They argue that almost all empirical social science proceeds by inference to the best explanation, with researchers generating alternative explanations, collecting facts relevant to their implications, and discriminating among the explanations in light of those facts. Evidence that fails to support an explanation contributes to this endeavour because it eliminates potential explanations and prevents researchers from mistaking an absence of evidence for support. Spirling and Stewart therefore recommend reporting null and negative results even for preferred explanations. On this view, a study that fails to reject the null hypothesis has not identified a best explanation, yet it can still show that a particular explanation is inconsistent with the available facts. ↩

  3. Examples include a regression discontinuity design with insufficient statistical power near the threshold to detect effects of substantive interest (Cattaneo et al., 2019) or a survey experiment in which respondents do not perceive the intended distinction between treatments. ↩

  4. The related problem of file drawer bias arises when the likelihood of reporting and publishing findings depends on their statistical significance. Franco, Malhotra, and Simonovits (2014) find that statistically insignificant results are less likely to be published. Using a more recent sample of papers, Moniz, Druckman, and Freese (2025) confirm these findings, though the difference in the rates of publication is substantially smaller than in previous decades. Both studies point to researchers’ decisions about writing up and submitting results as an important source of selection.

    Huntington-Klein et al. (2025) find substantial variation in the results of 146 research teams answering the same question, with decisions about data wrangling accounting for a large share of this variation. Stevenson and Fischman (2026) show how favouring statistically significant results can inflate reported effects, particularly when statistical power is low. They also argue that conventional standard errors omit uncertainty arising from reasonable alternative analytical choices. These findings reinforce the value of documenting unsuccessful projects and the decisions underlying them, helping others assess the evidence and inform subsequent research. ↩