Aligning Estimands, Identification, and Conclusions

Presented at the Causal Data Science Meeting (with Graham Goff).

Regression adjustment underpins observational social science, but control variable selection can change interpretable estimands. In this paper, we introduce a workflow that combines Directed Acyclic Graphs (DAGs) with automated empirical estimation to recover target estimands. To illustrate its utility, we examine all 2024 American Political Science Review (APSR) articles using observational regressions with controls. Incorporating author feedback on our DAGs, we find no colliders. However, 61% of articles include estimand-shifting mediators or their descendants, and 21% of articles interpret them as total effects without qualification. Removing DAG-inconsistent variables also decreases standard errors, increases coefficient magnitudes, and reduces statistical significance. Reweighting to recover the estimands not only yields similar patterns for coefficient magnitudes and statistical significance but also positivity failures in several studies. Especially given APSR’s flagship status, our results suggest that drawing assumptions before conclusions and using our R package, DAGassist, can improve empirical practice.