Publication Type
PhD Dissertation
Version
publishedVersion
Publication Date
5-2025
Abstract
This dissertation focuses on advancing and applying empirical program evaluation methodologies in Operations Management (OM), defining it not as mere prediction but as rigorously establishing causal effects of interventions. Chapter 2 lays the theoretical groundwork, defining empirical program evaluation within OM, detailing its alignment with causal inference, and outlining both classical econometric techniques and emerging Causal Machine Learning (CML) approaches for mitigating endogeneity. Building on this foundation, Chapter 3 presents an empirical application, evaluating the causal impact of a customer targeting strategy using live-streaming advertisements, employing Propensity Score Matching (PSM) in conjunction with linear regression models to derive actionable insights on traffic, purchase intention, and engagement. Finally, Chapter 4 studies a novel methodological contribution, the Difference-in-Differences Regression Discontinuity (DiD-RD) design, which integrates localization to enhance the credibility of causal inference in regional policy evaluation by addressing selection bias in geographically defined interventions, concluding with a discussion on the critical bandwidth selection problem from the perspective of the bias-variance trade-off.
Chapter 2 rigorously defines empirical program evaluation within Operations Management (OM) as a systematic, data-driven endeavor to assess the causal efficacy and impact of operational interventions, policies, and initiatives. Distinguishing between theoretical and empirical approaches, the paper emphasizes that while both utilize comparative statics, empirical evaluation uniquely addresses real-world data complexities to establish causality by mitigating endogeneity (e.g., omitted variable bias, reverse causality, measurement error, selection bias) within the Potential Outcomes Framework. The discussion categorizes evaluations into formative (for ongoing feedback) and summative (for overall impact), detailing how classical econometric methods—including Linear Regression, Difference-in-Differences (DiD), Regression Discontinuity (RD), and Instrumental Variables (IV)—serve summative evaluations by constructing valid counterfactuals under specific identifying assumptions and addressing issues like serial correlation. Furthermore, the chapter studies Causal Machine Learning (CML) as an emerging frontier, showcasing how it integrates machine learning's predictive power with causal inference rigor to model complex relationships and enhance robust causal effect estimation in dynamic OM environments. Ultimately, the paper underscores that rigorous empirical program evaluation is indispensable for evidence-based decision-making, enabling precise attribution of outcomes to operational changes and fostering enhanced efficiency and competitiveness.
Chapter 3 implements empirical program evaluation to assess the effect of implementing a new customer targeting strategy. The use of live-streaming advertisements on social media platforms has garnered significant popularity across various industries because of their potential to significantly enhance the marketing performance. In this paper, we adopt propensity score matching (PSM) and AR(1) auto-regressive linear models to investigate the impact of targeting the followers of key opinion leaders (KOLs) on advertising outcomes. We identify three categories of marketing outcomes: traffic, purchase intention, and engagement. Our empirical results indicate that a KOL targeting strategy reduces traffic, enhances purchase intention, but has no significant effect on engagement. To improve the overall marketing performance without incurring additional costs, we further conduct a counterfactual analysis to first identify critical parameters for specific marketing outcomes and then fine-tune non-critical parameters. Our findings offer practical guidance for advertisers seeking to enhance the effectiveness of their live-streaming advertising campaigns.
Chapter 4 studies the Difference-in-Differences Regression Discontinuity (DiD-RD) design, a novel methodology for credible causal inference in regional policy evaluation. It addresses the inherent limitations of traditional Difference-in-Differences (DiD) models, particularly the susceptibility of the parallel trends assumption to selection bias arising from systematic differences—driven by economic or other factors—between residents of treated and untreated administrative regions. The DiD-RD approach strengthens DiD by integrating the Regression Discontinuity (RD) concept of localization, restricting the analytical sample to units within a narrow geographic bandwidth h away from the administrative border c. This two-stage process directly mitigates selection bias by ensuring that within this narrow band, unobservable confounders (such as economic factors or social capital) that induce self-selection are plausibly balanced across the border, effectively creating a more comparable control group. A crucial aspect of implementing this design, which will be discussed at the end of this chapter, is the bandwidth selection problem, involving a critical trade-off between bias (favoring narrower bands for greater comparability) and variance (favoring wider bands for statistical power). The chapter formalizes the DiD-RD estimand as a more credible, albeit localized, Local Average Treatment Effect (LATE), amenable to event-study analysis for dynamic effects and pre-trend validation. While acknowledging potential limitations such as asymmetric local shocks and spillover effects, this chapter finishes by theoretically discussing how DiD-RD is useful with empirical program evaluation on a state-level healthcare policy as a case study.
Keywords
Empirical Program Evaluation
Degree Awarded
PhD in Business (Ops Mgmt)
Discipline
Operations and Supply Chain Management
Supervisor(s)
SHE, Zhaowei
First Page
1
Last Page
109
Publisher
Singapore Management University
City or Country
Singapore
Citation
TIAN, Kai.
Empirical program evaluation in operations management. (2025). 1-109.
Available at: https://ink.library.smu.edu.sg/etd_coll/794
Copyright Owner and License
Author
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.