POLI 171: A Summary

Policy Making with Data

Two big questions to address

• Why do we need to use data in policy analysis and evaluation

• How do we use data?

Why do we do this?

Turns out͕ there is a lot of things ǁe don͛t knoǁ about the ǁorld

Why do we do this?

Turns out͕ there is a lot of things ǁe don͛t knoǁ about the ǁorld

͙and a lot of things ǁe thought we knew about the world

Oh ƚhe ƚhings ǁe don͛ƚ knoǁ

In the past, many terrible policies have been made with arguably good

intentions

• The United States recruited and sent people who measured below

mental and medical standards to Vietnam͕ hoping to ͞training and

opportunitǇ͟ to the uneducated and poor

• China exterminated rats, flies, mosquitoes, and sparrows

in an attempt to protect crops

• India and many countries instituted a ban on child labor

• Australia fought a war ʹ and lost ʹ against emus

Rigorous empirical research is the only way to subject our

beliefs and intentions to test

How we do this

The backbone of our analysis is the Potential Outcome Framework

How we do this

The backbone of our analysis is the Potential Outcome Framework

• The potential outcome model

• Causal effects

• The fundamental problem of causal inference

• Causal estimands: ATE, ATT

• Omitted variable bias

The potential outcome model

The potential outcome model

• For everǇ ͞treatment͟ ;a policǇ͕ membership in an

organization/community/group, a given characteristic, etc.), and for

every outcome, each observation has two potential outcomes

• An outcome under treatment condition (Y1)

• An outcome under control condition (or, in the absence of the treatment) (Y0)

• Which of the two outcome is exhibited depends on treatment status

• Let the observed outcome be Y

• If observation is treatedÆ we observe only treated outcome: Y=Y1

• If observation is untreated Æ we observe only untreated outcome: Y=Y0

Causal Effect

Fundamental Problem of Causal Inference

1. We can never observe both Y1i and Y0i simultaneously

2. As a result, we can never know causal effect with certainty

Causal Estimand: Average Treatment Effect

(ATE)

Average Treatment Effect on the Treated (ATT)

Omitted variable bias

Omitted variable bias

What is NOT omitted variable bias:

• Variables that influence likelihood of getting treatment but absolutely no

independent relationship with outcome

• e.g. A thunderstorm makes large-scale protests less likely to happen (treatment), but

;arguablǇͿ has no independent relationship ǁith federal government s͛ ǁillingness to

implement social change

• In practice, quite difficult to find example of things that truly have no independent

relationship with outcome

• Variables that influence outcome but have no relationship with treatment

• e.g. The amount of sleep is correlated with adult height (outcome), but has no

relationship with the amount of milk consumed during childhood

• Also similar: Variables in how treated observations take up a treatment

• e.g. Whether people wear masks correctly influence COVID-19 infection likelihood

(outcome), but does not influence likelihood of wearing mask (treatment)

Omitted variable bias

• Selection bias:

• A characteristic of an individual that makes them systematically more or less

likely to select themselves into the treatment condition AND exhibit

systematically different outcome

• e.g. Diligence. Diligent students are more likely to attend review session (the

treatment) and also tend to score higher in exams (the outcome)

• Endogeneity (aka reverse causality)

• Where an individual s͛ outcome influences their tendencǇ to get treatment

• e.g. Healthy people tend to eat well and engage in regular exercise, which in

turn improve health

Identification strategies

1.Experimental methods: Randomized Control Trials

2.Non-experimental methods

• Matching

• Regression

• Difference-in-Differences

Randomized experiments

• What are the stages of an experiment?

• What does random assignment do?

• How to estimate the treatment effect in an experiment?

• How to improve precision?

• Assumptions?

• Advanced designs?

Stages of an experiment

Random Assignment Prevents Omitted

Variable Bias

Estimation in randomized experiments

We use the difference in means estimator, and test for its statistical

significance using a t-test.

All of this are included in R through the lm() function:

݈݉݀݁ ൏ െ ݈݉ሺ~ݐݎ݁ܽݐ, ݀ܽݐܽሻ

ݏݑ݉݉ܽݎݕሺ݈݉݀݁ሻ

Accuracy vs. Precision

(Unbiasedness vs. Reliability)

How to increase precision:

Increase the size of our sample

• Higher sample -> law of large number kicks in -> lower impact of extreme outliers

Make our treatment group smaller than control group

• Technically reduces precision, but allows you to offer much bigger sample size given same

cost

Controlling for pre-treatment variables

• Reduce variations in outcome that͛s not caused bǇ variations in treatment status

Differencing our outcome variable

• Reduce variations in outcome that s͛ not caused bǇ variations in treatment status

Blocking on pre-treatment variables

• Increases similarity between treated and control group with regard to blocked variables

Clustering

• Actually decreases precision in exchange for less costly implementation AND reduce chance

of spillover effect

How to increase precision:

Precision is reflected in standard error

Standard error: The standard deviation of a sampling

distribution of an estimate

Lower precision -> Larger standard error compared to the

estimated treatment effect -> lower p-value

Assumptions

Excludability:

OnlǇ the treatments and nothing else outside the researcher s͛

control are ͞assigned͟ to the groups

Non-interference/No spillovers/SUTVA:

One unit s͛ treatment status should not influence another unit s͛

outcome

Assumptions can never be tested!

Advanced designs

Multiple treatment arms

• One group receives no treatment

• One group receives treatment A1

• One group receives treatment A1 + A2

• One group receives treatment Aϭ н AϮ н Aϯ͙

• Effect of each component estimated by comparing one group with the one

immediate to it

Factorial experiment (Interaction effects)

• One group receives no treatment

• One group receives treatment A

• One group receives treatment B

• One group receives treatment A + B

• Effect of interaction effect estimated by comparing A+B effect with sum of A s͛ and B͛s

effect

Non-experimental designs

When we do this?

• We have some treated and control units

• We didn͛t assign the treatment

Methods

• Matching

• Regression

• Diff-in-diff

What we covered

• Intuition

• Assumptions

• Code

Matching: Intuition

• For each treated unit, find one control/untreated unit that resembles

it the most in pre-treatment variables

• Discard all control observations that have no match

• Then, pretend we have an experiment and perform the same analysis

Matching: Assumptions

• Selection on observables:

• Whatever drives selection into treatment or control group have already been

observed and measured

• Two units that have the same observed pre-treatment variables have the

same likelihood of being in treated or control group.

• Their eventual treatment status is ͞as-if͟ random

Matching: Code

Matching and estimation performed through Match() function in Matching package

݉ܽݐ݄ܿ.݈݉݀݁ ൏ െܽݐ݄ܿሺ, ݎ, , ൌ 1, ݁ݔܽܿݐ ൌ , ݎ݈݁ܽܿ݁ ൌ ,

݁ݏݐ݅݉ܽ݊݀ ൌ "", ݅ܽݏ݆݀ݑݏݐ ൌ ሻ

ݏݑ݉݉ܽݎݕሺ݉ܽݐ݄ܿ.݈݉݀݁ሻ

Y A vector of outcomes. Example: df$outcome

Tr A vector of treatment status. Example: df$treat

X A vector of pre-treatment variables to match on. Example:

df͕c;͞age͕͟͟income͕͟͟educ͟Ϳ

M M matches per treated unit

exact Whether to do exact matching

replace Whether to reuse matched control units

estimand Which quantity to estimate.

BiasAdjust Whether to do extra regressions to adjust for remaining imbalances. Needs

replace=TRUE to work.

Matching: Code

• Exact matching: Set the argument exact=TRUE in Match() function

• Tips: Try to use only categorical or binary variables

• Distance matching: Set the argument exact=FALSE in Match() function

• Default is normalized Euclidean distance, which is somewhat similar to Mahalanobis

distance

• Propensity score matching

• Manually calculate propensity score:

model. ݎ ൏ െ ݈݉ ݐݎ݁ܽݐ~ݔ1 ݔ2 ݔ3, ݀ܽݐܽ ൌ ݂݀

ݎ ൏ െ݈݉݀݁. ݎ$݂݅ݐݐ݁݀. ݒ݈ܽݑ݁ݏ

• Then put the vector of fitted values into the argument X=prop in Match() function

match.model ൏ െܽݐ݄ܿሺ, ݎ, ൌ ݎ, ൌ 1,

݁ݔܽܿݐ ൌ , ݎ݈݁ܽܿ݁ ൌ ,

݁ݏݐ݅݉ܽ݊݀ ൌ "", ݅ܽݏ݆݀ݑݏݐ ൌ ሻ

ݏݑ݉݉ܽݎݕሺ݉ܽݐ݄ܿ.݈݉݀݁ሻ

Matching: Code

Balance tests performed through MatchBalance() function

ܽݐ݄݈ܿܽܽ݊ܿ݁ ݂ݎ݉ݑ݈, ݀ܽݐܽ,݉ܽݐ݄ܿ. ݑݐ

formul Treatment status variable on left, pre-treatment

variables on right. Example: treat~x1+x2+x3

data The dataset containing observations to match

match.out Output of a Match() function. Include when you

want to compare before vs. after match

Regression: Intuition

• Do not discard any unit

• Include all pre-treatment variables into a regression model, and take

advantage of its poǁer to statisticallǇ ͞hold everǇthing constant͟

• We consider the coefficient of the treatment variable our estimated

treatment effect

• It s͛ like magic͕ but cooler

Regression: Assumptions

• Selection on observables

• Linear relationships of variables on outcome

• A bunch of other assumptions about the standard errors

Regression: Code

Simply use the lm() function

݈݉݀݁ ൏ െ ݈݉ ~ݐݎ݁ܽݐ ݔ1 ݔ2 ݔ3… , ݀ܽݐܽ

ݏݑ݉݉ܽݎݕሺ݈݉݀݁ሻ

Regression: Code

If you include a categorical variable in the model, or convert a

numerical variable into categorical using as.factor(variable), R will

perform a fixed effects regression

• Do this when you suspect observations from different groups behave

differently in ways you cannot fully measure

• When reading regression outcomes, focus on estimated treatment

effect and standard error of the treatment ʹ don͛t ǁorrǇ too much

about the many estimates of the fixed effects

Difference in differences: Intuition

• Two groups, two time periods

• In first period, no group receives treatment

• In second period, one group receives treatment

• We measure ;ϭͿ hoǁ first group s͛ outcome changes betǁeen Ϯ

periods͕ and ;ϮͿ hoǁ second group s͛ outcome changes betǁeen Ϯ

periods

• Take the difference between (2) and (1) to find the treatment effect

Difference in differences: Assumptions

• Parallel trends: Outcomes of treated group would have moved the

same way as the outcome group in the absence of treatment

• Stable Composition: Groups have same membership over time

• ͞Nothing else happens͗͟ Treatment is the onlǇ thing that happens to one

group and not other after the treatment

Difference in differences: Code

Estimation is performed through lm() function

First, find out if your data is in the long or in the wide format

Long format:

݈݉݀݁ ൏ െ ݈݉ ~ݐݎ݁ܽݐ ݂ܽݐ݁ݎ ݐݎ݁ܽݐ ∗ ݂ܽݐ݁ݎ ݔ1 ݔ2, ݀ܽݐܽ

ݏݑ݉݉ܽݎݕሺ݈݉݀݁ሻ

treat whether observation comes from group that eventually

gets treatment

after whether observation is in post-treatment period

x1, x2 additional controls

Difference in differences: Code

Estimation is performed through lm() function

First, find out if your data is in the long or in the wide format

Wide format:

݂݀$݂݂݀݅ ൏ െ ݂݀$1 െ ݂݀$0

݈݉݀݁ ൏ െ ݈݉ ݂݂݀݅~ݐݎ݁ܽݐ ݔ1 ݔ2, ݀ܽݐܽ ൌ ݂݀

ݏݑ݉݉ܽݎݕሺ݈݉݀݁ሻ

treat whether observation comes from group that eventually

gets treatment

after whether observation is in post-treatment period

x1, x2 additional controls

Difference in differences: Code

Be aware that the standard errors of diff-in-diffs estimates are often

wrong

Solutions: Clustered standard errors, HC standard errors,

bootstrapping, etc.

How can I remember all of this?

The ansǁer͗ No͕ Ǉou can͛t

The ansǁer͗ No͕ Ǉou can͛t

͙ but that s͛ alright

Iƚ s͛ alrighƚ ƚo forgeƚ sƚƵff

Causal Inference

• You͛re gonna forget all the Y1, Y0 stuff

• But Ǉou͛ve seen hoǁ good research is done

Statistics

• You͛re gonna forget bias correction and clustered SEs

• But you know good statistical analysis is not scary

R

• You͛re gonna forget all the messy arguments or how to fix a for loop

• But hopefullǇ Ǉou͛re not afraid of ǁriting code anǇmore

Key take-aways

Correlation is not causation

• Mainly because of selection bias

Compare like with like

• Find methods to eliminate selection bias

Think of the counterfactuals

• Use statistics to predict counterfactuals

No substitution for good on-the-ground research

• Assumptions are eǆamined through intense detective ǁork and ͞knoǁing the case͟

Seeks evidence to falsify your beliefs, not to confirm them

• Hypothesis testing matters in real life

What can you do with this knowledge

• Jobs in business analytics, government, or non-profit sector

• Data analysis

• Consulting

• Field research

What can you do with this knowledge

• Jobs in business analytics, government, or non-profit sector

• Data analysis

• Consulting

• Field research

• Bridge the ideological gap in debates on social issues

What can you do with this knowledge

• Jobs in business analytics, government, or non-profit sector

• Data analysis

• Consulting

• Field research

• Bridge the ideological gap in debates on social issues

• Support your causes