Creating New ExperimentsΒΆ
WhyNot easily and flexibly supports the creation of new causal experiments and new experiment designs. Below, we describe how to use primitives in WhyNot to construct a new experiment for both dynamical system models. For more examples or inspiration for experiment creation and design, look at the many examples in Examples.
In WhyNot, the class DynamicsExperiment
encapsulates
the logic necessary to define a causal inference experiment on a dynamical
system simulator. For any causal inference experiment, we always need to
specify:
- The treatment \(A\)
- The observed outcome \(Y\)
- The observed covariates \(X\).
- The treatment assignment rule
In a dynamical system simulation, we need to further specify:
- The dynamics (i.e. which simulator we use)
- The initial state distribution.
Once we specify these components, the
DynamicsExperiment
does the rest of the work to
efficiently run the simulations, assign treatment, and construct the
observational dataset. Before launching into the details of each of these
components, we first give an example on the Lotka-Volterra Model.
import whynot as wn
def sample_initial_state(rng):
"""Sample an initial state, i.e. a population of rabbits and foxes."""
rabbits = rng.randint(10, 100)
foxes = rabbits * rng.uniform(0.1, 0.8)
return wn.lotka_volterra.State(rabbits=rabbits, foxes=foxes)
def observed_outcome(run):
"""Compute the minimum fox population in the 20 years before year 80."""
return np.min([run[year].foxes for year in range(60, 80)])
rct = wn.DynamicsExperiment(
name="lotka_volterra_rct",
description="A RCT to determine effect of reducing rabbits needed to sustain a fox.",
simulator=wn.lotka_volterra,
simulator_config=wn.lotka_volterra.Config(fox_growth=0.75),
intervention=wn.lotka_volterra.Intervention(time=30, fox_growth=0.4),
state_sampler=sample_initial_state,
propensity_scorer=0.5,
outcome_extractor=observed_outcome,
covariate_builder=lambda run: run.initial_state.values())
Each of the arguments to DynamicsExperiment`
determines one aspect of the causal experiment.
- The
simulator
specifies that the experiment is run on the Lotka-Volterra simulator. - The
simulator_config
sets the parameters of the simulator. - The
intervention
specifies what intervention to perform for the treatment group. In this case, the intervention corresponds to reducing thefox_growth
parameter in year30
. - The
state_sampler
generates samples from the specified initial state distribution. - The
propensity_scorer
determines the probability of treatment assignment for a particular unit. Setting thepropensity_scorer
to a constant0.5
randomly assigns treatment with probability0.5
to each unit. - The
outcome_extractor
computes the observed outcome \(Y\) from a run of the simulator. - The
covariate_builder
extracts the observed covariates \(X\) from a simulator run.
This code can then be executed to generate the observational dataset.
>>> dataset = rct.run(num_samples=200, parallelize=True)
We can also generate experiments with confounding if the propensity_scorer
depends on the simulator state.
import whynot as wn
def confounded_propensity_scores(untreated_run):
"""Return confounded treatment assignment probability.
Treatment increases fox population growth. Therefore, we're assume
treatment is more likely for runs with low initial fox population.
"""
if untreated_run.initial_state.foxes < 20:
return 0.8
return 0.2
confounding_exp = wn.DynamicsExperiment(
name="lotka_volterra_confounding",
description=("Determine effect of reducing rabbits needed to sustain a "
"fox. Treament confounded by initial fox population."),
simulator=wn.lotka_volterra,
simulator_config=wn.lotka_volterra.Config(fox_growth=0.75),
intervention=wn.lotka_volterra.Intervention(time=30, fox_growth=0.4),
state_sampler=sample_initial_state,
propensity_scorer=confounded_propensity_scores,
outcome_extractor=observed_outcome,
covariate_builder=lambda run: run.initial_state.values())
In the previous two examples, we hard-coded several parameters into the
experiment specification. For instance, we set the treatment probabilities in
the confounding example to 0.8
and 0.2
depending on the initial state.
However, we often want to run experiments for a set of parameters. For instance,
rather then consider a single propensity score setting, we could study the
performance of a family of estimators as the strength of the confounding
varied. In WhyNot, the @parameter
decorator allows us to do precisely that.
import whynot as wn
@wn.parameter(name="propensity", default=0.9,
description="Treatment prob for group with low fox counts.")
def confounded_propensity_scores(untreated_run, propensity):
"""Return confounded treatment assignment probability.
Treatment increases fox population growth. Therefore, we're assume
treatment is more likely for runs with low initial fox population.
"""
if untreated_run.initial_state.foxes < 20:
return propensity
return 1. - propensity
confounding_exp = wn.DynamicsExperiment(
name="lotka_volterra_confounding",
description=("Determine effect of reducing rabbits needed to sustain a "
"fox. Treament confounded by initial fox population."),
simulator=wn.lotka_volterra,
simulator_config=wn.lotka_volterra.Config(fox_growth=0.75),
intervention=wn.lotka_volterra.Intervention(time=30, fox_growth=0.4),
state_sampler=sample_initial_state,
propensity_scorer=confounded_propensity_scores,
outcome_extractor=observed_outcome,
covariate_builder=lambda run: run.initial_state.values())
When a method is decorated with @parameter
, the run
method of the DynamicsExperiment
allows the parameter to be passed in. This make it very easy to generate a
sequence of observational datasets with as the parameter varies.
datasets = []
for propensity in [0.5, 0.7, 0.9, 0.95]:
dataset = confounding_exp.run(num_samples=1000, propensity=propensity)
datasets.append(dataset)
As the above examples suggest, DynamicsExperiment
is
very flexible. For all of the details of permissible specifications of the
state_sampler
, propensity_scorer
, etc., please refer to the
API
.