Evaluating Retention Interventions Using Causal Inference

A college launches a new advising program, and student retention rises from 72% to 78%.

Success, right?

Maybe. But perhaps the incoming class was academically stronger. Maybe financial aid increased at the same time. Perhaps fewer students were working full time, or the regional economy changed enough to make staying in school easier.

This is the problem evaluating retention interventions using causal inference is designed to solve.

Higher education institutions increasingly invest in coaching, emergency grants, tutoring, early alerts, redesigned orientation, mental health support, and other strategies intended to keep students enrolled.

Tracking whether retention improves afterward is useful, but it does not automatically tell administrators whether the intervention caused that improvement.

Causal inference asks a harder question: What would probably have happened to the same students if the program had not existed?

That counterfactual cannot be observed directly, so colleges need research designs that approximate it.

Done well, causal evaluation helps institutions move beyond attractive dashboards and good intentions. It shows which interventions actually change student outcomes, which students benefit most, and where limited student-success dollars should go next.

Why Correlation Can Mislead Retention Decisions

Suppose students who regularly meet with academic advisors have a persistence rate of 88%, compared with 67% among students who rarely meet with advisors.

It would be tempting to conclude that advising caused the 21-point difference.

The problem is selection.

Students who voluntarily schedule advising appointments may already be more organized, academically engaged, or motivated to graduate. They might persist at higher rates even without the extra meetings.

This is a classic example of why correlation does not automatically establish causation.

The missing counterfactual

At the center of causal inference is a simple idea.

For each student, a college would ideally like to observe two outcomes:

1. What happened when the student received the retention intervention.
2. What would have happened to that exact same student at the same time without it.

Only one of those outcomes can actually occur.

The second is the counterfactual.

Researchers therefore construct a comparison group that is intended to approximate what would have happened without the intervention.

The quality of that comparison determines how confidently a college can say that a program produced an effect.

The What Works Clearinghouse, operated by the U.S. Department of Education’s Institute of Education Sciences, uses formal standards to evaluate whether education studies provide credible evidence.

Its current standards address randomized controlled trials, quasi-experimental studies, regression discontinuity designs, and other approaches intended to estimate intervention effects.

This matters for college leaders because a simple before-and-after comparison may be useful for monitoring.

It is usually much weaker for establishing cause.

Use Randomized Trials When the Setting Allows It

Randomized controlled trials, or RCTs, offer one of the clearest ways to estimate whether a retention strategy causes better student outcomes.

Eligible students are randomly assigned to different conditions.

One group receives the new intervention, while another receives standard services or another comparison condition.

Because assignment is random, the groups should be similar on average at the beginning.

That means later differences in persistence can more credibly be attributed to the intervention rather than differences in who chose to participate.

CUNY ASAP shows what strong causal evidence can reveal

The City University of New York’s Accelerated Study in Associate Programs, or ASAP, is one of the best-known examples.

ASAP combines several supports, including intensive advising, academic services, financial assistance, and structured pathways designed to help community college students complete degrees.

MDRC evaluated the program using random assignment.

After three years, approximately 40% of students offered ASAP had earned a degree, compared with about 22% of students receiving the colleges’ regular services. ASAP students also earned more credits and transferred to four-year institutions at higher rates.

That result is much more informative than discovering that students who voluntarily joined ASAP graduated more frequently.

Randomization helps separate program impact from the motivation or characteristics of participants.

Another randomized study by Eric Bettinger and Rachel Baker examined individualized student coaching across multiple colleges. Students assigned to receive coaching were more likely to persist during the intervention and were also more likely to remain enrolled after the coaching period ended.

These examples show why randomized evaluations are so valuable.

But RCTs are not always practical.

A college may already have implemented a program for everyone. Ethical concerns may make random assignment inappropriate. Administrators may also lack enough eligible students or operational capacity to create a randomized study.

Fortunately, causal inference does not end when randomization is unavailable.

Use Quasi-Experimental Designs When Randomization Is Not Practical

Quasi-experimental methods attempt to build a credible comparison without randomly assigning students.

They require more assumptions than a well-executed randomized trial, but carefully designed studies can still provide useful causal evidence.

The What Works Clearinghouse explicitly recognizes high-quality quasi-experimental designs as a legitimate part of the education evidence base, although researchers must demonstrate that comparison groups are sufficiently similar on important baseline characteristics.

Several approaches can work well in higher education.

Propensity score and matched comparison designs

Suppose a college introduced optional peer mentoring three years ago but never randomized participation.

Researchers could identify students who participated and create a comparison group of nonparticipants with similar observable characteristics.

Matching variables could include prior GPA, age, credit load, financial aid status, academic program, enrollment intensity, and earlier persistence behavior.

Propensity score methods are one way to make those groups more comparable.

But matching has an important limitation.

It can only balance characteristics the researcher can observe and measure.

If students joining mentoring were more motivated in ways the dataset never captured, some selection bias could remain.

That is why matched designs should not automatically be described as equivalent to randomized trials.

Difference-in-differences

Difference-in-differences can be useful when a program is introduced at one campus, department, or student group while a reasonable comparison group does not receive it.

Imagine College A introduces proactive advising in 2026 while College B continues its existing advising model.

Retention rises four points at College A.

On its own, that looks promising.

But suppose retention also rises three points at College B because of improving economic conditions.

A simple before-and-after analysis would attribute four points to advising.

A basic difference-in-differences logic suggests the intervention may be associated with closer to a one-point improvement after accounting for the broader trend.

The method becomes much stronger when the treatment and comparison groups had similar trends before the intervention.

If those trends were already moving differently, interpretation becomes more difficult.

Regression discontinuity

Regression discontinuity is particularly useful when access to an intervention depends on a cutoff.

For example, students with a placement score below 70 might automatically receive intensive tutoring, while those scoring 70 or higher receive regular services.

Students scoring 69 and 70 are usually fairly similar.

Yet one receives the intervention and the other does not.

Researchers can examine whether outcomes change sharply at that cutoff.

The What Works Clearinghouse recognizes regression discontinuity as a design capable of generating credible local causal estimates when the assignment rule is properly implemented and students cannot manipulate their position around the cutoff.

The important word is local.

The estimated effect usually applies most clearly to students near the cutoff, not necessarily to the entire college population.

Define the Intervention Before Trying to Measure It

Causal inference cannot rescue a poorly defined retention program.

Consider the phrase “enhanced advising.”

At one campus, that might mean students meet an advisor every two weeks.

At another, it could mean an automated email once per semester.

If researchers combine those experiences under one label, the estimated effect becomes difficult to interpret.

Measure implementation as carefully as outcomes

Before evaluating impact, colleges should document what students actually received.

How many coaching sessions occurred?

Did students attend tutoring?

How quickly were emergency grants distributed?

Did advisors follow the intended outreach schedule?

Was the intervention delivered consistently across campuses?

This is sometimes called implementation fidelity.

It matters because an intervetion may fail for two very different reasons.

The underlying idea might be ineffective.

Or the college might never have delivered the program as designed.

Those conclusions lead to completely different decisions.

Detroit Promise Path illustrates the value of studying both implementation and outcomes.

The program added coaching and financial incentives to an existing scholarship. MDRC’s randomized evaluation found positive effects on persistence and credit accumulation, including an 8.1-percentage-point increase in second-semester enrollment in its early results.

However, four-year follow-up found no statistically significant effect on credential attainment even though earlier enrollment impacts had been positive.

That distinction is extremely useful.

It shows why colleges should not assume that an improvement in an intermediate outcome automatically becomes a graduation effect later.

Choose Retention Outcomes That Match the Theory of Change

“Retention” sounds like one outcome.

In practice, it can mean several different things.

A college might measure fall-to-spring enrollment, fall-to-fall retention, persistence anywhere in higher education, credits attempted, credits completed, transfer, degree completion, or time to graduation.

Different interventions should influence different outcomes.

An emergency grant designed to help students through an immediate financial crisis might reasonably affect next-semester persistence.

It would be less realistic to expect a small one-time grant by itself to transform six-year graduation rates.

An intensive multiyear advising model can be held to a broader standard.

Measure short-, medium-, and long-term outcomes

A useful evaluation might examine:

Short term: course completion, credit accumulation, registration, or semester-to-semester persistence.

Medium term: annual persistence, academic standing, transfer, or accumulation of milestone credits.

Long term: graduation, time to degree, employment, or postgraduate enrollment.

This helps institutions understand the pathway through which an intervention works.

Imagine coaching improves first-year persistence but produces no change in degree completion.

That raises an important question.

Did the program merely delay withdrawal?

Did students eventually encounter barriers the program was not designed to address?

Or was the follow-up period too short?

Causal evaluation becomes much more informative when it tests a theory of change rather than hunting for whichever outcome happens to show statistical significance.

Look Beyond the Average Treatment Effect

Suppose an evaluation finds that a new advising intervention increases retention by four percentage points.

Useful information.

But the average can hide large differences.

Perhaps the effect is eight points for part-time students and nearly zero for full-time students.

Maybe first-generation students benefit more than students whose parents attended college.

An intervention could even improve overall persistence while providing little benefit to the students facing the greatest barriers.

Ask who benefits

Causal analysis can examine heterogeneous treatment effects, meaning whether impacts vary across groups or circumstances.

Colleges might investigate differences by enrollment intensity, age, academic preparation, major, income, campus, or other relevant student characteristics.

These analyses require care.

If researchers divide a small sample into 20 subgroups and test everything, random variation will eventually produce some apparently exciting results.

Subgroup analysis works best when the college identifies important questions before seeing the results and has a reasonable theoretical explanation for why effects might differ.

Equity should be part of this conversation.

A retention strategy that works primarily for students who already have strong academic resources may still be useful, but it probably should not be marketed as closing persistence gaps.

Institutions should ask whether interventions are merely raising the average or meaningfully improving outcomes among students facing structural barriers.

Do Not Confuse Statistical Significance With Practical Value

A very large study can detect tiny effects.

That does not mean every statistically significant effect is worth scaling.

Suppose a text-message intervention improves fall-to-fall retention by 0.4 percentage points.

With enough students, that result might be statistically significant.

But college leaders still need to ask whether the gain justifies the cost and administrative effort.

Now imagine a coaching program improves retention by seven percentage points but costs substantially more.

The decision becomes more complicated.

Effect size and cost belong in the same conversation

Evaluation should consider:

How large is the impact?

How confident are we that it is real?

How much does the intervention cost per student?

How many additional students persist?

What is the cost per additional retained student or graduate?

Could the same resources produce greater effects through another strategy?

The CUNY ASAP evaluation is interesting partly because the program required meaningful investment but produced such large graduation gains that MDRC found its cost per graduate was lower than the cost under regular services at the three-year point.

This is why “cheap” and “cost-effective” are not the same thing.

An inexpensive intervention producing almost no improvement may offer poor value.

A more expensive intervention can be economically attractive when the resulting gains are substantial.

Test Whether Successful Results Transfer to New Settings

A strong causal effect at one institution answers an important question:

Did the program work there?

It does not fully answer another:

Will it work here?

This is the issue of external validity.

Colleges differ in student populations, geography, staffing, financial resources, academic policies, and surrounding labor markets.

A retention intervention developed for a large urban community college may need modification before it works in a rural college or a four-year university.

Replication therefore matters.

The ASAP model offers an instructive example.

Three Ohio community colleges adapted the CUNY model and were evaluated using random assignment.

MDRC reported that the Ohio programs nearly doubled three-year graduation rates and increased transfers to four-year colleges, producing impacts similar in magnitude to the original CUNY results.

That kind of replication strengthens confidence that the underlying model is not effective only because of something unique about one campus.

But adaptation creates another challenge.

Change too little, and a program may not fit the local institution.

Change too much, and the college may no longer be implementing the evidence-based model that produced the original effect.

Good replication identifies the core components that should remain intact while allowing operational details to reflect local conditions.

Build Causal Evaluation Into Normal Student-Success Operations

Causal evaluation should not be reserved for major research grants.

Colleges can increasingly design everyday initiatives in ways that make learning possible.

When demand exceeds available program capacity, random assignment may sometimes provide both a fair allocation mechanism and a valuable evaluation opportunity.

When policies use eligibility thresholds, regression discontinuity may already be available naturally.

When implementation occurs in phases, colleges may be able to compare early and later campuses or cohorts.

Administrative data can also reduce evaluation costs because enrollment, credits, grades, and completion outcomes are often already recorded.

The key is planning before launch.

Researchers cannot easily reconstruct a strong evaluation design after everyone has already received the intervention and baseline information was never collected.

That is why institutional research teams should be involved while retention programs are being designed.

A relatively simple question can change the entire process:

“How will we know whether this caused better outcomes?”

Asking that before implementation encourages colleges to define comparison groups, outcomes, timing, data requirements, and success criteria in advance.

It also makes disappointing results useful.

If a rigorous study shows that an intervention does not improve persistence, that is not necessarily failure.

The college has learned something valuable before spending millions of dollars scaling an ineffective strategy.

Evaluating retention interventions using causal inference helps colleges move from asking whether student outcomes changed to asking whether a specific strategy actually caused the change.

Randomized trials offer powerful evidence when they are feasible, while quasi-experimental methods such as matching, difference-in-differences, and regression discontinuity can provide credible alternatives when randomization is not practical.

But research design is only part of the job.

Colleges also need clear interventions, meaningful outcomes, implementation data, subgroup analysis, cost information, and enough follow-up to see whether early gains last.

For student-success leaders, the next step is simple: before launching the next retention initiative, define the counterfactual.

Do not just ask whether retention improved. Ask what would have happened without the program – and design the evaluation so you can answer it.