1.3 · The Quant Research Lifecycle

1.3.1 The lifecycle at a glance

This diagram is the map of the whole course. Almost every later session zooms into one of its boxes.

fig 1.3.1 — the research lifecycle: nine stages, two hard gates, and a loop that turns retired strategies into new questions

Read it as a production pipeline, because that is what it is. The top row is research: you turn a question into a signal — a number, computable from data available at the time, that forecasts something — and simulate trading it in a backtest. The bottom row is deployment: you size it, trade it with fake money on live data (paper trading), then with real money, and watch it until it stops working. Two stages are sticky yellow because they are gates: validation, where an idea must beat a statistical bar it did not choose, and paper trading, where it must survive contact with real market data and real execution.

📌 Key idea — the backtest is not the finish line

Beginners treat stage 4 as the goal: once the backtest looks good, they are done. Professionals treat it as the middle. A backtest is one noisy measurement of a hypothesis, taken on data you have probably already looked at. Everything after it exists to answer one question: is this real, or is it the best of many accidents?

🧭 Margin note — the engineering analogy

You already know this pipeline. Question = design doc; signal = the code; backtest = unit tests; validation = a test suite you did not write; paper trading = staging; live = production with a small canary; monitoring = observability; retirement = decommissioning. The difference is that in markets your tests can pass by luck — which is why this course spends a whole unit (Session 6.1 onward) on making them fail honestly.

1.3.2 Start with a question, not a backtest

A strategy without an economic story is a pattern in noise waiting to be discovered. Every durable edge answers three questions before any code is written.

fig 1.3.2 — the three questions every hypothesis must answer, illustrated with short-term reversal

A hypothesis is a falsifiable claim about why prices behave a certain way and who is on the other side. "Stocks that fell a lot this week tend to bounce next week" is an observation. "Stocks that fell a lot this week bounce because the sellers were in a hurry and liquidity providers charge for absorbing them" is a hypothesis — it predicts where the effect should be stronger (illiquid names, stressed markets) and weaker (after costs, at scale). Those predictions are what let you test it without data-mining. Session 1.2 — Where Returns Come From catalogues the honest sources of edge you can build a story on.

Weak starting point	Why it fails	Strong starting point
"Let's see which indicators predict returns"	You will find some — by chance (§1.3.4)	"Forced sellers overshoot; who is forced, and when?"
"This pattern made money in the backtest"	No reason for it to continue	"This is a risk premium: who bears the risk, and why do they accept the price?"
"An ML model on 500 features"	Thousands of hidden trials	"Three features, each with a story, combined simply"
"It works on Bitcoin since 2023"	One asset, one regime	"It should work across assets where the same mechanism applies"

💡 Looking ahead

Unit 7 turns this into a method: Session 7.5 — Causal Factor Investing uses causal graphs to check that a factor's story is not just a correlation with a hidden cause. Unit 8 walks through the strategy families whose stories have survived decades of scrutiny.

1.3.3 Gates, not vibes

A gate is a go/no-go decision with a criterion written down in advance. Without gates, every idea drifts forward on enthusiasm.

Stage	Gate question	Typical kill criterion	Taught in
1 · Question	Is there a plausible counterparty and mechanism?	No answer to "who pays me?"	Session 1.2, Unit 7
2 · Data	Could I have known this at the time?	Any look-ahead or survivorship you cannot remove	Unit 4
3–4 · Signal & backtest	Does it survive realistic costs?	Net Sharpe below a floor you set in advance	Session 6.2–Session 6.5
5 · Validate	Is it better than the best of my failed tries?	Deflated Sharpe below threshold; holdout negative	Session 6.6–Session 6.8
6 · Size	Does it add to the existing portfolio?	Correlated with what you already run; no diversification	Unit 9
7 · Paper	Do live signals and fills match the simulation?	Implementation gap larger than the edge	Unit 15
8–9 · Live	Is live performance inside the expected range?	Drawdown or decay beyond the pre-agreed limit	Session 9.9

The cheapest place to kill an idea is the first gate that can kill it. Killing it at the question stage costs an afternoon. Killing it after paper trading costs months. Killing it live costs money — and, worse, the temptation not to.

fig 1.3.3 — an illustrative idea funnel: a researcher who kills most ideas early is working, not failing

⚠️ Pitfall — "I'll decide what counts as good once I see it"

Choosing the bar after seeing the result is the single most common way smart people fool themselves. The mind is superb at explaining why this result is good enough. Write the kill criterion first; if you later want to change it, write down why and treat it as a new, separately counted trial.

1.3.4 The garden of forking paths

Why do gates matter so much? Because if you try enough strategies, one of them will look excellent even when none of them has any skill. The lab proves it.

The lab builds strategies with zero true edge — returns that are pure noise — and gives each 5 years of history. It then plays researcher: try N strategies, keep the one with the best Sharpe ratio (return per unit of risk), and repeat the whole project 200 times to see what typically happens.

fig 1.3.4 — the best of N skill-less strategies: the more you try, the better the winner looks

fig 1.3.5 — the same winners on five unseen years: indistinguishable from zero, however impressive they looked

Try one strategy and its in-sample Sharpe averages 0.04 — noise around zero, as it should. Try 100 and the winner averages 1.13; try 1,000 and it averages 1.43, a number many funds would advertise. With 1,000 tries, a winner with a Sharpe above 1.0 is essentially certain: it happened in 200 of 200 runs. Out of sample, the winner earns −0.02. There was never anything there.

This is not a simulation quirk; it is a theorem. An estimated annual Sharpe ratio has a standard error of about $1/\sqrt{Y}$ after $Y$ years when the truth is zero, and the expected maximum of $N$ such noisy estimates is, approximately:

\E\Big[\max_{n \le N} \widehat{\SR}_n\Big] \;\approx\; \frac{1}{\sqrt{Y}}\Big[(1-\gamma)\,\Phi^{-1}\big(1 - \tfrac{1}{N}\big) + \gamma\,\Phi^{-1}\big(1 - \tfrac{1}{N e}\big)\Big]

where $\Phi^{-1}$ is the inverse normal CDF and $\gamma \approx 0.577$ is the Euler–Mascheroni constant (the formula needs $N \ge 2$ ; for a single strategy the expected estimate is simply 0). For $N = 1{,}000$ and $Y = 5$ it predicts 1.46; the lab measured 1.43. This is the False Strategy Theorem (López de Prado & Bailey, 2021), and Session 6.7 — Overfitting Statistics: PSR, DSR, MinTRL & PBO turns it into a correction you apply to every backtest: the deflated Sharpe ratio.

Strategies tried (N)	Best in-sample Sharpe	Theory	P(best > 1.0)	Winner out of sample
1	0.04	0.00	2%	−0.02
10	0.68	0.70	12%	−0.03
100	1.13	1.13	72%	−0.01
1,000	1.43	1.46	100%	−0.02

⚠️ Pitfall — "but I didn't try 1,000 strategies"

You almost certainly did. Every parameter you tuned, every lookback you tweaked, every universe you narrowed, every "let me just try it without 2008" is a trial. So is every idea a colleague tried before you and told you about, and every strategy in a paper that was published because it worked. Counting them honestly — a trial log — is the only defence (highly correlated variants count as fewer independent trials; Session 6.7 shows how to estimate the effective number). Even one honest test is noisy: with five years of data, a skill-less strategy shows a Sharpe above 0.5 by luck 13% of the time.

1.3.5 The research log & pre-registration

Two documents make the lifecycle honest: a pre-registration written before you look, and a research log that records everything you did after.

Pre-registration means writing down, before touching the test data, exactly what you will test and what result would kill the idea. Medicine made it mandatory for clinical trials in the 2000s, and psychology and economics adopted it after their replication crisis; it is the quant equivalent of writing the test before the code. The lab writes one for a short-term reversal idea:

# preregistration R-001 — committed to git BEFORE looking at the data
hypothesis:       "Stocks with the worst 5-day return outperform over the next 5 days,
                   because liquidity providers are paid to absorb selling pressure."
who_pays_us:      "impatient sellers demanding immediacy"
universe:         "US large caps, point-in-time index membership"
data:             "2005–2019 in-sample; 2020–2024 held out, untouched"
signal:           "rank of 5-day return, rebalanced weekly, dollar-neutral"
primary_metric:   "annualised Sharpe after 10 bp one-way costs"
variants_allowed: 12
kill_criterion:   "deflated Sharpe ratio (a probability) below 0.95 in-sample, OR holdout Sharpe below 0"
holdout_rule:     "evaluated exactly once, after the in-sample decision is frozen"

The research log is the commit history of your thinking (Session 1.5 sets up the reproducible workbench that makes it trustworthy): every variant run, every result, every decision and its reason, with a date. It sounds like bureaucracy; it is actually leverage. It gives you the true trial count for deflating results (Session 6.7), it stops you re-running dead ideas, and when a strategy fails live it tells you exactly what you assumed.

📌 Key idea — the holdout is a single-use resource

Data you have looked at is in-sample, whatever you call it. The moment you check a holdout, adjust, and check again, it has joined your training set. Treat out-of-sample data like a one-time password: once used, it is spent. The only unlimited source of fresh out-of-sample data is time — which is what paper trading buys.

Record	When	What it protects against
Pre-registration	before any test	moving the goalposts; hindsight stories
Trial log	every run	undercounting trials; re-testing dead ideas
Decision memo	at each gate	enthusiasm carrying an idea past a failed gate
Live monitoring report	weekly / monthly	slow decay nobody notices until the drawdown

1.3.6 Paper, live & decay

The bottom row of the lifecycle is where most of the money is lost — not because the research was wrong, but because reality differs from simulation, and edges erode.

Paper trading runs the strategy on live data with simulated fills. It catches problems a backtest cannot: data that arrives late or gets revised, signals that depend on prices you could not actually trade at, costs larger than assumed. Then comes live trading, small — sized so that being wrong is survivable — and scaled up only as live results land inside the range the backtest predicted.

And every edge decays. Once an anomaly is published, others trade it: McLean and Pontiff found that published return predictors earn about 26% less out-of-sample (after the original study's sample ends but before publication) and 58% less after publication. Crowding, changing market structure and regulation all erode edges over time. Alpha decay is not a failure of the strategy; it is the normal lifecycle of an edge, which is why the loop in fig 1.3.1 ends at retirement and feeds back into new questions.

fig 1.3.6 — two researchers, one idea: the difference is the process, not the intelligence

💡 Looking ahead

The evidence on how far backtests overstate live results — and on who actually makes money trading — is the subject of Session 1.4 — The Engineer's Edge, Blind Spots & Honest Odds. The mechanics of paper and live trading are Unit 15; deciding when to switch a strategy off is Session 9.9 — Performance, Attribution & When to Switch Off.

1.3.7 Try it

Open the companion lab and become the researcher who finds alpha in noise — on purpose, so you recognise it later.

🔧 Try it (15 min, in the lab)

1. Before running anything, use the formula in §1.3.4 to predict the best in-sample Sharpe for $N = 100$ strategies with $Y = 10$ years. Write your prediction down.
2. Change YEARS_IS to 10 and rerun. Did more data shrink the best fake Sharpe the way the formula says (by $1/\sqrt{2}$ )?
3. Success looks like this: your prediction is within 0.1 of the lab's number, and you can explain in one sentence why longer histories protect you — and why they never make $N$ irrelevant.

1.3.8 Glossary · Check yourself · Go deeper

Hypothesis — a falsifiable claim about why prices behave a certain way and who is on the other side.

Signal — a number computable from data available at the time that forecasts a future return.

Backtest — a simulation of how a trading rule would have performed on historical data.

Out-of-sample — data not used in any way to build or choose a strategy; spent once looked at.

Paper trading — running a strategy on live data with simulated money and fills.

Alpha decay — the erosion of an edge over time as others trade it or markets change.

Research log — a dated record of every trial, result and decision in a research project.

Pre-registration — writing down the test, metrics and kill criteria before seeing results.

Kill criterion — a pre-agreed result that ends an idea at a gate, no arguments.

Check yourself

1. A colleague shows a strategy with a 5-year backtest Sharpe of 1.4, found after "a few hundred" variations. How impressed should you be?

Not very. With no skill at all, the best of 100 five-year backtests averages a Sharpe of about 1.1, and the best of 1,000 about 1.4. Ask for the trial count, deflate the Sharpe for it (Session 6.7), and ask whether an untouched holdout was used exactly once.

2. At which stage should an idea ideally die — and why?

At the earliest gate that can kill it, usually the question or data stage. The cost of killing an idea rises by orders of magnitude at each stage — an afternoon, then weeks, then months, then real money — and so does the emotional investment that makes killing it harder.

3. You test on your holdout once, it fails, you tweak the rule and test on the holdout again. What is the holdout now?

In-sample. The tweak was chosen using the holdout's result, so it no longer measures out-of-sample performance. Fresh out-of-sample data now has to come from time: paper trading or a later data period.

4. After three months of paper trading, a strategy's Sharpe is 0.3 against a backtest of 1.1. Kill it?

Not on performance alone: three months of returns cannot distinguish a Sharpe of 1.1 from 0 — the standard error is around 2, so the t-statistic is about 0.5. Check the implementation instead: are signals and fills matching the simulation? The pre-registered paper-trading gate should specify a minimum duration and what "matching" means.

5. Why must the kill criterion be written before the result is known?

Because after seeing the result you will, sincerely, find reasons why it is good enough. It is the research version of writing the test before the code: the criterion constrains you only if it exists before the temptation does.

🔧 Homework (~1 h before Session 1.4)

1. Write a pre-registration for a trading idea you have had, using the template in the lab (section 4). Commit it to a git repository so it has a timestamp.
2. In the lab, set TRIALS = [1, 10, 100, 1000, 5000]. Predict the N = 5,000 result with the formula first, then run it.
3. Read Bailey, Borwein, López de Prado & Zhu (2014), below — fourteen pages, and the best argument you will read for why this session exists.

📚 Go deeper

Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2014). "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance." Notices of the AMS 61(5), 458–471. doi:10.1090/noti1105
López de Prado, M. & Bailey, D. H. (2021). "The False Strategy Theorem: A Financial Application of Experimental Mathematics." American Mathematical Monthly 128(9), 825–831. doi:10.1080/00029890.2021.1965068
López de Prado, M. (2018). Advances in Financial Machine Learning, ch. 11–14 (backtesting). Wiley. ISBN 978-1-119-48208-6
Harvey, C. R., Liu, Y. & Zhu, H. (2016). "…and the Cross-Section of Expected Returns." Review of Financial Studies 29(1), 5–68. doi:10.1093/rfs/hhv059
McLean, R. D. & Pontiff, J. (2016). "Does Academic Research Destroy Stock Return Predictability?" Journal of Finance 71(1), 5–32. doi:10.1111/jofi.12365
Wiecki, T., Campbell, A., Lent, J. & Stauth, J. (2016). "All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms." Journal of Investing 25(3), 69–80. doi:10.3905/joi.2016.25.3.069

← Prev: 1.2 · Sources of Return Next: 1.4 · Edge & Odds →

Educational material only — not investment, legal or tax advice. Backtests and simulations are hypothetical. Disclaimer · Text CC BY 4.0 · Code MIT.