Set Up Your Quant Lab

1.5.1 Reproducibility is a research requirement

Session 1.3 asked you to log every trial and pre-register every test. Both are worthless if running the same trial twice can give two different numbers — so before any research, you build a workbench where it can't.

Scientists separate two ideas that everyday speech blurs. A result is reproducible when the same input data, code and conditions of analysis give consistent results; it is replicable when a new study, with its own data, reaches the same conclusion (National Academies, 2019). Most of this course is about replicability — whether an edge survives data it has never seen, which is what out-of-sample testing and all of Unit 6 are for. Reproducibility is the humbler precondition. It is an engineering problem, so it is yours to solve on day one.

Think of any research number as the output of a function:

Pin all four inputs and $f$ behaves like a pure function: anyone, on any machine, next year, gets the number you logged. Leave one loose and your "result" is a random variable wearing a number's clothes.

Why is this a research requirement rather than tidiness? Because every tool of Session 1.3 assumes it. A research log entry says "variant 7: Sharpe 1.12"; if rerunning variant 7 gives 0.94, which number was the trial? A pre-registration commits you to one test; if a library upgrade quietly changes how that test computes, you ran a test you never registered. And when a live strategy drifts from its backtest (Unit 15), the first debugging step is to reproduce the backtest's number exactly — you cannot bisect the gap between two numbers when one of them won't sit still.

The evidence says reproducibility fails by default. When Pimentel and colleagues attempted 863,878 executions of Jupyter notebooks from GitHub, only 24.11% ran without errors and only 4.03% produced the same results as the original. Published research does little better: working from the authors' own data and code, Chang and Li reproduced the key conclusions of 22 of 67 macroeconomics papers from 13 well-regarded journals (33%), and 29 of 59 even with the authors' help (49%; the 59 exclude six papers that used confidential data and two that needed software Chang and Li did not have).

1.5.2 uv, Python 3.12 & the lockfile

One tool installs Python, creates the project's virtual environment, resolves every dependency into a lockfile and runs your code inside it — fast enough that you stop avoiding it.

uv, from Astral, replaces the pile of pyenv, venv, pip and pip-tools that Python projects used to need. Three ideas do the work. A virtual environment is a folder of installed packages that belongs to one project (.venv/), so upgrading pandas for one project cannot break another. A lockfile (uv.lock) records the exact version and hash of every package — including the dependencies of your dependencies — for every operating system at once. And uv run executes a command inside the environment after checking that it still matches the lock.

Install uv, then the course

# macOS / Linux
$ curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
PS> powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# or a package manager: brew install uv · winget install --id=astral-sh.uv -e · pipx install uv

$ git clone https://github.com/georgevrs/quant-notebook
$ cd quant-notebook
$ uv sync                                        # Python 3.12 + core + dev tools, exactly as locked
$ uv run python labs/u01/s05_quant_lab_setup.py   # this session's lab

You don't need Python installed first. The one-line file .python-version says 3.12, and if no matching interpreter is on your machine, uv downloads a managed one the first time it needs it. You never activate the environment by hand either: prefix commands with uv run. Update uv itself with uv self update (standalone installer) or your package manager.

This repository's pyproject.toml asks for Python ≥ 3.12 and the core research stack — numpy, pandas 3, polars, pyarrow, DuckDB, scipy, statsmodels, arch, scikit-learn and matplotlib — plus a dev dependency group (pytest, jupytext, ipykernel, ruff and friends) that uv installs by default. Heavier tools sit in optional extras that you add when a unit needs them:

1.5.3 The daily loop: notebooks, scripts & Colab

Every lab is a Python file that opens as a notebook. That one trick keeps Jupyter's exploratory feel and gives you code you can diff, review and run from a terminal.

The loop is short on purpose. Sync, run the lab top to bottom, change one thing, rerun and compare. Each lab writes every number its page quotes to a results file — this one to labs/u01/out/s05.results.json — so git diff shows exactly which numbers your change moved. If a number moved that you didn't mean to move, stop: one of the four layers is loose.

Why the .py file is the source of truth

A Jupyter notebook (.ipynb) is a JSON document holding your code and its outputs, execution counters and base64-encoded charts. Change one line and git diff shows a screenful of noise; merge two branches and you get conflicts inside JSON. Jupytext stores a notebook as a plain script instead. In the percent format every lab here uses, # %% starts a code cell and # %% [markdown] starts a text cell:

# %% [markdown]
# ## 2 · Same seed, same numbers

# %%
SEED = 42

a = np.random.default_rng(SEED).standard_normal(3)
b = np.random.default_rng(SEED).standard_normal(3)
c = np.random.default_rng(SEED + 1).standard_normal(3)

The course's build regenerates each .ipynb from its .py with the outputs stripped, so the notebook is a build artifact — like a compiled binary, never edited by hand. Your research repository can do the same, or skip the .ipynb altogether: VS Code shows Run Cell above every # %% in a .py file, and JupyterLab with jupytext installed opens a .py as a notebook (right-click → Open With → Notebook).

1.5.4 A research repository

The course repository is laid out to teach a hundred labs. Your own research needs a layout that keeps raw data sacred, results traceable and secrets out of git.

Read the tree from the data's point of view. Downloads land in data/raw/ and are never edited — if a vendor file is wrong, you fix it in code, so the fix is visible and repeatable. Code in src/ turns raw into data/processed/, which you can delete and rebuild at any time. Notebooks in research/ import from src/ instead of copy-pasting functions between notebooks, and anything imported gets a test in tests/. Pre-registrations are committed before the test they describe, so git's timestamp proves the order (Session 1.3). And results/ holds small committed summaries, each with its run manifest — the commit, package versions, data fingerprint and seed that produced it (§1.5.6), while bulky outputs stay out of git. The widely used Cookiecutter Data Science template makes the same raw/processed split; preregistrations/ is this course's addition. uv init my-research gives you the skeleton — pyproject.toml, .python-version, a src/ package, a .gitignore and a fresh git repository — and you add the rest.

Git hygiene: never commit data, secrets or bulk

Three things stay out of git. Data, because most market-data licences forbid redistribution and a public repository is redistribution (Session 4.1 — The Data Landscape, Vendors & Licensing), and because git is built for text, not gigabytes. Secrets, because git never forgets. Bulk outputs — large result files, experiment-tracker folders — because they bury the diffs that matter. A starting .gitignore:

# environment: rebuilt by uv sync
.venv/
# data: re-download or rebuild, never commit
data/
# secrets: commit .env.example (variable names only), never .env (the values)
.env
# notebook and run noise
.ipynb_checkpoints/
mlruns/
results/**/*.parquet

API keys live in environment variables — typed into your shell profile or a git-ignored .env file — and code reads them at run time. The course does this for SEC EDGAR, whose API requires a descriptive User-Agent:

import os

user_agent = os.environ["SEC_USER_AGENT"]   # set in your shell or .env, never typed into a notebook

1.5.5 Seeds that actually reproduce

A seed makes randomness repeatable. It is also a trial in disguise, and a shared random stream leaks — two facts that catch out experienced ML engineers.

A pseudo-random generator is a deterministic function of its random seed. In NumPy you create a Generator from a seed and hand it to everything that draws. Every lab starts the same way:

lab = qn.Lab("1.5")  # seeds the random generators: every run gives the same numbers
rng = lab.rng

Two generators seeded with 42 both begin 0.3047, −1.0400, 0.7505; seed 43 begins 0.2442 and never looks back. The lab also hashes 100,000 seeded integers into a fingerprint with qn.repro.fingerprint, 5dad31ce62bf5227. Run it on your machine and compare: integer generation involves no floating-point arithmetic, so it should match exactly.

That promise has fine print. NumPy guarantees the same stream only for the same NumPy build on the same machine, and it allows Generator methods to change their streams in feature releases (a new X.Y version). So the randomness layer rests on the environment layer: pinning NumPy in uv.lock is part of what makes seed=42 mean anything.

A seed is a trial

Run the same strategy — true Sharpe ratio 0.39, 5 years of daily data, zero risk-free rate for simplicity — under 20 seeds, and its measured Sharpe ranges from −0.78 to 0.98. That is not a bug. It is estimation noise of about $1/\sqrt{Y}$ = 0.45, the rule from Session 1.3. But it means a seed is a trial: rerun with seed after seed until the backtest looks good and you have run a best-of-20 search and reported the winner — seed 3, at 2.5× the true Sharpe. Picard (2021) found the same in deep learning: scanning up to 10,000 seeds on CIFAR-10, "it is surprisingly easy to find an outlier" that performs much better or much worse than average. The rule: fix the seed before you look. To learn how much the seed matters, run many and report the distribution — never the best one.

Shared streams leak

Seeding is necessary, not sufficient. In the lab, one pipeline builds five years of synthetic returns, then bootstraps a lower confidence bound for their mean (2,000 resamples). With one shared generator, the first run (empty cache) builds the data and bootstraps: the bound is −0.24%. The second run finds the data in the cache and skips the build — so the bootstrap starts at draw zero instead of after the data's draws, and reports −0.32%. Same seed, same data, different answer. Give each component its own child stream with spawn and both runs report −8.10% — a different figure from the shared version because the data now come from their own stream; what matters is that it no longer depends on the cache.

def run_pipeline(cache: dict, streams: str) -> float:
    """One run of data → bootstrap. `streams` is 'shared' (one generator) or 'spawned' (two)."""
    if streams == "shared":
        data_gen = boot_gen = np.random.default_rng(BOOT_SEED)
    else:
        data_gen, boot_gen = np.random.default_rng(BOOT_SEED).spawn(2)
    if "returns" not in cache:                  # cache miss: build (this consumes draws)
        cache["returns"] = build_returns(data_gen)
    return bootstrap_low(cache["returns"], boot_gen)

The legacy global generator, np.random.seed(0), is the same leak with no way to plug it: any library that draws from it shifts your numbers, invisibly. qn.Lab still seeds it as a safety net for old libraries; your own code should never draw from it.

1.5.6 The data cache & a first query

Download once, fingerprint, reuse — then ask the data questions in whichever language suits the question: pandas, SQL or a lazy dataframe.

qn.data_cache() returns a git-ignored folder: data/cache/ inside the repository, ./quantnb_cache anywhere else (Colab), or wherever the QUANTNB_CACHE environment variable points. Labs that need real data fetch it into this folder once, and only when qn.offline() is false. The course's CI sets QUANTNB_OFFLINE=1, so every lab must also run from synthetic data. This lab practises the pattern on a synthetic panel: 50 assets × 10 years of daily returns — 126,000 rows — with volatilities known to run from 15% to 45%.

def load_or_build(params: dict, gen: np.random.Generator) -> tuple[pd.DataFrame, str, Path]:
    """Read the panel from the cache if it is there and intact; otherwise build and write it."""
    CACHE_DIR.mkdir(parents=True, exist_ok=True)
    data_path = CACHE_DIR / f"panel-{params_key(params)}.parquet"
    meta_path = data_path.with_suffix(".json")
    if data_path.exists() and meta_path.exists():
        df = pd.read_parquet(data_path)
        if json.loads(meta_path.read_text(encoding="utf-8"))["fingerprint"] == content_fingerprint(df):
            return df, "hit", data_path
        print("cache entry failed its fingerprint check: rebuilding")
    df = build_panel(params, gen)
    df.to_parquet(data_path, index=False)
    meta = {"params": params, "fingerprint": content_fingerprint(df), "rows": len(df),
            "built_utc": datetime.now(UTC).isoformat(timespec="seconds")}
    meta_path.write_text(json.dumps(meta, indent=2), encoding="utf-8")
    return df, "miss", data_path

Three rules make a data cache safe, and each one closes a hole you have already met. Name files by everything that decides their content: the file name carries a hash of the parameters, the seed and random stream that drew the data, and the versions of the code that drew it (NumPy may change its streams between feature releases), so changing any of them can never silently reuse stale data. Fingerprint the content, not the file: the lab writes the same panel with two compression codecs and gets two files with different bytes but one content fingerprint — a pyarrow upgrade would change the bytes too, since writers stamp their version into the file. Give the builder its own random stream (§1.5.5), so a hit and a miss leave every later number untouched. Run the lab twice: the first call is a miss, the second a hit, and results.json comes out identical.

A first query, three ways

Now ask the panel a question — annualised mean and volatility per asset — in three dialects. DuckDB runs SQL directly on the Parquet file, with no server and no load step. polars builds a lazy query plan and, when you call collect(), reads only the two columns the plan needs (lazy evaluation). pandas reads the file and groups in memory.

SQL_STATS = """
SELECT ticker,
       count(*)                        AS n_days,
       avg(ret) * 252                  AS ann_mean,
       stddev_samp(ret) * sqrt(252)    AS ann_vol
FROM read_parquet(?)
GROUP BY ticker
ORDER BY ticker
"""


def stats_polars(path: Path) -> pd.DataFrame:
    query = (pl.scan_parquet(path)                         # lazy: nothing is read yet
               .group_by("ticker")
               .agg(n_days=pl.len(),
                    ann_mean=pl.col("ret").mean() * ANN,
                    ann_vol=pl.col("ret").std() * math.sqrt(ANN))
               .sort("ticker"))
    return query.collect().to_pandas()

The lab asserts that pandas, DuckDB and polars agree to within 1e-12 and asks for nothing more. On this panel polars happens to match pandas exactly, while DuckDB's means differ from pandas' in the last digit or two: engines may sum in a different order or with a different algorithm, and floating-point addition is not associative. Summing the lab's million synthetic returns forwards and then backwards gives two answers 6.7e-14 apart. That is why results are compared with a tolerance, here and in the course's CI, which reruns labs from a fresh checkout and checks every recorded number against the committed one to a relative 1e-6.

Synthetic data also lets you grade the answers against the truth. Ten years of daily data pin every asset's volatility to within 2.0 percentage points; the same ten years miss the true 7% mean return by as much as 40 percentage points. The mean is the number you know least well — Session 2.3 returns to this.

1.5.7 Where reproducibility breaks

Nobody decides to be irreproducible. It happens one convenient shortcut at a time — and every shortcut loosens one of the four layers.

1.5.8 Try it

Open the companion lab and prove that your workbench reproduces — then break it on purpose, so you know what the failure looks like.

1.5.9 Glossary · Check yourself · Go deeper

Check yourself

📚 Go deeper

1. National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility and Replicability in Science. The National Academies Press. doi:10.17226/25303
2. Sandve, G. K., Nekrutenko, A., Taylor, J. & Hovig, E. (2013). "Ten Simple Rules for Reproducible Computational Research." PLoS Computational Biology 9(10), e1003285. doi:10.1371/journal.pcbi.1003285
3. Pimentel, J. F., Murta, L., Braganholo, V. & Freire, J. (2019). "A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks." Proceedings of the 16th International Conference on Mining Software Repositories (MSR), 507–517. doi:10.1109/MSR.2019.00077
4. Chang, A. C. & Li, P. (2022). "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say 'Often Not'." Critical Finance Review 11(1), 185–206. doi:10.1561/104.00000053
5. Picard, D. (2021). "Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision." arXiv:2109.08203
6. Raasveldt, M. & Mühleisen, H. (2019). "DuckDB: an Embeddable Analytical Database." Proceedings of the 2019 International Conference on Management of Data (SIGMOD), 1981–1984. doi:10.1145/3299869.3320212
7. Astral. uv documentation — installation, projects, locking and syncing. https://docs.astral.sh/uv/
8. NumPy Developers. "Compatibility policy" (numpy.random). NumPy documentation. https://numpy.org/doc/stable/reference/random/compatibility.html
9. Jupytext contributors. Jupytext documentation — text notebooks and the percent format. https://jupytext.readthedocs.io/
10. Wiggins, A. (2011, updated 2017). "The Twelve-Factor App — III. Config." https://12factor.net/config
11. GitGuardian (2026). "The State of Secrets Sprawl 2026." https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/
12. DrivenData. Cookiecutter Data Science (v2) — a standard project structure for data science. https://cookiecutter-data-science.drivendata.org/