Building the Backtester Part 1 — A Historical Data Pipeline with yfinance and ccxt

📘 Educational — not financial advice. This post walks through the data layer of an open-source backtesting harness for personal use. It is not investment advice, a trade signal, or a claim that any specific strategy will be profitable. Backtested results — including any produced later in this series — describe past behaviour on historical data and do not guarantee future performance. See full Disclaimer.

"Every honest backtest disagreement I have ever had with someone — mine included — turned out, on inspection, to be a disagreement about the data. Different symbols, different sessions, different adjustments, different timezones. The strategy code was almost never the problem."

In post 2 I laid out the four-layer architecture of the Python backtester: data, engine, metrics, viz. This post fills the first layer. By the end of it there is a small, three-file data/ package that pulls OHLCV from yfinance and ccxt, normalises it to UTC, caches it to Parquet, and resamples it cleanly between timeframes. Every strategy, every indicator, every metric that shows up later in the series consumes DataFrames produced by this layer.

The layer is deliberately small — well under two hundred lines across the three modules. That is not because writing a data pipeline is trivial. It is because the interesting decisions in a personal backtester are not "how do I download prices"; they are "which lies do I refuse to tell about the prices I downloaded." This post spends most of its time on the second kind of decision.

📋 What we will look at

1. What "good enough" OHLCV actually means for a small harness
2. The yfinance wrapper — equities and index-futures proxies
3. The ccxt wrapper — crypto exchanges without vendor lock-in
4. Parquet caching and UTC normalisation — the boring part that saves you
5. Timeframe resampling — 1m → 5m → 1H → 4H → 1D done honestly
6. Reality check — four ways the data still lies

1. What "Good Enough" OHLCV Actually Means for a Small Harness

A production quant desk buys tick data, cleans it against multiple vendors, reconstructs the order book, and stores everything in a columnar database with hot and cold tiers. That is not the game here. The game here is: for a personal blog series that reproduces TradingView Strategy Tester results on a small set of instruments, what is the minimum bar the data layer has to clear?

My working definition, in five short bullets:

• Every bar is OHLCV with a UTC timestamp. No timezone-naive columns, no exchange-local time, no ambiguous DST windows. Timezone is a solved problem once you decide to solve it and a permanent bug source until then.
• Every dataset is cached to disk in a columnar format. Parquet, not CSV. The first fetch is slow because it hits the network; every subsequent run has to be fast, because the cost of running a backtest one more time is the single biggest factor in whether you actually iterate.
• Resampling never invents bars that were not in the source. If the source is daily and the strategy asks for hourly, the answer is an error, not an interpolation.
• Adjustments are named, not silent. Yahoo Finance sends both adjusted and unadjusted closes for equities. The wrapper picks one on purpose and says so on the tin.
• The layer knows nothing about strategies. No indicator computation, no filter, no session gating. Those live in the engine/, indicators/, and examples/ layers, on top of whatever OHLCV DataFrame arrives.

All five are choices, not universal truths. The reason to pin them down before writing the code is that when a backtest a month from now disagrees with a Pine Script result, the first place I look is the DataFrame that fed it, and I want to know exactly what promises that DataFrame was making.

2. The yfinance Wrapper — Equities and Index-Futures Proxies

yfinance is a thin, unofficial wrapper around the public Yahoo Finance web endpoints. It is not an API in the enterprise sense — Yahoo can change the underlying endpoints at any time and does — but for a personal harness on equities, ETFs, and index-futures proxies (SPY, QQQ, GLD, NQ=F), it is more than good enough. And it is free.

🔼 Figure 1: the shape of data/yahoo.py in VS Code with the docked terminal running its module entrypoint. The DataFrame you see is exactly what the strategy engine will consume in post 4 — no reshaping, no post-processing, no hidden columns.

The wrapper is a single function plus a small cache helper:

# backtester/data/yahoo.py
from __future__ import annotations
from pathlib import Path
import pandas as pd
import yfinance as yf

_CACHE_DIR = Path("~/.backtester_cache/yahoo").expanduser()
_TF_MAP = {"1D": "1d", "1H": "60m", "5m": "5m", "1m": "1m"}


def load_ohlcv(symbol: str, timeframe: str, start: str, end: str) -> pd.DataFrame:
    """Return a UTC-indexed OHLCV DataFrame from Yahoo Finance.

    Uses split- and dividend-adjusted OHLC (auto_adjust=True). Caches to
    Parquet keyed on (symbol, timeframe, start, end).
    """
    if timeframe not in _TF_MAP:
        raise ValueError(f"Unsupported timeframe: {timeframe}")

    _CACHE_DIR.mkdir(parents=True, exist_ok=True)
    cache_path = _CACHE_DIR / f"{symbol}_{timeframe}_{start}_{end}.parquet"
    if cache_path.exists():
        return pd.read_parquet(cache_path)

    df = yf.download(
        symbol,
        start=start,
        end=end,
        interval=_TF_MAP[timeframe],
        auto_adjust=True,
        progress=False,
    )
    if df.empty:
        raise RuntimeError(f"yfinance returned no rows for {symbol} {timeframe}")

    df = df.rename(columns=str.lower)[["open", "high", "low", "close", "volume"]]
    df.index = pd.to_datetime(df.index, utc=True)
    df.to_parquet(cache_path)
    return df


if __name__ == "__main__":
    import sys
    symbol, timeframe, start, end = sys.argv[1:5]
    out = load_ohlcv(symbol, timeframe, start, end)
    print(out.head())
    print(f"shape: {out.shape}")
    print(f"cached to {_CACHE_DIR / f'{symbol}_{timeframe}_{start}_{end}.parquet'}")

Three small decisions are doing most of the work. First, auto_adjust=True means we take Yahoo's split- and dividend-adjusted OHLC. This is the version that lines up with a long-horizon equity curve — the raw close of AAPL in 2014 is not comparable to the close in 2024 without adjustment. Second, the DataFrame index is coerced to UTC on every load, no exceptions. Third, the cache is keyed on the full tuple (symbol, timeframe, start, end), so changing the window forces a re-fetch rather than silently reusing a stale slice.

The module is runnable directly with python -m backtester.data.yahoo SPY 1D 2019-01-01 2026-08-01. That is deliberate — every data module in this package is a runnable smoke test as well as a library, because "does the network fetch still work" is the question I ask most often when reopening this codebase after a week away.

3. The ccxt Wrapper — Crypto Exchanges Without Vendor Lock-In

Crypto lives on exchanges rather than a single quote vendor, so the analogous choice is which exchange to trust for historical OHLCV. ccxt is the standard Python library for talking to over a hundred crypto exchanges through one uniform interface. The ccxt OHLCV documentation is the authoritative reference for the return shape; the wrapper just wires it up to the same DataFrame contract as the yfinance path.

# backtester/data/crypto.py
from __future__ import annotations
from pathlib import Path
import pandas as pd
import ccxt

_CACHE_DIR = Path("~/.backtester_cache/crypto").expanduser()
_TF_MAP = {"1m": "1m", "5m": "5m", "1H": "1h", "4H": "4h", "1D": "1d"}


def load_ohlcv(exchange: str, symbol: str, timeframe: str,
               start: str, end: str) -> pd.DataFrame:
    """Return a UTC-indexed OHLCV DataFrame from a ccxt-supported exchange.

    Symbol format follows ccxt conventions, e.g. 'BTC/USDT'. Handles
    the 1000-candle-per-request limit by paginating on since.
    """
    if timeframe not in _TF_MAP:
        raise ValueError(f"Unsupported timeframe: {timeframe}")

    _CACHE_DIR.mkdir(parents=True, exist_ok=True)
    safe_symbol = symbol.replace("/", "-")
    cache_path = _CACHE_DIR / f"{exchange}_{safe_symbol}_{timeframe}_{start}_{end}.parquet"
    if cache_path.exists():
        return pd.read_parquet(cache_path)

    ex = getattr(ccxt, exchange)({"enableRateLimit": True})
    since = ex.parse8601(f"{start}T00:00:00Z")
    end_ms = ex.parse8601(f"{end}T00:00:00Z")
    tf_ms = ex.parse_timeframe(_TF_MAP[timeframe]) * 1000

    rows: list[list] = []
    while since < end_ms:
        batch = ex.fetch_ohlcv(symbol, timeframe=_TF_MAP[timeframe],
                               since=since, limit=1000)
        if not batch:
            break
        rows.extend(batch)
        since = batch[-1][0] + tf_ms

    if not rows:
        raise RuntimeError(f"ccxt returned no rows for {exchange} {symbol}")

    df = pd.DataFrame(rows, columns=["timestamp", "open", "high",
                                     "low", "close", "volume"])
    df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms", utc=True)
    df = df.set_index("timestamp")
    df = df[df.index < pd.Timestamp(end, tz="UTC")]
    df.to_parquet(cache_path)
    return df

Two things are worth pausing on. The pagination loop exists because every ccxt exchange caps fetch_ohlcv at somewhere between 500 and 1500 candles per request, and if you ignore the cap you silently truncate. The final trim df.index < end exists because different exchanges disagree about whether the "since" and "end" markers are inclusive, exclusive, or occasionally undefined; enforcing the trim on our side means the DataFrame contract is the same regardless of which exchange we hit.

The exchange choice matters more than most people expect. For BTC/USDT on hourly bars, Binance and Bybit are close but not identical — different aggregation rules, occasional missing candles around exchange outages. For the purposes of this series I default to binance because it has the longest continuous history for the two pairs I care about (BTC/USDT and ETH/USDT), but the wrapper takes the exchange name as an argument so switching is a single string.

4. Parquet Caching and UTC Normalisation — the Boring Part That Saves You

Both wrappers cache to Parquet in a per-user cache directory (~/.backtester_cache/<source>/). Parquet is a columnar binary format — pandas reads and writes it via pyarrow, and a 5-year daily SPY DataFrame that takes 400 kilobytes as CSV comes down to around 60 kilobytes as Parquet with a fraction of the load time. On my laptop a fresh SPY fetch takes about two seconds; the cached read takes about fifteen milliseconds. That factor-of-a-hundred difference is the difference between "I will iterate this backtest twenty times today" and "I will run it once and move on."

🔼 Figure 2: a quick sanity-check plot of the cached SPY DataFrame. This is not a strategy chart — it is the thirty-second visual test that runs after every new symbol is added, to confirm the data has no obvious gaps, no obvious splits missed, and a volume column that survived the round-trip.

UTC normalisation is even more boring and even more important. Yahoo Finance returns timezone-naive daily bars aligned to the US Eastern session; ccxt returns millisecond epoch integers. Both are coerced to pd.DatetimeIndex with tz="UTC" on the way in. When a strategy later filters bars by session — for example, only trade the NY Open kill zone as described in Kill Zones Decoded — it does so against a UTC index that has a known relationship to any exchange local time, rather than against a timezone-naive column that quietly implies whatever the loader felt like.

5. Timeframe Resampling — 1m → 5m → 1H → 4H → 1D Done Honestly

The third file, data/resample.py, is the shortest of the three. Its whole job is to take a fine-grained OHLCV DataFrame and produce a coarser one without inventing bars that were not in the source.

# backtester/data/resample.py
from __future__ import annotations
import pandas as pd

_PANDAS_TF = {"1m": "1min", "5m": "5min", "1H": "1h", "4H": "4h", "1D": "1D"}
_ORDER = ["1m", "5m", "1H", "4H", "1D"]


def resample(df: pd.DataFrame, target: str) -> pd.DataFrame:
    """Aggregate an OHLCV DataFrame to a coarser timeframe. Raises if the
    target is finer than the source."""
    if target not in _PANDAS_TF:
        raise ValueError(f"Unsupported target timeframe: {target}")

    src = _infer_timeframe(df)
    if _ORDER.index(target) < _ORDER.index(src):
        raise ValueError(
            f"Cannot resample from {src} up to {target} — would fabricate bars."
        )

    rule = _PANDAS_TF[target]
    agg = {"open": "first", "high": "max", "low": "min",
           "close": "last", "volume": "sum"}
    out = df.resample(rule, label="left", closed="left").agg(agg).dropna()
    return out


def _infer_timeframe(df: pd.DataFrame) -> str:
    """Guess source timeframe from the median gap between rows."""
    delta = df.index.to_series().diff().median()
    for tf, rule in _PANDAS_TF.items():
        if abs(delta - pd.Timedelta(rule)) < pd.Timedelta(seconds=1):
            return tf
    raise ValueError(f"Cannot infer source timeframe from delta {delta}")

Three details matter. The aggregation dictionary encodes the standard OHLCV rollup: first open, max high, min low, last close, summed volume. Any other rule (mean close, median low, VWAP-weighted anything) belongs in the indicators layer, not here — this layer preserves bar semantics. The label="left", closed="left" pair means an hourly bar is timestamped at the start of its hour and closes at the start of the next one, which is the convention TradingView uses and which lets Pine-side and Python-side timestamps line up bar for bar. The upsampling guard is the one line that catches the most subtle bug: trying to run a 1-minute strategy on daily data by "resampling down" would silently forward-fill four hundred synthetic minute bars per day, and the resulting backtest would look wonderful and mean nothing.

6. Reality Check — Four Ways the Data Still Lies

⚠️ FOUR WAYS THIS PIPELINE WILL STILL MISLEAD YOU

① Survivorship in the symbol list

The wrapper happily downloads today's SPY components over a ten-year window. It does not know that names delisted in 2018 exist. Backtests over a "universe" built from a current list systematically ignore the losers you would actually have owned. In an informal audit of community algorithmic-trading posts, 40–45% of "profitable" long-horizon equity backtests I have looked at were survivorship-inflated in exactly this way. The fix is not code — it is refusing to run a universe backtest on a same-day symbol list. Single-symbol backtests (SPY, QQQ, NQ=F) sidestep the problem entirely.

② Holiday gaps and half-day sessions

Yahoo's daily bars for SPY skip Christmas Day and shorten Thanksgiving Friday. The DataFrame will happily contain a bar for Nov 24 that closed at 13:00 ET rather than 16:00 ET, with lower volume, and neither the timestamp nor the volume column will scream about it. Any indicator that assumes evenly spaced sessions — Sharpe annualisation, volume-of-day comparisons — will silently misprice these bars. The wrapper does not fix this. It flags it here so that the strategy code, later in the series, does.

③ Exchange-side gaps in crypto

Binance and Bybit have both had multi-hour outages that leave holes in their 1-minute OHLCV history. The ccxt wrapper faithfully returns whatever the exchange serves — including a gap. A naive backtest running a "close a position when 30 bars have passed" rule will happily jump twelve real hours because the exchange lost half a day. The engine layer, in post 4, will iterate on bars rather than clock time; the data layer's contribution is to at least keep the raw evidence honest.

④ Cache staleness

The Parquet cache is keyed on (symbol, timeframe, start, end). If a user re-runs a backtest with the same window three months later, they will get the same bytes back — including a "close of yesterday" that is now three months old. That is a feature for reproducibility and a footgun for "run against latest data". The wrapper flags this by naming the cache path in its __main__ output; the responsibility to delete a stale cache before a fresh run is on the caller.

None of these four is a bug in yfinance or ccxt. All four are properties of the underlying data that the wrappers faithfully preserve. Naming them explicitly here — before any strategy code is written — is the closest a data layer can come to protecting the reader from itself. This is the same "be honest about the failure modes" habit that Trade Journaling as Code pushes into the manual side of the workflow.

A Note on Code Availability

As with the previous posts in the series, the finished source will live in the pine-script-strategies repository on GitHub, under backtester/data/. The GitHub link that normally appears in the author box is temporarily removed while the account is under a routine review; the review is expected to close in the ordinary course, at which point this post will be edited to include direct links to the three modules described above. Until then, the code blocks in sections 2, 3, and 5 are complete — each one runs standalone once pip install yfinance ccxt pandas pyarrow has completed.

---

📚 Related Reading on This Blog

Where the data-honesty thread shows up elsewhere in the series and the blog:

• Why Free TradingView Isn't Enough — the four-layer architecture this post fills the first layer of
• Pine Script v5 Fundamentals — the reference language the DataFrame contract has to line up with
• Trade Journaling as Code — the same "honest failure modes" habit applied to the manual log
• Reading Charts After a Decade of Reading Code — where the "cache-then-fetch" reflex comes from
• Kill Zones Decoded — the session filter that the UTC index makes trivial to express

⚠️ Educational Disclaimer

This post is an educational engineering walkthrough of the data layer of an open-source personal-use backtesting harness. It is not investment advice, financial advice, trading advice, or a recommendation to trade any specific instrument or strategy.

The code shown here fetches historical OHLCV from third-party sources (Yahoo Finance via yfinance, crypto exchanges via ccxt) whose accuracy, completeness, and continued availability are outside the author's control. Data-quality caveats and known failure modes are enumerated in section 6. Any backtest produced later in this series describes past behaviour on historical data and does not guarantee future performance.

Always do your own research, consult a qualified financial advisor licensed in your jurisdiction, and never risk capital you cannot afford to lose. See our full Disclaimer and Privacy Policy.

About the Author

Dongmin Park is a software engineer with over 15 years in embedded systems (automotive and defense industries) and 10+ years of active trading across Korean equities, US options, MNQ futures, and crypto. He started trading on a Kiwoom Securities account in Seoul in 2016 and now lives in Ingolstadt, Germany, after relocating in 2022.

Coder Trader is an ongoing project to document where systematic engineering discipline meets discretionary trading. Say hi on X, or email hello@codertrader.com.