Python backtesting: my case against pre-built libraries

That is the central problem with most Python backtesting libraries. They make strategy research look finished before the difficult parts have even started. A few lines generate signals. A portfolio appears. The equity curve rises. Then the same logic reaches a live execution environment and discovers that candles do not fill orders, indicators do not know about latency, and a theoretical position size is not the same thing as a tradable position.

I am not against reusable software. I am against confusing convenience with market realism.

A pre-built Python backtesting library gives you an abstraction layer. That layer can save weeks of repetitive engineering. It can also hide the exact assumptions that determine whether your result is an edge or a spreadsheet artifact. When the strategy is simple and the data is clean, this trade-off is acceptable. When the model depends on intrabar sequencing, multiple data frequencies, partial fills, machine learning features, or execution constraints, the abstraction becomes a liability.

The question is not whether a library can produce a backtest. Almost all of them can.

The question is whether you can explain every fill, every timestamp, every fee, every position transition, and every source of look-ahead bias.

The hidden cost of abstraction in standard frameworks

A standard Python backtesting library usually imposes a worldview on the problem.

Backtrader, for example, is built around a Python-only, event-driven architecture. That model is intuitive. A new bar arrives, indicators update, the strategy makes a decision, and the broker simulation processes the order. It resembles the way many people mentally describe trading.

Markets do not operate according to the mental model of a clean bar-by-bar loop.

Within one five-minute candle, price may move through the stop, the limit, and the eventual close. The data may not tell you which event happened first. A market order may be submitted at the close but filled at the next available quote. A limit order may touch the displayed price without receiving a fill. A strategy can calculate a signal on one timestamp while accidentally using a feature that was finalized several seconds later.

The framework has to choose an interpretation. If you do not choose it explicitly, the library chooses for you.

That is the first hidden cost.

The second is that many libraries represent a trading system through a rigid set of objects: strategy, broker, data feed, indicator, analyzer, and order. These abstractions are useful until the system you are modeling no longer fits them naturally. Multi-asset portfolios, asynchronous feeds, order-book data, dynamic transaction costs, exchange-specific rules, and online machine learning pipelines all create friction.

The friction often appears as custom callbacks. Then custom broker logic. Then patched data feeds. Then special cases inside order handling. At some point, the framework is no longer reducing complexity. It is becoming another system you must debug.

An abstraction is useful only while it preserves the information your execution model depends on.

This is why I treat a library-generated equity curve as an output, not as evidence. Evidence begins with the execution assumptions.

A basic framework may silently assume:

• Every bar has enough liquidity to fill the requested quantity.
• A stop order is filled at the stop price.
• A limit order is filled whenever the market touches the limit.
• Fees and slippage are static.
• Orders are processed in a fixed sequence that matches reality.
• The portfolio can rebalance without market impact.
• All data feeds are synchronized cleanly.
• The strategy can act at a price that was not available when the signal existed.

None of these assumptions is universally valid. Some are useful simplifications. The problem is that they can remain invisible inside a friendly API.

That invisibility has a measurable cost in research quality. It increases model risk while making the code look less complicated than the trading problem actually is.

Event-driven versus array-based architectures

Performance is where the debate usually becomes simplistic.

People say event-driven engines are slow and vectorized engines are fast. That is directionally correct, but incomplete. The architecture determines not only runtime. It determines what kind of question the engine can answer.

Backtrader processes strategy logic through Python events. This is straightforward for sequential rules: moving-average crossovers, threshold signals, scheduled rebalances, and simple stop logic. But the same design can experience single-threaded execution bottlenecks, slow optimization cycles, and high memory usage on large historical datasets.

The bottleneck is not necessarily the trading rule. It is the number of Python-level operations required to replay the data and invoke the strategy logic.

VectorBT takes a different route. It relies on array-based operations powered by NumPy, Pandas, and Numba compilation. Instead of iterating through every hypothetical strategy variation as a conventional Python loop, it can evaluate large parameter grids through vectorized computation. In suitable workloads, that means testing more than 1,000 parameter variations without turning the research process into a long-running serial simulation.

That is a major advantage for exploratory research.

It is not a universal replacement for event simulation.

Array-based methods are efficient when the problem can be expressed as operations over matrices and time series. They are less natural when every order changes the state of the next decision. Position-dependent sizing, queue priority, partial fills, bracket orders, portfolio margin, and path-dependent execution rules are not impossible in a vectorized system. They are simply more difficult to express without compromising clarity.

The practical distinction looks like this:

The table is not a ranking. It is a warning against using one architecture for every task.

For signal discovery, vectorized computation is often the better tool. For execution research, a stateful event loop is often easier to audit. For a serious system, I usually want both capabilities available, even if they live in separate layers.

The mistake is treating a fast research engine as a realistic broker simulator. Speed does not repair bad fill logic.

Why custom Python backtesters become necessary

A custom Python backtester is not automatically superior. It is superior when the default assumptions of a pre-built library are more expensive than the engineering required to replace them.

That threshold arrives earlier than many retail developers expect.

Consider a strategy that trades liquid futures on hourly bars. A conventional library may be adequate. The signal is calculated at the bar close, the next bar provides the execution window, and the cost model can be approximated without distorting the conclusion. If the strategy has a large sample size, low parameter sensitivity, and modest turnover, the abstraction may be entirely reasonable.

Now change the problem:

• The model consumes five-minute features and executes on one-minute data.
• The position size depends on current portfolio volatility.
• Orders can be partially filled.
• Stops and targets compete inside the same candle.
• Transaction costs vary with volatility and liquidity.
• A machine learning model is retrained on a rolling window.
• Multiple assets update at different timestamps.
• The portfolio has exposure, margin, and concentration limits.
• The research result must be reproduced by the live execution service.

This is not a generic indicator backtest anymore. It is an execution model.

The custom engine gives you control over the components that matter:

1. Data alignment.

You decide which timestamp represents feature availability, signal generation, order submission, and fill confirmation. This is where many forms of look-ahead bias begin.

2. Order matching.

You define whether a market order fills at the next quote, the next trade, the bar open, or a modeled price with slippage. You can represent partial fills instead of treating liquidity as infinite.

3. Position accounting.

You control average entry price, realized and unrealized P&L, commissions, funding, borrow, margin, and forced liquidation logic.

4. Portfolio state.

You can make exposure limits, cash constraints, volatility targeting, and cross-asset interactions first-class objects rather than afterthoughts.

5. Execution pipelines.

The same order and risk objects can be connected to paper trading or live execution with less conceptual drift between research and deployment.

6. Diagnostics.

You can log why an order was rejected, delayed, resized, or filled at a different price. A single aggregate return figure cannot do this.

The cost is engineering time. A production-grade custom crypto backtester has been estimated at roughly 53 to 95 hours across data pipelines, order matching, position sizing, performance metrics, visualizations, and bug fixes. At a development rate of $75 per hour, that corresponds to approximately $3,975 to $7,125 in development cost.

That estimate is not a universal quote. It is a useful correction to the fantasy that a custom engine is either a weekend script or an institutional-scale project. A functional engine is achievable. A trustworthy engine is where the work accumulates.

The first version usually handles the happy path. The second version handles everything that invalidates the first result.

The code is not the hard part

Most developers can write a loop that enters when RSI crosses a threshold. That is not a backtesting engine.

The difficult questions are operational:

• What happens if two orders are triggered at the same timestamp?
• Does a stop execute before a target when both are inside the bar range?
• Can the strategy reuse capital before the prior fill is confirmed?
• How are gaps handled?
• What happens when the requested size exceeds available volume?
• Does a feature calculated from daily data become available at midnight, session close, or the next trading session?
• Are delisted assets retained in the dataset?
• Are corporate actions applied before or after signal construction?
• How are missing bars distinguished from zero-volume bars?
• Can a prediction generated at time \(t\) access a label or normalization statistic derived from \(t+1\)?

These are not cosmetic details. They change the distribution of returns and drawdown.

A custom engine forces the assumptions into code. That is uncomfortable, but productive. Ambiguity is a hidden variable. Explicit rules make it testable.

Bridging research and production execution

A backtest fails most often at the boundary between research and execution.

Research code tends to operate on complete datasets. Live systems operate on incomplete information arriving at irregular times. The distinction is structural.

In a historical dataframe, the high and low of a candle are already known. At runtime, they are not. In a backtest, a feature may appear neatly aligned with the target timestamp. In production, the upstream process may publish that feature after a delay. In research, a market order can be assigned a clean next-bar price. In production, the order encounters spread, queue position, latency, rejected quantities, and venue rules.

This is why a technically sophisticated signal can still produce a weak live system. The predictive model may be fine. The execution translation is not.

I separate the problem into four layers:

1. Signal layer

This layer answers whether the market state meets the model’s entry or exit condition. It should not pretend to know how the order will be filled.

A signal might be generated from trend, volatility, mean reversion, order flow, or machine learning output. The signal layer produces intent, not a guaranteed transaction.

2. Portfolio layer

The portfolio layer determines what can be traded. It converts model intent into target exposure while respecting capital, leverage, volatility, correlation, and concentration limits.

This is where a strategy that appears profitable at the asset level often collapses. The individual trades may have an edge, but the portfolio cannot hold them in the required size without creating unacceptable drawdown or turnover.

3. Execution layer

This layer turns target exposure into orders. It models order type, urgency, slicing, slippage, fees, latency, and fill probability.

If the backtester does not contain an execution layer, it is not measuring execution. It is measuring a price transformation.

4. Accounting and diagnostics layer

This layer tracks cash, margin, realized P&L, unrealized P&L, fees, financing, exposure, and risk statistics. It should also preserve an event log.

The event log matters more than most dashboards. When the equity curve changes unexpectedly, I want to inspect the sequence that produced the change. Which order was submitted? What data was available? What quantity was requested? What quantity filled? Which cost was applied?

If the system cannot answer those questions, debugging becomes guesswork.

That is also why an automated strategy deserves an execution audit before anyone argues about indicator selection. A systematic trading platform audit should examine order handling, broker connectivity, risk controls, and operational failure modes—not just whether the backtest chart looks profitable.

The production test is not whether the signal survives live prices. It is whether the entire state machine survives incomplete information.

Machine learning increases the burden.

Many pre-built Python backtesting libraries were designed around deterministic indicators and rule-based strategies. They can be extended for machine learning, but the integration is rarely frictionless. Feature generation, rolling retraining, prediction latency, label construction, model versioning, and leakage controls all need explicit handling.

Specialized research platforms such as Qlib may provide stronger support for machine learning workflows and multi-factor modeling. That does not mean a specialized platform solves execution realism. It means the research problem and the trading problem should not be forced into the same abstraction.

A model pipeline needs its own validation:

• Features must be timestamped by availability, not by the period they describe.
• Scaling and normalization must be fitted only on information available at the time.
• Retraining windows must be defined before the test begins.
• Hyperparameter selection must be separated from the final evaluation sample.
• Predictions should carry a model version and generation timestamp.
• Missing predictions must not be silently converted into neutral signals.
• The backtest must measure turnover and capacity, not just directional accuracy.

Accuracy is a weak metric when the strategy is exposed to costs and drawdown. A model can predict direction slightly better than chance and still lose money through turnover. A model with stronger accuracy can fail because its edge disappears during volatility regime changes.

The relevant output is not a prediction score in isolation. It is the distribution of net returns after execution constraints.

Build your own engine or use a compiled tool?

The choice is not binary.

The real decision is whether you need custom semantics or only faster computation.

If your strategy is a conventional daily or hourly system with market-on-open execution, a small number of assets, simple position sizing, and no complex intrabar assumptions, a pre-built library is often the correct engineering decision. You can spend the saved time on robustness checks, alternative datasets, and out-of-sample validation.

If you need large parameter sweeps over clean signals, VectorBT is a strong option because array-based operations and Numba compilation can accelerate research substantially. Its ability to evaluate broad parameter grids is valuable when the objective is to map sensitivity rather than to simulate every exchange event.

But do not confuse parameter throughput with statistical validity. Testing 1,000 variations quickly does not give you 1,000 independent discoveries. It gives you a larger multiple-testing problem.

A strategy that survives a broad search only after selecting the best equity curve may be overfit. The correct response is not to stop using fast tools. It is to record the search procedure, reserve untouched data, test neighboring parameters, and examine whether the edge persists across regimes and cost assumptions.

A custom engine becomes more defensible when at least one of these conditions is true:

• The strategy depends on intrabar ordering or tick-level execution.
• Order state and partial fills materially affect the result.
• The portfolio contains multiple interacting assets or venues.
• Position sizing is dynamic and path-dependent.
• Transaction costs cannot be represented by a fixed percentage.
• The model uses rolling machine learning retraining.
• Live execution must share logic with the research environment.
• The existing framework requires more patches than the system would require from a clean design.

There is a third route: use a specialized framework with a more realistic event model instead of writing every component yourself. Modern engines such as NautilusTrader demonstrate that pre-built software is not automatically unsuitable for production. The issue is fit. A framework built for institutional deployment may be a better foundation than a lightweight research library, but it still requires audits, tests, and a clear understanding of its fill semantics.

I would make the decision through a small capability matrix rather than a loyalty contest between libraries:

The best engine is the one whose failure modes you understand.

How I would structure a custom backtester

I would not begin by recreating a full exchange. That is how projects become expensive without becoming more accurate.

I would begin with the smallest state machine that captures the strategy’s actual risk.

The minimum useful components are:

1. Immutable market data inputs.

Store raw data separately from cleaned and feature-enriched data. Never overwrite the evidence used to generate a result.

2. Timestamped feature availability.

Every feature should carry a clear rule for when it becomes observable. A daily close is not available at the start of the same trading day merely because the dataframe contains it.

3. Explicit order objects.

An order should include submission time, quantity, side, type, limit or stop price, time-in-force, and status transitions.

4. A fill model.

The model should state how prices, spread, slippage, volume, and partial execution are determined. If the data cannot support a realistic assumption, use a conservative one and label it.

5. Portfolio accounting.

Track cash, gross and net exposure, leverage, margin, fees, financing, realized P&L, and mark-to-market value.

6. Event logging.

Record every signal, order, fill, rejection, cancellation, and position change. The log is the audit trail.

7. Invariant tests.

Cash should reconcile. Position quantities should reconcile. A canceled order should not generate a fill. A fill should alter the position exactly once. These tests catch bugs before performance analysis begins.

8. Reference cases.

Use small synthetic datasets where the correct result can be calculated by hand. Test gaps, simultaneous stops and targets, missing bars, insufficient liquidity, and delayed data.

This design is less glamorous than adding another indicator. It is more valuable.

A custom Python backtester should also remain modular. The signal model should not know the internal details of the broker simulator. The execution model should not rewrite portfolio accounting. The data layer should not silently mutate timestamps to make a strategy look better.

That separation makes it possible to replace one component without invalidating the rest of the research.

It also makes failure visible. If the strategy only works when a specific fill assumption is enabled, that is not a nuisance. It is a result about model fragility.

The standard library is not the enemy

There is a recurring mistake in quantitative development: treating custom code as a mark of seriousness.

It is not.

A badly written custom backtester can be slower, less tested, and more misleading than a mature library. Plain Python loops are not automatically more realistic than vectorized operations. A hand-built engine can still contain look-ahead bias, survivorship bias, incorrect fee treatment, and impossible fills.

Custom does not mean correct.

Likewise, pre-built libraries are not inherently disposable. They are useful when their abstractions match the problem. Backtrader can be practical for event-driven strategy research. VectorBT can be highly effective for array-based analysis and parameter exploration. Specialized platforms can reduce the burden of integrating machine learning and multi-factor research. Production-oriented engines can provide infrastructure that would take far longer to reproduce internally.

The decision should be based on the behavior you need to model, not on the number of GitHub stars or the emotional appeal of owning every line of code.

I use three questions:

• Can I state the engine’s execution assumptions in plain language?
• Can I inspect the event sequence behind any material P&L result?
• Can the research logic survive the transition to live or paper execution without changing its meaning?

If the answer is no, the problem is not necessarily that the library is bad. The problem is that the abstraction is no longer aligned with the strategy.

My conclusion

Python backtesting libraries are excellent at reducing the cost of initial research. They are less reliable as universal representations of market execution.

That distinction matters.

Use a pre-built framework when it gives you a clean, auditable model with enough speed for the research question. Use vectorized and Numba-compiled tools when the workload is dominated by array operations and parameter exploration. Build a custom Python backtester when order state, data timing, portfolio constraints, or execution realism determine the validity of the result.

The engineering estimate of 53 to 95 hours is not a reason to avoid custom development. It is a reason to scope it properly. Build the minimum engine required to model the strategy’s real failure modes. Do not spend months recreating features that do not affect the decision.

And do not pay for speed with ambiguity.

A backtest is only as credible as the assumptions it exposes. If the engine hides the assumptions that create the edge, the edge has not been measured. It has been manufactured.