Trading with AI: why in my system the backtest decides and the LLM only vetoes

When someone asks me "does the LLM decide when to buy or sell?", my answer is always the same: no. And that design decision is, for me, what separates a reliable trading system from an experiment that is hard to audit. I'm Esteban Burgos, Forward Deployed Engineer, and in this post I share how I designed Crypto Trader, my multi-agent system for trading BTC and ETH on Binance, with sandbox, testnet, and live modes. I won't be publishing balances, positions, or trading results —that information doesn't leave my environment— but I can share the design, which may be useful to other teams evaluating putting AI into systems that make economic decisions.

The problem with letting an LLM "decide"

LLMs are very good at reasoning over ambiguous context, synthesizing scattered information, and detecting patterns that a rigid rule didn't anticipate. But a language model doesn't have, by design, guarantees of determinism, nor a verifiable track record of how well that specific decision performed in the past. When real money is at stake, that lack of traceability is a risk I decided not to accept as the primary source of decision-making.

That's why in Crypto Trader I inverted the usual order: the decision to trade (enter, exit, adjust size) originates from rules calibrated through backtesting. The LLM comes in afterward, with a bounded role: it can veto a trade if it detects a risk signal that the rule didn't capture, but it can't invent a trade that the backtest didn't authorize.

Backtest-calibrated rules, the source of the decision

The heart of the system is a backtest engine of approximately 6,900 lines that runs on real candles, not on synthetic data or idealized simulations. Every rule that ends up trading on testnet or live went through that engine first, which measures its historical behavior with market data that actually occurred.

I developed this following Spec-Driven Development (SDD): I documented 12 specs that define the expected behavior of each component of the engine and the agent system, and throughout the process I logged 73 fixes derived from those specs. Each fix was tied to a specific spec, which means that when something is corrected, there's evidence of why it was corrected and what behavior was expected instead. That traceability lets me say, with evidence, that a rule is calibrated and not simply "looks good on the chart."

The practical consequence is that the LLM is never the source of the entry or exit signal. The signal comes from a rule that has already shown, on real data, measurable behavior. The LLM operates on that signal, it doesn't replace it.

Eight agent roles, one model per score

Crypto Trader isn't a single giant prompt disguised as "trading AI." It's a system with eight agent roles, each with a bounded responsibility: market analysis, risk assessment, signal validation against the backtest, verification of news context or relevant events, exposure and position size control, compliance auditing of the rules, execution monitoring on Binance, and the final veto role before an order goes out to sandbox, testnet, or live.

Separating responsibilities into distinct roles has a concrete advantage: each agent can be evaluated independently. And here's something I consider central to the design: the model that executes each role is chosen by score, not by preference or trend. Each role has its own evaluation criteria —accuracy on its specific task, output consistency, cost, latency depending on how critical the role is— and the model that scores best on that criteria is the one assigned to that function. A compliance auditing role doesn't need the same profile as a market context analysis role, and forcing a single model for all eight tasks is, in my experience, a simplification that ends up costing decision quality.

This role-based architecture with models chosen by score also gives me flexibility going forward: if a model appears that scores better on the criteria for a specific role, I replace it in that role without touching the rest of the system. The multi-agent design isn't aesthetic, it's the way I keep the system evaluable component by component.

Risk gates and testnet verification before production

No rule change or model adjustment ever gets to trade with real money without first passing through explicit risk gates and testnet verification. The flow I follow is always the same: sandbox to validate logic without real exposure, Binance testnet to validate execution against a real market environment but without capital at risk, and only afterward, if the risk gates are satisfied, the move to live mode.

The risk gates are exactly where the LLM exercises its veto power: if a risk agent detects a condition outside the expected parameters —even if the backtest-calibrated rule indicates to trade— the operation is halted. The LLM doesn't need to "invent" an alternative or propose a different trade; its function is exclusively negative: block when something doesn't add up. That asymmetry —the rule proposes, the LLM can only veto, never add— is, for me, the key to incorporating a language model's judgment without inheriting its lack of determinism in the final decision.

Why it may matter for fintech teams

For a team evaluating putting AI agents into a system that moves money, a useful question may be not so much "how smart is the model," but "who has the authority to decide, and what evidence got them there." In my case, the authority rests with a rule validated against a backtest engine of about 6,900 lines running on real candles, documented with SDD across 12 specs and 73 traceable fixes. The AI adds judgment across eight specialized roles, each with its model chosen by score, and its only lever over the final operation is the veto, exercised through risk gates verified first in sandbox, then in testnet, and only then in production.

What I took away

• Defining who has the authority to decide (here, the backtest-validated rule) is a design decision that comes before choosing any model.
• An AI with veto power, rather than proposal power, adds judgment without inheriting a lack of determinism in the final decision.
• Splitting the system into bounded roles makes it possible to evaluate and replace each model by score without touching the rest.
• Tying every fix to a spec leaves evidence of why something changed and what behavior was expected.
• Sandbox, testnet, and risk gates before production are a path I'm not willing to skip.

Closing

This is a design I'm still learning from and adjusting. If you're working on something similar, or your team is evaluating how to incorporate AI into critical decisions, I'd be glad to exchange experiences: feel free to reach out on LinkedIn. I also describe how I apply this approach in projects on my website.