Home
/
Research
/
How to test your rules before your agent trades
Guides

How to test your rules before your agent trades

Before your AI agent trades real money, test its rules twice: once on past data, then live with virtual money. Here is how to do both, and the 3 traps that make tests look better than they are.

The Portfolio Lab Research Team

Updated 

November 7, 2026

contents

Before your AI agent trades real money, test its rules twice: once on past data, then live with virtual money. This guide explains both steps and the 3 traps that make tests look better than they are.

Key takeaways

  • Testing has 2 stages: a backtest on past prices, then a live test with virtual money.
  • A backtest shows how rules would have behaved in the past. It does not show the future.
  • Lookahead bias uses information that was not available at the time.
  • Overfitting tunes rules to past data so closely that they fail on new data.
  • Keep some data hidden while you build, and test on it only once.
  • A live test with virtual money checks your rules on prices nobody knew in advance.

Why test rules before your agent trades?

Testing tells you whether your rules do what you think before real money is at risk. An AI agent follows rules exactly. If the rules are flawed, the agent repeats the flaw on every run.

Testing cannot prove a rule will make money. It can catch rules that are broken, unclear, or built on luck.

Step 1: Backtest on past data

A backtest runs your rules over past prices to see how they would have behaved. Many charting and strategy tools can run one without code.

1.  Write your rules in plain words: what to buy, how much, and when to change.

2.  Set aside the most recent part of your data. Do not look at it yet.

3.  Run the rules on the older data.

4.  Include trading costs and realistic order prices.

5.  Look at how the rules behaved, including their worst stretches.

6.  Run the rules once on the data you set aside.

That last step matters most. Data the rules never saw during building is a fairer test than data you tuned them on.

Step 2: Test live with virtual money

A live test, also called paper trading or forward testing, runs your rules on today's prices with virtual money. Nobody knows tomorrow's prices, so this test cannot be fooled by hindsight.

It also shows practical problems a backtest misses, such as orders that do not fill or data that arrives late. Run it long enough to see calm and rough markets.

One limit remains. A 2025 study by Sui and Wang found the same investors showed stronger biases with real money than in simulated accounts. So start small when you switch to real money.

What are the 3 traps that fool tests?

Three traps make test results look better than real life. Each one is easy to fall into and hard to spot afterward.

Trap 1: Lookahead bias

Lookahead bias means your test used information that was not available at the time. For example, a rule might use a day's closing price to decide a trade made that morning.

Another form is survivorship bias. Testing only on companies that still exist today leaves out the ones that failed, which makes the past look kinder than it was.

Trap 2: Overfitting

Overfitting means tuning rules so closely to past data that they fail on new data. If you try enough versions, one will look great by chance.

It happens faster than most people expect. The American Mathematical Society shared research showing that with 5 years of data, as few as 45 versions can produce one that looks strong by luck.

Trap 3: Ignoring costs

A backtest that assumes free, instant trades at the printed price will look too good. Real trades pay fees, spreads, and slippage. Rules that trade often suffer most.

How do you know a test result is believable?

A believable result holds up after you remove every shortcut. Ask these questions about any test, yours or anyone else's.

  • Was it checked on data hidden during building?
  • How many versions were tried before this one?
  • Does it include realistic costs?
  • Did it use only information available at each moment?
  • Does the live test behave like the backtest?
  • Is the idea simple enough to explain in one sentence?

Researchers apply the same caution to their own field. Harvey, Liu, and Zhu (2016) argue that most claimed findings in financial economics are likely false and need a higher bar of proof.

For more on why tested bots still fail, read do AI trading bots actually work?

What should you do once your rules pass?

1.  Freeze the rules. Do not keep tweaking them after each week.

2.  Give your agent the exact written rules.

3.  Start with a small amount of real money.

4.  Keep trade approvals on at first.

5.  Compare live behavior to the test, and retire the rules if they break.

Rules that keep changing are a sign of overfitting. See why the best trading models refuse to learn and judge the behavior, not the returns.

Where Portfolio Lab fits

Portfolio Lab is a strategy research and simulation tool from alphaAI Capital. It lets you build investment strategies and test them in live markets with virtual money. You can connect it to AI apps like Claude or ChatGPT, so your agent has a strategy's plan to follow.

Frequently asked questions

What is backtesting in trading?

Backtesting runs a set of trading rules over past prices to see how they would have behaved. It shows the past, not the future.

Is backtesting enough before trading real money?

No. Follow it with a live test using virtual money, then start with a small real amount.

What is lookahead bias?

It is when a test uses information that was not available at the time of the trade, which makes results look better than they could be.

What is overfitting?

Overfitting is tuning rules so closely to past data that they fail on new data. Testing many versions makes it more likely.

What is out-of-sample testing?

It means testing rules on data you kept hidden while building them. It is a fairer check than the data you tuned on.

How long should I paper trade before using real money?

There is no set rule. Run it long enough to see both calm and rough markets.

Can my AI agent backtest for me?

AI apps can help write and run tests. You still need to check for lookahead bias, overfitting, and missing costs.

Put the research to work.

Build, test, and deploy your own strategies in Portfolio Lab. Free to start.

Start free

Educational & Research Disclosure: The content provided is for informational and educational purposes only and is not intended to constitute investment advice, a recommendation, solicitation, or offer to buy or sell any security. Any discussion of market trends, historical performance, academic research, models, examples, or illustrations is presented solely to explain general financial concepts and does not represent a prediction, guarantee, or assurance of future results. Past performance is not indicative of future results. All investing involves risk, including the possible loss of principal.

Related posts

Guides

How to start AI trading: finding the best AI trading app for beginners

There is no single best AI trading app for beginners, only the best fit for you. Here are the 4 main types, a step-by-step way to start, and what can go wrong.

‍

Minimal dark data visualization on a near-black background, with a glowing green chart and a subtle grid.
Robinhood

Robinhood portfolio: how to track, balance, and manage your account

A simple plan keeps a portfolio balanced. Learn how to track holdings, identify single-stock concentration, and follow clear rules.

Guides
Robinhood

How to set up a Robinhood Agentic account

A Robinhood Agentic account is a separate account built for agentic trading. Here is how to turn it on in the Robinhood app and add money in a few minutes.