Home
/
Research
/
Why your AI gives a different answer every time
Research

Why your AI gives a different answer every time

An AI agent built on a chatbot writes each answer fresh, so the same question can produce two different portfolios. That is fine for brainstorming and risky for money. Here is why it happens and how fixed rules fix it.

The Portfolio Lab Research Team

Updated 

November 7, 2026

contents

An AI trading agent built on a chatbot writes each answer fresh, so the same question can produce two different portfolios. That is fine for brainstorming and risky for money. Fixed rules fix it.

Key takeaways

  • Chatbots like Claude and ChatGPT generate text word by word, with some randomness built in.
  • A 2023 study found ChatGPT gave different code for the same task in about half or more of problems tested.
  • Even settings meant to remove randomness do not fully remove it on hosted AI services.
  • In trading, varied answers mean varied orders, which is a hidden risk.
  • The fix is to give the agent a fixed plan to carry out, not a question to answer.
  • Let rules decide what to hold. Let the agent handle the orders.

The short answer

Your AI gives different answers because it writes each reply from scratch, choosing words by probability. Small differences early in a reply can lead to a very different answer by the end.

For an essay, that variety is useful. For an AI trading agent, it means two identical requests can lead to two different sets of trades.

Why do chatbots vary their answers?

Chatbots vary their answers because they generate text one piece at a time, picking each next word from a range of likely options. Most apps add some randomness to that choice on purpose, so replies sound natural.

Even with randomness turned down, hosted AI services are not fully consistent. A 2025 study set 5 AI models to their most consistent settings and ran 8 tasks 10 times each. Accuracy still varied by up to 15% between runs.

How much do the answers change?

The answers can change a lot. Ouyang and colleagues asked ChatGPT to write code for the same 829 problems many times. For 47% to 76% of tasks, depending on the test set, no two requests produced code with the same test results.

They also found that setting the model's randomness to zero reduced the problem but did not remove it. Trading questions are not code, but the same mechanism is at work.

Why does this matter for trading?

In trading, a different answer becomes a different order. Ask an agent "what should my portfolio hold?" on Monday and again on Tuesday. You may get two different lists, even if nothing in the market changed.

  • You cannot test a strategy that changes every time you ask.
  • You cannot tell if a bad result came from the market or from the AI's mood that day.
  • Extra trades from shifting answers add costs.
  • Mistakes are harder to trace.

FINRA is the Financial Industry Regulatory Authority. It lists hard-to-trace reasoning and hallucinations, or confident false statements, as risks of AI agents.

How do rules fix the problem?

Rules fix the problem by moving the decision out of the chatbot. A fixed rule gives the same output for the same input every time. The agent's job shrinks to carrying out what the rule says.

Agent decides Rules decide, agent carries out
Same input, same orders? Not always Yes
Can it be tested on past data? Hard Yes
Can you trace why a trade happened? Often not Yes, back to the rule
What the AI does Picks investments Places the orders the plan calls for

Portfolio Lab sums it up: language models read, explain, and execute. They don't decide.

How do you make your agent more consistent?

1.  Give the agent a written plan with target holdings and sizes.

2.  Ask it to compare your account to the plan, not to suggest ideas.

3.  Use the same saved prompt every time.

4.  Run it at the same time each day.

5.  Ask it to list orders before placing them, and compare runs.

6.  Tell it to stop and ask you if anything in the plan is unclear.

Timing helps too. See when to run your Portfolio Lab prompt. For the research on AI picking its own trades, read do AI trading bots actually work?

Where Portfolio Lab fits

Portfolio Lab is a strategy research and simulation tool from alphaAI Capital. It lets you build investment strategies and test them in live markets with virtual money. You can connect it to AI apps like Claude or ChatGPT, so your agent has a strategy's plan to follow.

Frequently asked questions

Why does ChatGPT give different answers to the same question?

It writes each reply word by word, picking from likely options with some randomness. Small early differences grow into different answers.

Can I make an AI give the same answer every time?

You can reduce variation, but research shows hosted AI models still vary even on their most consistent settings.

What is an AI trading agent?

An AI trading agent is an AI app, such as Claude or ChatGPT, connected to a broker so it can place orders for you.

Is it bad if my AI trading agent gives different answers?

For picking investments, yes. It means your strategy changes each time. For carrying out a fixed plan, it matters much less.

Does Claude also give different answers?

Yes. All chatbots built on large language models generate text this way, so all of them can vary.

How do I get consistent trades from an AI agent?

Give it a fixed written plan and a saved prompt, and ask it to carry out the plan rather than make choices.

What is a hallucination in AI?

A hallucination is when an AI states something false with confidence. FINRA lists it as a risk of AI agents.

Put the research to work.

Build, test, and deploy your own strategies in Portfolio Lab. Free to start.

Start free

Educational & Research Disclosure: The content provided is for informational and educational purposes only and is not intended to constitute investment advice, a recommendation, solicitation, or offer to buy or sell any security. Any discussion of market trends, historical performance, academic research, models, examples, or illustrations is presented solely to explain general financial concepts and does not represent a prediction, guarantee, or assurance of future results. Past performance is not indicative of future results. All investing involves risk, including the possible loss of principal.

Related posts

Guides

How to start AI trading: finding the best AI trading app for beginners

There is no single best AI trading app for beginners, only the best fit for you. Here are the 4 main types, a step-by-step way to start, and what can go wrong.

‍

Minimal dark data visualization on a near-black background, with a glowing green chart and a subtle grid.
Robinhood

Robinhood portfolio: how to track, balance, and manage your account

A simple plan keeps a portfolio balanced. Learn how to track holdings, identify single-stock concentration, and follow clear rules.

Guides
Robinhood

How to set up a Robinhood Agentic account

A Robinhood Agentic account is a separate account built for agentic trading. Here is how to turn it on in the Robinhood app and add money in a few minutes.