In a randomized trial, 50 experienced analysts working with an AI agent sided 86% of the time with whatever option a secret coin flip had told the agent to favor, and about half still called the final judgement their own. This is both a warning and an opening: Good advisors will make those calls visible. Good clients will ask to see them.
The dog knows the way
Most mornings I walk our dog, Charlie, around the neighborhood - or, does he walk me? As I get to enjoy the early daylight, he doesn't miss a rock or tree to sniff; with a sudden change of direction, I follow willingly. Lost in thought, I rarely remember our exact route.
Last week, Anthropic, the company behind Claude, published a study that reads like my morning walk. If you pay a professional for judgment, and especially one that has quietly added an AI assistant to the team, this affects you too.
What the study found
The researchers gave 50 professional analysts a realistic assignment. All were experienced at reviewing data and figuring out what actually causes what. Their fictional client was a farming cooperative deciding on whether to expand a paid advisory program for its members, provided it lifted productivity by roughly 3 – 4%.
Along the way, each analyst hit four forks in the road, each with two perfectly reasonable options. Each choice could move the answer more than enough to clear or miss the board's hurdle.
Here is the twist: Before each session, a coin flip secretly decided which way the analyst's AI agent would argue for. The AI agent would change its mind if evidence demanded it and would never mislead, but it also never revealed its mechanics to the analyst.
The results:
- 86% of the time, analysts took the side the coin had picked. Instead of the expected 50%.
- Stated views adjusted. When analysts had already declared a preference, they dropped it 60% of the time if the agent disagreed, versus 6% when it agreed.
- Predictive power of coin flips. Knowing only the agents' preference, the researchers could predict 82% of the ultimate recommendations, versus 61% for a simple guess.
- Borrowed words and no credit. Only 15% mentioned the agent, and 77% justified their choices using the arguments of the agent - sometimes word for word.
- Nobody felt the lead. Analysts following the agent on every fork were as confident as the rest (71 vs. 73 out of 100).
- Seniority didn't help. Job experience, degrees, and test scores had no predictive power.
On the bright side: analysts using the agent finished faster and made fewer serious errors (6% vs. 55%).
What this means for your money
A financial plan is built on the same kind of forks. What return should we assume? When should you claim Social Security? Each question has more than one defensible answer: Each is a judgment call, and not a calculation.
Here is the uncomfortable part: Small forks can lead to big consequences. Over 25 years, $1 million grows to about $3.4 million at 5% a year, and to about $4.3 million at 6%. That 1% difference, roughly $900,000, can decide whether you can afford to retire at 65 or not. Mistakes can be caught - judgment calls not. A plan built on 5% and a plan built on 6% expected returns both look perfectly reasonable, and there is no right answer to check them against. That's why, as the researchers note, deferring to an AI is so hard to spot: The work looks sound either way.
What good advisors do differently
When a machine does the arithmetic in minutes, clients stop paying for it, and judgment becomes the scarce product.
- Pick your route before you start the chat. Write down your view and key assumptions first. If you change your mind, you can ask "why."
- Make AI argue against itself. Ask it, or another model: "Give me the best argument for the other choice."
- Keep a log. For every recommendation, list the judgment calls that drove it. It not only summarizes your advice, but is also the kind of evidence examiners are starting to look for (see the SEC's 2027 examination priorities).
- Invest time saved in judgment. Analysts using AI's support in the study saved 25 minutes (median). Allocate some of that time to judgment calls.
Human approval alone is weak control; independent judgment is a strong one. An advisor who doesn't overrule the machine is not supervising it, they are just co-signing.
For clients: five questions that expose a lazy advisor
A lazy advisor is not one who uses AI, but one who lets the AI make the calls and then signs off on them.
- "Which assumptions led to the recommendation?" A good advisor has the fork log ready; a lazy one only the summary.
- "Do you use AI on my account? For what, who checks it?" Listen for specifics, including what happens to your data (more on that in Your Chatbot Can't Keep a Secret — Can You?).
- "When did you last disagree with your AI?" You want to hear a concrete story.
- "What is the best argument against this recommendation?" If your advisor cannot argue the other side, they never thought about it.
- Write down your own route. Going into the meeting with some of your right answers helps you challenge your advisor's assumptions and pre-baked recommendations.
Pick the route before the leash
This morning I tried an experiment: I decided to walk uphill first. At the end of the driveway, Charlie leaned right, as he typically does and there was a perfectly good reason to go downhill first. However, I led Charlie left, up the slope, and he was perfectly happy sniffing different trees.
The point is that following should be a choice I make, not a habit or story I tell after the fact.
The same goes for your financial decisions. The good news is that AI is going to make good advisors faster, more thorough, and less error-prone. It is our job to ensure that a trusted human with sound judgment backs every decision at every major fork.
So, decide where you are going before you clip on the leash, and then - by all means - let the dog pull you.
Sources
- Anthropic, How AI agents shape experts' judgment calls, published Oct 27, 2026, and the full paper, "Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls"
- SEC Division of Examinations, Fiscal Year 2027 Examination Priorities, and WilmerHale's summary (published Oct 23, 2026)