Why "agentic" execution fails the audit test, and what AI is actually good for in ERP quality
There's a story that made the rounds on LinkedIn this spring. It goes something like this:
A senior executive at an unnamed company lays off his entire QA team in favor of "agentic testing" and saves $1.2 million in salary and benefits.
In a subsequent deployment, the agent hallucinates a promo code that prices every product on the website at $0.00. More than $6 million in products sell for nothing before anyone catches the issue.
The executive, now under some scrutiny, asks the Director of Engineering to get the laid-off QA lead on the phone to consult on the fix. For free, of course.
The story is outlandish, and more than likely it is not true. Plenty of people have tried to verify it independently, and the company in question has never been identified.
But whether the story is true almost doesn't matter. What matters is that it is entirely believable. That is the real indictment of the current climate.
"Agentic" everything is all the rage. The idea of replacing or augmenting headcount with tokens has become attractive. Worse, companies that don't have something agentic in the stack worry they will look like slow adopters. So they adopt tools with questionable returns at best, and significant losses at worst, all in the name of appearing current.
Riding this trend, a wave of "agentic testing" platforms has appeared, and plenty of organizations are now asking whether they should improve or replace their existing testing practice with agentic tools. Some platforms advertise the ability to create, execute, and maintain tests with no human in the loop at all.
For ERP, this is the wrong approach. Here is why.
Probabilistic tools, deterministic work
Agents and LLMs are probabilistic technologies. They respond to a prompt with the answer that is probably correct based on their training. You can improve the model and the prompts over time, but the answer will always rest on "probably."
Regression testing is deterministic by design. A tester performs the same steps, with the same data, every time. They did it this way last cycle, and they will do it this way next cycle. If the inputs are the same and the outcome changes, there is a defect.
That is the entire foundation of regression testing: known inputs, expected outcomes. If the inputs vary from run to run, you cannot tell whether a changed outcome came from the inputs or from the code. There is no way to confidently log a defect when the change cannot be isolated to a single cause.
An agent that improvises its way through a test plan breaks that foundation. Every creative execution path is a variable you didn't control.
Nowhere does this matter more than in Dynamics 365
Microsoft ships updates on Microsoft's schedule, not yours. Every One Version service update is a forced regression cycle for finance, supply chain, and every process your business actually runs on. The question is never whether you will retest. It is whether you can retest fast enough, and prove what you tested.
That second part is the killer. Regulated environments running on Dynamics 365 (SOX controls, GxP, financial services) cannot simply say the tests passed. They must attest that the results are valid and show an auditor exactly what was executed, how, with what data, and with what outcome. Every release. Every time.
An agent that executes a test differently this cycle than it did last cycle cannot give you that. "The agent probably covered it" is not a sentence you want to say to an auditor.
Use AI where it belongs
None of this means AI has no place in testing. It means AI has a specific place.
At The TestMart, we built Horizon to use the best of both worlds. AI does what it is good at: authoring tests from intent, self-healing scripts when Microsoft moves the furniture, and reducing the lift of building and maintaining the suite. Execution, though, is always deterministic. The test steps captured in the suite are logged and maintained so they run the same way every time, and the outcomes rest on a valid, repeatable process.
AI speeds the work. Determinism enters where it matters. You get the leverage of AI and still hand certified, defensible results to audit.
Agents run on "probably." Probably doesn't cut it when the auditor comes calling.