Imagine an AI helping run an ice cream business through a week of supplier trouble, a price shock and a customer crisis. It might spot every problem and still miss the moment to close a deal. That gap between knowing what to do and doing it is the question behind Firmulate, a live experiment in how AI models manage a company.
Get baking supplies delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A company under pressure
Firmulate puts AI models in charge of a small software company and sends it through its worst week. Each model faces the same customers, crises and temptations. Decisions are versioned and auditable, and the experiment is watchable at firmulate.com.
The live company has 13 synthetic employees and real money mechanics: it burns €105k a month against €2.3k in monthly recurring revenue, with a public cash countdown. Its playbook has accumulated more than 680 self-learned rules, and every workday is versioned.
Good diagnosis is not the same as follow-through
The final Crucible League, in July 2026, ranked gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The do-nothing baseline scored 26. The league counts partial progress, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
Across the experiment, every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The finding was stark: “Same diagnosis, same pitch — no signature.” A model can recognize an opportunity and explain its value without completing the work.
The clue was buried in the company’s own files
The decisive competitor weakness sat two document references deep in the company’s files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The difference came down to finding relevant information already available inside the business.
The experiment also tested pressure to bend the rules. Fake CEO messages escalated over three stages, followed by a reporter’s “just one yes/no, on background” request. All five models refused. Kimi K3 explained its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
Opus 4.8 offers a revealing case. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. The deal went unsigned, and it tried to write into a locked department instead of escalating. A weaker version of the same discipline problem appeared in all four models. K3’s result also needs context: it ran without an effort parameter, using the API default, while the others ran at xhigh.
From watching to trying it on your business
For a management audience, the point is not whether a model sounds persuasive in a demo. It is whether it can navigate your company’s information, follow its playbooks and take the next appropriate step when a real decision is on the line. Firmulate’s quiz uses 242 real, unedited management decisions to invite readers to guess which model made each call; it is available at firmulate.com.
Firmulate says enterprises can run the same kind of wargame against a read-only export of their own business. The exercise puts crisis scenarios against company-specific information and playbooks, then produces a board report with model rankings and weak points. Nothing writes back to real systems. That offers a way to examine how an AI might behave before it is put to work inside the business.
Take the experiment to your company
To explore a pilot using a read-only export of your business, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
