Calibrating Copilot Ep. 4: We Gave Copilot 191 Pages and Asked It to Build an App
For Episode 4 of Calibrating Copilot, Dino Caputo and I put vibe coding to the test. Can Microsoft Copilot build a working business application from a complex set of requirements?
The challenge
We found a 191-page manual for cyber liability insurance for health professionals. Most of it was application material and underwriter information that should be ignored, but about seven core pages contained rating tables and fairly sophisticated calculations, including exponents and Weibull curves, which I had to look up on Wikipedia and still do not fully understand. Actuaries apparently use them.
We asked Copilot to use the manual as the calculation specification and build a fully functional, interactive premium quotation app. It needed to implement all applicable rates, tables, formulas, limits, interpolation rules, credits, debits, and caps; let the user change every input required for a valid quote; and show both the final premium and an itemized calculation. Critically, we told it not to invent calculation rules that were not in the specification and to identify anything that was genuinely ambiguous. When you are quoting insurance, you do not want an LLM getting creative.
We tested Copilot Chat (Basic), Microsoft 365 Copilot, the Researcher agent, Cowork at Medium and Max, and Cowork with the GPT-5.6 Sol model selected. As a bonus, we compared the results with Claude.
Copilot Chat: do not use it for vibe coding
Copilot Chat could not process the full PDF. Even after I gave it just the seven relevant pages, four out of five attempts produced an app whose Calculate button did nothing, because output limits prevented it from delivering the complete source code. It then wasted 20 minutes offering workarounds it could not actually deliver: a ZIP file, then a base64-encoded file with three options to choose from, none of which worked. The one time it produced a complete app, it was ugly and its results were off by a few dollars because of a factor that should have been 1 but was 1.01. Dino only got a placeholder. One star.
Microsoft 365 Copilot: the winner
Microsoft 365 Copilot built a good-looking, working app in just under four minutes. It created a single HTML file stored in OneDrive, which made it trivial to share with others. Its results were accurate and consistent with the other tools, and it included sample scenarios for validation. My version added a print button we did not ask for, and both versions listed the specification ambiguities they found in the manual, exactly as instructed.
Our two runs also produced visibly different apps from the same prompt and input, a good reminder that AI is probabilistic.
Researcher: a great story, but no app
Researcher took a while and then told a wonderful story about the self-contained, browser-based quoting tool it had built. The download links were not clickable and pointed to an inaccessible sandbox. When I asked it to repackage the app, it took another 10 to 15 minutes and then served up a file I had previously created with Cowork Max, presenting it as its own work. Zero stars. To be fair, Researcher is not designed for this.
Cowork: impressive, but it builds more than you asked for
Cowork Max built a full React application, roughly 50 TSX files, and published it as a Power App. It was accurate and polished, took 16 minutes and 35 seconds, and used 544 AI credits, about $5. However, every Cowork run defaulted to creating a Power App, which means you use credits to build it, and anyone who runs the published app may consume usage credits too. We hit messages saying we did not have enough credits to use the app and that access to managed apps was unavailable, even though we have pay-as-you-go billing enabled.
Cowork with GPT-5.6 Sol chose a different color scheme and was the most complete of the Copilot options. It added a section for mid-term policy changes and cancellations, and when certain custom inputs created an ambiguity, it declined to calculate a premium and explained exactly why. But converting its Power App into a standalone HTML app brought the total to 40 minutes, and the second step used more credits than the first.
Bonus: Claude
Claude Opus 5 High produced arguably the most complete app. For every ambiguity in the manual, it offered a dropdown so you could choose how to interpret the rule, and its ambiguity register highlighted exactly which ambiguities affected the quote on screen. Since Anthropic models are available within Copilot, that is something we want to test in a future episode.
We also tried Copilot's experimental Advanced Reasoning option. It ran for 90 minutes, repeatedly assured me it was making progress, and produced nothing.
Key lessons
Copilot Chat is not suited to vibe coding; even when it produces an app, the results may be wrong.
Microsoft 365 Copilot is very capable of creating standalone HTML apps, and you can iterate as many times as you like without generating an AI credit bill.
As Dino put it, Cowork optimizes for completeness while M365 Copilot optimizes for sufficiency. Agentic AI tends to expand the scope unless you constrain it.
Tell AI what not to build. If you want a standalone HTML app from Cowork, say so explicitly, or it will build a Power App.
Validation still matters, no matter how convincing the app looks.
Our recommendation: for vibe coding a business tool like this, use Microsoft 365 Copilot. It was fast, accurate, and did not cost anything beyond the license.
Sufficiency is a licensing strategy too
Dino's distinction between completeness and sufficiency applies well beyond vibe coding. Most organizations do not need the most powerful Copilot option for every person; they need the right option for each person's actual work, with Cowork reserved for the tasks that genuinely justify it.
That is exactly what our Copilot Calibration service is built to solve. Using your existing Microsoft 365 usage data, we deliver person-by-person recommendations: who should get a Microsoft 365 Copilot license, which licenses should be reassigned, where Cowork makes sense, and where targeted training would pay off. It is fixed-price and requires minimal effort from your team.
