Calibrating Copilot: Which Copilot Writes the Best Executive Project Summary?
Microsoft now offers several flavors of Copilot, and knowing which one to use for a given task is far less obvious than it should be. That is the gap our new series, Calibrating Copilot, sets out to close. In each episode, Dino Caputo (13-time Microsoft MVP for Copilot and Teams) and I take one real business use case, run the identical prompt through each Copilot option, and rate the results on accuracy, completeness, speed, and cost.
Our first test: turning a long, detailed project document into an executive summary.
The test
The source was a detailed project description of roughly 65 pages. We asked each tool to summarize it into an executive brief of no more than 1,800 words covering five items:
Key project objectives, emphasizing specific quantitative goals
Important assumptions associated with the project
Critical project milestone dates
Risks associated with the project
Unanswered or unknown issues
The prompt also told Copilot to reference only this document, to base the summary on verifiable facts it contains, and to write in a clear, concise style suitable for an executive briefing. Dino and I each ran the same prompt against the same document in all four tools, then averaged our scores. Where a model choice was offered, we left it on Auto.
Copilot Chat (Basic): free, fast, and a bit inconsistent
The free Copilot Chat finished in about 26 seconds. Its biggest annoyance was a citation on every single bullet, all pointing back to the same PDF; not exactly clear and concise. In my run it also missed two important numbers: the revised project budget of $2.5 billion and the estimated $1.5 trillion lifetime benefit. Dino's run caught both, which is a useful reminder that the same AI can produce different results on different runs.
Accuracy averaged four stars (I gave it three, Dino gave it five). On cost, you largely get what you pay for, but it is hard to beat zero.
M365 Copilot (Premium): quick and well organized
The paid M365 Copilot ($30 per user per month) averaged about 33 seconds and shared the same repetitive citation habit. Funnily enough, our results flipped: my run captured both key figures while Dino's missed the $1.5 trillion benefit. Overall it scored higher than Copilot Chat on accuracy and completeness, and Dino felt it made the best use of H1 and H2 headings, producing the easiest document to navigate. Because the license is effectively all-you-can-eat, there is no incremental cost per summary.
Researcher agent: the deepest analysis
Researcher comes included with an M365 Copilot license. It produced the most complete output: fuller paragraphs, every key number, and a good narrative explaining how the budget grew over time. It was also the only tool that placed the executive summary at the top of the document, where an executive would expect it.
The downsides: the same repeated citations, heavy bolding that loses meaning after a while, and headings that were less navigable than M365 Copilot's. Researcher also asks clarifying questions before it starts (how long, what to include, or "use my best judgment"), so do not type the prompt and walk away; it will wait for your answer. It took about 2 minutes and 45 seconds, and converting the result to a Word document took just five seconds.
Scores: accuracy 4.5 stars, completeness 5 stars, and 4 stars on cost.
Cowork: the best-looking document, at a price
Cowork produced the cleanest Word document of the group: no citation clutter and some well-structured tables. It did, however, run about 250 words over the 1,800-word limit by Word's count (Dino noted that if you strip out citations and hyphenated words it comes close to compliance, so this one is debatable).
Speed was the surprise. The on-screen summary took roughly a minute and a half, but when Cowork offered to create a Word document, that step took 6 minutes and 13 seconds in my run. Dino compared the two and found Cowork was rebuilding the document from scratch rather than copying the text across, which explains both the better formatting and the time.
Cowork also consumes AI credits. At the list price of one cent per credit, the initial summary used 108 credits (about $1.08), and generating the Word document added another $3.82. Call it four to five dollars per project summary.
Scores: accuracy 3.5 stars, completeness 4 stars, and 3 stars on cost.
The verdict
A clear pattern emerged: the two chat-based tools stayed disciplined on length but sacrificed some depth, while the two agentic tools (Researcher and Cowork) produced richer documents at the expense of the word limit.
For summarizing a detailed project document, our recommendation is Researcher. It delivered the best combination of accuracy, completeness, and value. Dino felt it was close between Researcher and Cowork, and Cowork's polish may be worth it in some situations.
A few dollars per summary might not sound like much, but cost is relative: multiply it across your document volume and frequency and it adds up quickly. One goal of this show is to build practical benchmarks for what common Copilot use cases actually cost, because today there are very few.
Have a use case you would like us to test? Leave it in the YouTube comments and we will run it through the calibration lab.
Is the right Copilot in the right hands?
This episode shows that the "best" Copilot depends on the task. The same is true of the people in your organization. Paying $30 per user per month for someone well served by Copilot Chat wastes budget; leaving a heavy user on the free tier, or running Cowork without watching credit consumption, leaves productivity and money on the table.
That is exactly what our Copilot Calibration service is built to solve. Using your existing Microsoft 365 usage data, we deliver person-by-person recommendations: who should get an M365 Copilot license, which licenses should be reassigned, where Cowork makes sense, and where targeted training would pay off. It is fixed-price and requires minimal effort from your team.
