Calibrating Copilot Ep. 8: Can Copilot Predict the Future?
For Episode 8 of Calibrating Copilot, Dino Caputo and I tackled a big question: can Copilot predict the future? We put Microsoft 365 Copilot, the Researcher and Analyst agents, and Cowork at Light, High, and Max through a forecasting exercise, then compared their predictions with actual results.
Why flight data?
We originally planned to use financial data, but we ran into something researchers call temporal data leakage. Even when you tell a model to use only historical data, it was trained on much of the internet and already 'knows' what happened later, which makes a fair test very difficult. Flight delay data is far less likely to be baked into a model's training.
Canadian airlines report this data but it is not made public, and EU data is less detailed, so we used US flight records: a compressed folder of just under 70 MB. Several files exceeded Excel's limit of 1,048,576 rows, so you could not even open them in Excel to do the analysis yourself. (For the curious, Excel supports up to 16,384 columns.)
The prompt
Using the attached US flight records, forecast August 2026 percentages for arrival delays of 15 minutes or more and for cancellations on every listed route. Clean and analyze the raw data, identify seasonal, airline, and airport patterns, and briefly explain the forecasts and confidence levels. Deliver a forecast table, an executive-ready route risk map, and a comparative heat map, using only these files with no external research.
Cowork: beautiful visuals
Cowork Max produced a gorgeous risk map overlaid on the United States, with eight high-risk routes in red. Atlanta to Orlando, for example, carried a 33% chance of arriving 15 or more minutes late. It also produced delay and cancellation heat maps and an Excel workbook, taking 32 minutes and about $10 in credits. Cowork High was similar and only slightly less polished. Cowork Light took two minutes and $0.67, but its map lacked the US outline, its boxes overlapped, and its heat map was harder to read.
Microsoft 365 Copilot, Researcher, and Analyst
Microsoft 365 Copilot could not handle the ZIP file, and it also choked on the 11 or 12 extracted files. Dino's clever workaround was to put the files in OneDrive and give Copilot the share link, which worked perfectly. In about 16 minutes, it produced a single Excel workbook. Its 'map' was really a scatter chart plotting cancellation and delay percentages rather than geography, but its heat map was a sortable Excel table, which is more useful than a static image.
Researcher asked clarifying questions about optimizing for predictive accuracy, how to express confidence, and report length, then produced the only Word report of the group, with more narrative. The Analyst agent, finally on a genuinely analytical task, created separate image files for the risk map and heat map; its map also was not geographic.
The moment of truth
We gave the actual August 2026 data to ChatGPT and Gemini and asked them to score each forecast. Microsoft 365 Copilot did best overall, ranking first for cancellations and second for delays, with a balanced normalized error of 36%. Cowork High and Cowork Max, with the most impressive visuals, finished at the bottom.
All six Copilot options underestimated delays on every route. Their cancellation forecasts were also too optimistic: where they predicted roughly one cancellation per 100 flights, some routes saw closer to two. Most strikingly, a traditional forecast that simply used the average delays and cancellations from 2024 and 2025 was more accurate than any of the AI forecasts.
One more lesson: when we first compared the results, ChatGPT flagged that our actuals file showed zero cancellations on routes with many flights and called it suspicious. It was right; our source extract had missed the cancellations. Read the thinking and output, not just the final answer.
Interesting facts from the historical data
More than half (51.4%) of Seattle to San Francisco flights historically arrived 15 or more minutes late, despite being a short hop.
Direction matters: some routes had significant delays in one direction and not the other.
LaGuardia to O'Hare and Boston to Washington Reagan accounted for 53% of all cancellations in the data set, despite representing only 19% of flights.
The verdict
Can Copilot predict the future? Not reliably, at least in our experiment. It produces plausible forecasts, and Cowork can produce visuals so polished that they draw you in, but as Dino put it, Cowork delivered more sizzle than steak. Sophisticated AI does not automatically beat simple statistical methods. If you need a forecast, start with traditional techniques, and if you use AI, slow down and verify the results against the data.
Measure before you trust
The most important lesson from this episode applies to Copilot investments generally: impressive output is not the same as measurable value. Organizations should base decisions about who gets which Copilot license on actual usage data, not on how good the demo looks.
That is exactly what our Copilot Calibration service is built to solve. Using your existing Microsoft 365 usage data, we deliver person-by-person recommendations: who should get a Microsoft 365 Copilot license, which licenses should be reassigned, where Cowork makes sense, and where targeted training would pay off. It is fixed-price and requires minimal effort from your team.
