(This column originally appeared in Forbes)
For most of my years in the IT world there has been one tried and true rule: never, EVER buy a Microsoft product when it’s first launched. Whether it was Windows or Office or SQL Server, Microsoft has always been notorious for releasing products before they were truly ready for prime time and then letting the user community suffer with it while bugs were fixed.
With all the software tools and automation now available, Microsoft has been doing a better job at launching products with fewer problems. However, in early 2024 Microsoft released its latest AI offering called Copilot Pro and unfortunately the experience so far has been similar to when they first released Windows in 1985.
Copilot Pro – priced at $20 per individual per month (there’s additional pricing for organizations) – aims to use AI to automate many business functions in Office. Microsoft promises that AI will help create better word docs and craft more insightful emails, design great PowerPoint presentations, transcribe and monitor Teams meetings and turn any novice spreadsheet user into a power user with its natural language assistance.
Unfortunately, that’s not all happening yet. We’re not even close. And Microsoft seems to be repeating history with its premature launch.
If you don’t believe me just watch this ten minute video from Luke Barousse, a self-described nerd who’s also a data analyst with a popular YouTube channel. Barousse took a dive into Copilot Pro and, although while doing his very best to keep things positive, he could barely hide his frustration as he struggled with the application.
Barousse dug into Excel, where Copilot Pro promises to help users” generate formulas, analyze and summarize data, and add helpful visuals to spreadsheets.” That would be great, right?
It would be, if only a typical user could get it installed. Barousse is a power user. He installed it. Then couldn’t find it.
“I had to go into my account information and click ‘update license’,” he reported. “Then once I restarted Excel, it was right there.” I wonder how much time he had to spend to unravel that mystery. I wonder how much time a non-power-user would have to spend.
When he finally launched Excel Barousse was encouraged to try out Copilot’s features. The application came with a bunch of pre-installed data which great when prompted. Except that was Microsoft’s pre-installed data. But what about actual data from an actual business? Different story.
Barousse uploaded a large Excel file with a million rows containing information about people searching for jobs. No go. He was unable to get any responses to his prompts. He confirmed that the problem wasn’t a local machine issue – he was running 50% of memory and using a MacBook Pro, with a virtual machine in order to run Windows with 16GB of memory. It was “more than enough,” he said. Still…nothing.
So he bailed on the million-row spreadsheet and instead tried Copilot on a spreadsheet with 40,000 rows. Things spun until “one eternity later,” and, after “thinking and thinking and thinking” he gave up. So he reduced the rows to 10,000 then to 5,000 and Copilot kept “locking up.” Finally, he got things going with just 500 rows of data which is not a lot if you want to rely on an AI tool like Copilot Pro to provide the kind of analysis that a human can’t do.
But even with only 500 rows, Barousse was underwhelmed with the result.
It provided “bland looking” visuals that kept repeating no matter how he asked to re-do them. He was hoping to generate the slick looking multiple visualizations that Microsoft showed with its pre-installed data but he couldn’t do anything like it with his data.
Another frustration: he couldn’t work with his local files. Apparently Microsoft forces you to have your files saved in either OneDrive or SharePoint which limits a lot of options.
But at least it’s secure right? Uh, not so much.
According to Barousse, Microsoft says that when you’re signed into Copilot with Microsoft Entra ID you get “commercial data protection for free which means chat data isn’t saved” because “Microsoft has no eyes on access and your data isn’t used to train the model.” But this is for business users on networks. Unfortunately, individuals using Copilot Pro are out of luck.
“It’s still not as safe as it sounds to put confidential data into it,” he laments.
Barousse certainly doesn’t want to make any enemies at Microsoft but you could tell he struggled to put a good face on this product. He didn’t have much to say about Copilot elsewhere in Office but for sure he wasn’t thrilled with what he saw so far with Excel.
“I am disappointed in the performance of Microsoft Excel,” he said. “Specifically, I wasn’t able to recreate a lot of the different functionality that Microsoft has shown as YouTube videos, but more importantly, I’m most concerned about the data limitations that Co-pilot has in that I had to limit my data to around 500 rows to get Copilot to even work.”
He also wants to make sure that viewers know that that Excel is in its “preview” version “so it’s only gonna get better.” Wait a second…preview version? Yeah, you have to read the little footnote about Excel on the bottom of Microsoft’s blog post announcing the product launch to find that out.
But Copilot’s problems are not just Excel’s problems. Already there are a growing number of users complaining about performance issues and problems ranging from inaccurate suggestions to limited language support.
So when my clients are asking about Copilot I’m telling them to stay away from it this year. You should do the same. Let Microsoft work out the kinks. In the meantime, try to figure out exactly what problem will it solve? How will it help your business? Who will use it the right way? What’s the ROI?
Perhaps by the time you’ve thought about and answered these questions Copilot will actually work as promised. I’m confident that it will. But just not this year.
