The Impact and Future of Generative AI in Business

Microsoft’s report, “Generative AI in Real-World Workplaces,” highlights a key finding: the difference between perceived productivity and measurable time savings with generative AI in business tools. Employees feel a reduced mental burden and perceive an increase in productivity, along with enhanced job satisfaction and higher-quality work when using AI, even if the measured time saved is relatively small. Let’s dive into the how to justify generative AI tools and key findings from the Microsoft report.

For example, while users of Microsoft 365 Copilot feel more efficient, a study shows only a 4% reduction in email interaction time, with some organizations achieving up to 25%. Additionally, Copilot users created and edited more documents: heavy users of Word, Excel and PowerPoint saw a 13% increase, and some organizations experienced up to a 30% rise. This increase may be due to Copilot making it easier to produce and revise output or that Copilot users are reinvesting saved time into additional document creation and editing. The findings are a mixed bag, supporting both the glass-half-empty and glass-half-full worldviews of those who argue for and against the introduction of workplace AI.

Justifying Generative AI Tools

Justifying the ROI of AI productivity tools can be challenging since many benefits aren’t easily quantified. However, the productivity gains are real, as anyone who has first-hand experience with clawing back 5-30 minutes in their day can attest. Chatbots and AI-enhanced office apps are just the beginning, with many more Large Language Model-enabled apps being built by startups. Instead of waiting for more commercial-off-the-shelf (COTS) tools to arrive, businesses that want to maintain a competitive edge should start AI experimentation with proof of concepts around key processes to see where they can make a difference.

As artificial intelligence continues to progress, Microsoft’s report presents a detailed and current overview of the influence that generative AI applications are having on knowledge-based work. Although Microsoft is deeply involved in AI development and has vested interests in this field, the report delivers a clear and thoroughly cited examination of how AI tools are transforming productivity and operations in business environments today.

Key Findings from the Microsoft Report

The generative AI report synthesizes results from over a dozen studies, including what is claimed to be the largest randomized controlled trial of generative AI in organizations. These studies span various use cases and methodologies:

1. A large-scale field experiment involving over 6,000 employees across 60+ organizations, measuring the impact of Copilot for Microsoft 365 on email, meeting and document behaviors.

2. Surveys of Microsoft Copilot users, exploring perceived benefits and time savings across different job functions.

3. Studies on the impact of AI-assisted coding tools on overall code quality and developer experience.

4. Analysis of how AI-augmented search differs from traditional search in complexity and domain focus.

5. Studies on the impact of AI tools in specific roles, such as security professionals and sales teams.
6. Experiments on AI’s effect in multilingual contexts and its impact on cognitive load.

The Complexity of Measuring AI’s Impact

One of the most intriguing findings from the report is the discrepancy between perceived and measured productivity gains. This tension between subjective experience and objective measurements highlights the complexity of evaluating AI’s impact in the workplace.

At the experiential level, employees consistently report feeling more productive and saving time with AI tools, though these savings of 5-30 minutes each day are hard to quantify. Users describe increased efficiency, improved work quality and greater job satisfaction. For instance, in the M365 Copilot Usage in the Workplace Survey, customer service and sales professionals reported high levels of agreement when asked about increased productivity with Copilot. Users also reported feeling more fulfilled in their work and experiencing improvements in work quality when using Copilot.

However, objective measurements often show more modest gains. The Early Access Program Telemetry Study found an average 4% reduction in email interaction time, with some organizations seeing up to 20-25% reduction. These objective measures, while positive, don’t fully capture the subjective benefits reported by users. The report notes a “common disconnect between the time savings people report from Copilot use and the actual time savings measured.” This disconnect was observed across different studies and individual studies that collected survey and telemetry data.

The report suggests several reasons for this discrepancy:

• Enjoyment Factor: People may find using Copilot more enjoyable, which can affect time perception.

• Ease of Information Extraction: Copilot may make it easier to process information, making time seem to pass more quickly.

• Time Reinvestment: People may use the time saved to do additional work, such as creating and editing more documents.

This complexity in measuring AI’s impact presents a challenge for businesses in quantifying the ROI of AI tools in knowledge work. However, it’s important to recognize that the subjective benefits reported by employees are valuable in and of themselves, potentially contributing to job satisfaction, employee retention and overall workplace effectiveness. Additionally, these subjective experiences may drive long-term productivity gains and innovation, suggesting that a more holistic evaluation approach is necessary to fully understand and leverage the benefits of GenAI tools.

The Rise of “Shadow AI” and Compliance Concerns

The above lack of clarity of objective measures makes defending an ROI difficult. For businesses driven by such practical considerations and looking to bring in GenAI tooling today, a better focus will be on tractable matters like compliance. The Work Trend Index Survey revealed that 78% of respondents who used AI had utilized at least some AI tools not provided by their organization.

This “Shadow AI” phenomenon highlights a critical point from the report: the high user percentage indicates a widespread adoption of non-endorsed AI tools in the workplace, presenting likely compliance and security risks for organizations. Companies may be unaware of what data is being processed by these unapproved tools, potentially leading to data privacy violations or intellectual property risks.

The high rate of “Shadow AI” usage suggests there’s a strong demand for AI tools among employees, creating an urgency for businesses to provide approved, secure AI solutions to meet this demand and mitigate risks. Unlike the harder-to-track productivity gains, addressing “Shadow AI” provides a clear, actionable focus for businesses. Implementing approved GenAI tools can be framed as a risk mitigation strategy, which is often easier to justify in terms of ROI.

Beyond Productivity: AI as a Strategic Imperative

Regardless of whether the decision to adopt AI is driven by numerical factors, compliance or a general need to experiment and innovate, we are only beginning to build business-focused applications based on LLMs. The chatbot user interface popularized by ChatGPT is just the first mode of interacting with LLMs. Microsoft and Google quickly followed it up by providing integrations into existing product suites for more focused scenarios like document drafting and presentation creation. However, these productivity and knowledge tools are not the ultimate applications of AI in business.

We are just scratching the surface of how LLMs can be applied to business processes. The rapid progression from general chatbots to integrated productivity tools shows how quickly the field is evolving, suggesting many more innovative ways of leveraging LLMs in business contexts are on the horizon. While productivity enhancements are valuable, they don’t necessarily provide a sustainable competitive advantage. The real potential lies in applying AI to core business processes unique to each organization. By focusing on these processes, companies can create AI applications tailored to their specific needs and market positioning, yielding more significant and sustainable benefits than off-the-shelf solutions. This approach aligns with the economic theory of comparative advantage, where businesses gain by specializing in areas where they are most efficient, leveraging AI to enhance their unique strengths.

Realizing AI’s full potential will require a period of agile experimentation. This approach allows businesses to discover unique applications of AI that align with their specific goals and challenges.

Rethinking Business Processes for AI Integration

Business processes selected for AI integration should be reorganized around a task-based view. This approach allows for a detailed and flexible integration of AI, identifying specific tasks where AI can add value, even if initially small. Viewing processes as a series of tasks helps pinpoint areas for AI implementation and ensures these tasks can be measured and optimized individually. This approach enables future integration of new AI capabilities without overhauling entire processes, helping businesses future-proof their operations. It further allows for targeting new tasks and areas as AI becomes more capable.

A task-based approach allows for precise measurement of AI’s impact whether existing KPI data is available or not. After a baseline is established, any new AI-assisted productivity improvements can be reliably measured. This will provide for a much more quantifiable ROI compared to the state of the individual productivity gains discussed above.

The Path Forward: Strategic AI Adoption

As businesses navigate the complex landscape of AI adoption, several key considerations emerge:

• Look beyond off-the-shelf solutions to explore how AI can be customized for your specific business processes.

• Embrace rapid experimentation to discover unique applications of AI that align with the unique advantages your business has in the marketplace.

• Partner with AI experts who understand both the technology and its business applications to guide your adoption strategy.

By taking a strategic, process-oriented approach to generative AI in business integration, businesses can position themselves to reap both immediate benefits and long-term competitive advantages as AI technology continues to evolve.

Click here for more tips on safe and effective AI adoption.