Microsoft is significantly expanding the artificial intelligence features within Windows 11, focusing on Copilot's ability to create, edit files, and accept voice commands. This move marks a renewed effort to integrate a voice-driven assistant into the operating system, building on lessons learned from past initiatives like Cortana. The goal is to make AI a fundamental part of how users interact with their PCs, moving beyond app-specific functionalities into core Windows features.
Key Takeaways
- Copilot in Windows 11 now features a "Hey, Copilot" voice activation.
- New Copilot Vision allows AI to interpret screen content for assistance.
- AI capabilities are expanding into core Windows features like Search and File Explorer.
- Microsoft aims for AI agents to create and modify files in the background.
- Enhanced security and privacy measures are in place, learning from past rollouts.
Copilot Embraces Voice and Screen Understanding
The latest updates to Windows 11 introduce a new voice activation phrase: "Hey, Copilot." This allows users to summon the AI chatbot without needing a mouse or keyboard. Users can also use the dedicated Copilot key or the familiar Windows + C keyboard shortcut, previously used for Cortana. Saying "goodbye" will dismiss Copilot once a task is complete.
This development comes as Microsoft aims to succeed where its previous voice assistant, Cortana, did not. The company hopes that Copilot's advanced capabilities will provide a more effective and versatile user experience. The limitation of earlier assistants, such as Siri and Alexa, was their reliance on a predefined list of actions. Complex requests often led to general web searches, shifting the burden back to the user.
Did You Know?
The Windows + C keyboard shortcut, now used for Copilot, was originally designated for Cortana. This highlights Microsoft's intention to replace and improve upon its previous voice assistant efforts.
Copilot Vision Rolls Out Globally
Microsoft has also announced the worldwide rollout of Copilot Vision. This feature has been available in the US since mid-June and is now expanding to all markets where Copilot is offered. Copilot Vision can analyze the content displayed on a screen or within an application window. It then offers relevant guidance or feedback.
For example, it can assist users with complex tasks in Excel or provide suggestions based on a collection of photos or a list of items. This functionality aims to make the AI assistant more context-aware and helpful in real-time scenarios. A beta version of Gaming Copilot, an offshoot of Copilot Vision, is also being tested to provide in-game walkthroughs and advice.
AI Integrates into Core Windows Features
Historically, most AI additions to Windows, including the Copilot app and generative features in apps like Notepad and Paint, have been contained within specific applications. However, Microsoft is now embedding AI features deeper into the fundamental components of Windows. This includes core elements like the taskbar and Start menu, granting AI new capabilities to analyze, edit, and manage files.
The standard Search field in Windows 10 and 11 is transforming into an "Ask Copilot" field. While it will retain the ability to search local files, Microsoft envisions it as a primary keyboard-driven interface for Copilot. This offers an alternative for users who prefer not to use voice commands. Details on whether the traditional search functionality will remain an option are still emerging.
According to Yusuf Mehdi, Microsoft's Consumer Chief Marketing Officer, "AI PCs" should be able to recognize input "naturally, in text or voice," guide users based on screen content, and allow AI assistants "to take action on your behalf."
Copilot Actions Enhance File Interaction
A new feature called Copilot Actions will significantly expand how Copilot interacts with local files. Microsoft provided examples such as sorting recent vacation photos or extracting information from PDFs and other documents. This early preview will focus on a limited set of use cases but is designed to be a "general-purpose agent" capable of interacting with both desktop and web applications. This broad scope gives Copilot the potential to augment or even replace traditional keyboard and mouse inputs for various interactions.
Background Information
Microsoft's push for "agentic" AI refers to software that can perform tasks in the background after receiving instructions. This allows users to continue with other activities while the AI works. It represents a shift towards more autonomous AI assistance.
Furthermore, Microsoft is revisiting Copilot's ability to adjust PC settings. Earlier versions of Copilot had this capability, which was later removed. The updated Copilot will attempt to answer plain-language questions about PC settings by providing a link to the relevant section within Windows' extensive Settings app. This aims to simplify the process of configuring system options.
- Copilot Connectors: Already rolled out to Windows Insiders, these connectors grant Copilot access to services like Gmail and Dropbox.
- Document Creation: Copilot can now export chat content into Word, PDF, Excel, or PowerPoint files for further editing.
- File Explorer AI Actions: Appearing in the right-click menu, these allow direct file manipulation, including batch-editing images and summarizing documents.
These features, combined with Copilot Vision's ability to interpret full Office document contents, embed AI into everyday tasks rather than segregating it into isolated applications. The exact public release dates for these new features are not yet known, and some may remain exclusive to the Windows Insider program.
Lessons Learned from Previous AI Rollouts
Microsoft appears to have applied key lessons from the initial rollout of Windows Recall. The company's initial plan for Recall involved a direct release with the first wave of Copilot+ PCs, bypassing the Windows Insider Preview program. This program typically allows power users, developers, and security researchers to test upcoming features and provide feedback on bugs, security flaws, or other issues before a general release.
Security researchers who accessed the early Recall version discovered significant vulnerabilities. The feature stored extensive personal information in plain text, making it easily exploitable by anyone with access to the PC. Additionally, it was enabled by default, requiring users to manually opt out. Microsoft subsequently withdrew that version, spent nearly a year redesigning its security architecture, and subjected it to months of testing through the Windows Insider channels before its eventual rollout to Copilot+ PCs.
Security Enhancements
To protect user privacy with Copilot Actions, Microsoft employs several measures: AI agents run with dedicated user accounts, mandatory code-signing is enforced, and agents are granted only the minimum necessary privileges. All agent activities are documented for user review.
The revised Recall product, while still presenting some privacy considerations inherent in screenshotting and storing months of PC usage history, is significantly more refined. The most critical security flaws have been addressed, and the feature is now off by default. Similarly, Copilot Actions are currently disabled by default.
Dana Huang, Microsoft Corporate Vice President of Windows Security, provided a detailed explanation of the steps taken to protect user privacy and security with Copilot Actions. These measures include running AI agents with dedicated user accounts to limit data access, mandatory code-signing, and granting agents only the essential privileges for their tasks. All agent activities will also be documented, allowing users to verify actions and correct errors. These new features will undergo testing through the Windows Insider channels first, aiming to identify and resolve any serious flaws before a wider public release.