Get ready, Android users! A new feature is brewing in Android 16 QPR3, hinting at a future where your phone can do even more. This update introduces 'Screen automation' on the Pixel 10, designed to facilitate 'computer use,' potentially revolutionizing how we interact with our devices. But what exactly does this mean? Let's dive in.
Currently, Google's efforts in this area, powered by Gemini, are focused on desktop web applications, as seen with the Gemini Agent available for AI Ultra subscribers. The move to Android seems like a natural progression, and Android 16 QPR3 Beta 2 is preparing for this with a new 'Screen automation' permission.
What's the 'Screen automation' all about?
This information comes from an 'APK Insight' post. These posts involve deconstructing the latest app versions that Google uploads to the Play Store. By examining the code within these files (called APKs for Android apps), we can get a glimpse of potential future features. Keep in mind that these features might not always make it to the final product, and our understanding of them might be imperfect. However, the closer a feature is to completion, the more likely we are to see how it will function.
In Android 16 QPR3 Beta 2, navigating to Settings > Apps > Special app access reveals a new 'Screen automation' option: 'Allow apps to help you complete tasks by interacting with other apps’ screen content.' Interestingly, this permission is currently exclusive to Pixel 10 devices and is not available on the Pixel 9 series. Whether this exclusivity will change in the future remains to be seen.
The Google app, which is responsible for Gemini, appears to be the only supported application at the moment. Users will have three options: 'Always allow,' 'Ask every time' (the default), and 'Don’t allow.'
This app will have the ability to observe and interact with the content on other apps' screens to help you complete tasks, even when those apps are running in the background.
The limited information available today refers to this as 'computer_control.'
Here are some additional details:
-
string name=”request_computer_control_access_dialog_description”This app will be able to see and interact with other apps’ screen content to help you complete tasks, even when the apps are in the background. -
string name=”automated_app_launch_warning_dialog_description”To view task progress, open the %1$s app.\n\nIf you stop the task now, you may lose your progress. To restart, open the %1$s app. -
string name=”automated_app_launch_warning_dialog_stop”Stop task & open app
So, how does 'computer use' work?
For Google, 'computer use' involves agents navigating the web and applications 'just as humans do: by clicking, typing, and scrolling.'
Back in May, Google showcased Project Astra (used to test future Gemini Live capabilities), demonstrating its ability to scroll through Chrome for Android and interact within the YouTube app.
The Future is Now?
Google is laying the groundwork for all of this in Android 16 QPR3, but the exact timeline for the Gemini integration remains uncertain. This could mean a significant leap forward in how we use our phones, but it also raises questions about privacy and control.
What do you think? Are you excited about the potential of screen automation and computer use on Android? Do you have any concerns about how this technology might impact your privacy? Share your thoughts in the comments below!
