OpenAI launches ChatGPT app for Apple Vision Pro

A glimpse of the future of human-AI interaction

The launch of ChatGPT for the Vision Pro marks a significant milestone for OpenAI, which aims to create artificial intelligence that can benefit humanity. It offers a glimpse of what the future of human-AI interaction might look like: more natural, more intuitive, and more immersive.

Rather than typing into a chat interface, in the near future, we will be more frequently dictating our needs to an AI assistant app by speaking to it, and even feeding images to it from the real world simply by looking at the problem we need to solve (such as making a recipe using available ingredients from our kitchen by laying them on the table, looking at them, and asking our AI assistant).

One of the first apps for visionOS, the operating system for Apple Vision Pro

With its release today, ChatGPT become one of the first of more than 600 new apps that have been built for visionOS, the operating system that powers Apple Vision Pro. VisionOS is designed to provide a seamless and immersive experience for users who want to interact with digital content in the real world.

According to Apple, visionOS apps can take advantage of the Vision Pro’s features, such as Optic ID, the biometric authentication system that uses eye tracking and iris recognition; Spatial Audio, the sound system that creates realistic and directional audio effects; and VisionKit, the framework that enables developers to create apps that can see, hear, and speak. It’s still unclear if the new ChatGPT app for visionOS will take advantage of any of these advanced features.

The new ChatGPT app is also part of a broader industry push toward multimodal AI, a type of artificial intelligence that can process and understand inputs from different modes or modalities, such as text, speech, images, and videos. This means that it can recognize and interpret various forms of data, not just one type, which makes it more versatile and adaptable to different situations.

A new way to chat with AI using text, images, and voice

Existing versions of ChatGPT can already chat with text, in addition to images and voice. This will likely become a lot easier with the Vision Pro, where users will be able to show ChatGPT images more easily and intuitively, enabling complex problem-solving such as troubleshooting why their car engine won’t start, or exploring the contents of their fridge to plan a meal, or analyzing a math problem simply by looking at it.

Users will still be able to generated content as well simply by describing them to ChatGPT, such as inventing new logos, comic strips, and photorealistic scenes right in the chat. All of this will be done even more intuitively, simply by dictating from the headset or looking at a real world object with the headset on and asking for information about it.

ChatGPT is available for free for visionOS users starting today, with an optional subscription for ChatGPT Plus, which gives access to GPT-4’s features and faster response times. The app can be downloaded from the visionOS App Store, where users can also find hundreds of other apps that are customized for different purposes and interests, such as creative writing, marathon training, or math tutoring.

VentureBeat has reached out to OpenAI for additional comment about the development of the ChatGPT app for Vision Pro, and we will update the article if and when we hear back.