Apple has filed a patent for a complex mixed reality headset, potentially including sensors that track usersâ€™ facial features, eyes, and hands. Variety reported on the patent application earlier today; it was published last week and originally filed in March 2019. Itâ€™s part of a long string of Apple mixed (or augmented) reality patents, and, like its predecessors, it indicates that Apple is broadly examining how AR might translate from phones to glasses.
The â€œDisplay System Having Sensorsâ€ patent covers a head-mounted display, a collection of cameras and other sensors that track the external environment, and a different set of sensors that track the wearerâ€™s eyes. The components all work together to display a virtual 3D image thatâ€™s somehow based on the real world. Beyond that, there are a lot of potential variations. The headset could include hand sensors, â€œhead pose sensors,â€ sensors that capture detailed information about real-world lighting conditions, â€œeyebrow sensors,â€ and â€œlower jaw sensorsâ€ that track movement of the mouth and jaw.
As Variety notes, Apple describes a system thatâ€™s different from most AR glasses, which generally project partial images that still let you see the real world. At least one variant of its system uses pass-through video in which cameras capture a video feed, then the headset composites digital objects onto it and displays the resulting images on a VR-like screen. That tallies with an earlier CNET report that said Apple was testing a combination VR / AR headset. However, the patent is open-ended, and this is just one of many options.
Eyebrow and jaw sensors, meanwhile, would be invaluable for porting Appleâ€™s Animoji and Memoji avatars into a headset. We know Appleâ€™s competitor Facebook is already building VR headsets that can track facial expressions and translate them onto virtual avatars. And hand tracking is a well-established control option for VR and AR, although itâ€™s still not the primary option for any major devices.
As usual, we donâ€™t know whether Apple is building products based on these patents. DigiTimes notably claimed this month that Apple had put its AR glasses plans on hold, although that rumor remains unconfirmed by other sources, and there are reasons to doubt that Apple is abandoning AR glasses. However, Apple also hasnâ€™t said anything publicly about plans for glasses, either. Itâ€™s mostly stuck to promoting its phone-based ARKit platform.