At Leap Motion, we’re always looking to advance our interactions in ways that push our hardware and software. As one of the lead engineers on Project North Star, I believe that augmented reality can be a truly compelling platform for human-computer interaction. While AR’s true potential comes from dissolving the barriers between humans and computers, I also believe that it can help improve our abilities in the real world. As we augment our reality, we augment ourselves.

With its best-in-class field-of-view, refresh rate, and resolution, the North Star headset has proven to be an exceptional platform for representing high-speed motions with small objects. So when David asked me to put together a quick demo to show off its ability to interact with spatial environments, I knew just what to build. That’s right – table tennis.

当我们扩大现实时,我们也扩大了自己。Click To TweetWith this demo, we have the magic of Leap Motion hand tracking combined with a handheld paddle controller. The virtual ball soars through the air and bounces on the real table. And, of course, an AI opponent to challenge you.

While augmented reality table tennis is a lot of fun, it also demonstrates a key concept that’s largely unexplored in mixed reality right now – artificial skills training for real-world scenarios. In VR, we can shape the experience to optimize learning a task or behavior. AR elevates this potential with familiar real-world environments, allowing us to contextualize learned skills. By overlaying virtual indicators and heuristics onto the user’s view, we can even help them develop a deeper intuition of the system.

What if you could see the future? Using parabolic equations of motion, our table tennis demo easily predicts where the ball will go. By showing this prediction in the form of a trajectory, we now have a superpower – without changing how the game itself behaves! By keeping the physics simulation authentic, the hand-eye coordination and muscle memory built up while training in AR can transfer directly to the real world.

Our AI opponent takes our training to the next level. Under the hood, the AI paddle uses the same action/reaction logic to calculate the reflected trajectory of the ball as your paddle. In other words, its motions must be physically correct to play the game. To this end, we implemented akinematic formulation of the Cubic Bezier Curveto naturally drive the AI’s paddle to the correct position and velocity at the correct time.

A visualization of the Bezier Curves driving the opponent paddle.

如果你能预见未来呢#AR plus#AI可以在一定程度上赋予你这种能力。Click To Tweet这种特殊类型的运动路径允许我们指定在球到达抛物线轨迹的顶点后,划桨应该击中球。一个高水平的身体准确性是很重要的,因为它提供了一个例子(或榜样)为用户学习时,开始玩游戏。我们不希望我们的用户变得气馁或沮丧,如果他们的对手是传送到球或使用其他不公平的诡计得分。

The realism and physical reproducibility of this demo were built with the intent that the user should grow in their understanding of the system by interacting with it. As a medium, AR has the potential to improve how we learn about and interact with the real world. Simulations like this have the unique ability to adjust their difficulty downward to accommodate novices和向上to challenge experts in a whole new way – appealing to players at all skill levels.

Eventually, as AR systems become more advanced and lifelike, we will be able to practice against “impossibly difficult” artificial opponents and use that intuition in the real world like never before. Current and near-future professions may be aided by advanced AR training systems that allow us to casually achieve levels of skill that previously required months of determined practice.

With the advent ofcentaur chess和其他collaborations between humans and AI, we’re just barely scratching the surface of what might be possible. We now have access to abilities that our ancestors could have scarcely imagined. Though the age of swords and their utility has long since been relegated to the distant past, it is my belief that the greatest swordsman of all time has not yet been born.