Google's Starline project has a secret weapon: eye contact

After experiencing a video call through Google's Project Starline, the most impressive part of the demo was the ability to make meaningful eye contact.

Like many others, I've participated in more than my fair share of video calls over the past four years. Work moved online, which meant meetings moved online, and for a while our friends and family were also only available online. We're past the worst of this period, but the inherent downsides of video calls still seem fresh and obvious. “Zoom fatigue” is still part of our common vocabulary.

One of the many things I worry about on video calls is where I should be looking. The other person is often in the middle of my screen, while the camera is usually on top. At first my eye is drawn to the person, but in front of the camera I feel like I'm looking down. If I look at the camera instead, I only see the other person from my peripheral vision.

Can you tell I've been thinking about this a little too much?

It all became a little more real for me a few years ago when my partner moved halfway across the country to accept a job offer. I used to be embarrassed about where to look in video calls, and I was doubly so until I could move to join her, too. Standard video calls were all we had, so we made do. But it was much harder to make a human connection.

Over the past few years, Google has been building a better vision of what video calling could be, having first revealed Project Starline in 2021. The premise is interesting. Instead of seeing a 2D video of the other person, you're shown a live animated 3D rendering, created in real time, no less.

I had the opportunity to try out Project Starline last month at Google I/O and thought I knew what to expect. “3D” became a buzzword in the early 2010s after the breakout success of Avatar in 2009. Best Buy was offering demos of 3D TVs, and even some phone makers were experimenting with glasses-free 3D displays – someone remember the HTC Evo 3D?

My contribution to this trend was to pre-order the Nintendo 3DS before its launch in 2011 (thanks to the other members of the “3DS Ambassador Program”). The main idea of ​​the handheld was to split the visual output of the screen to display different images to each of your eyes. Since the Nintendo 3DS had no way to track your face, you essentially had to hold it at a specific distance and angle for the effect to work properly.

Was it impractical? Yes. But was it really that cool? Yes too.

Based on these early experiences with glasses-free 3D, I expected the Starline project to be a little shaky but doable. I spoke briefly in person with Andrew Nartker, Google's general manager for the Starline project, then sat at an empty table in front of the unassuming hardware. It's been over a month since my demo, and I still vividly remember my face lighting up with astonishment when the call started.

Andrew now sat across from me, as if a portal had opened where a flat screen once had been. I moved from side to side in my swivel chair, but the illusion did not disappear. Moving to the side simply revealed a different angle of the other person. Obviously (and perhaps unsurprisingly), 3D technology has advanced significantly since the release of the Nintendo 3DS. I assume the same cameras that recorded me in 3D also adjusted the display to the location of my eyes.

Even though my magical introduction to Starline has been lost to time, we are lucky to have a recording of Damien experiencing the same demo – and reacting almost the same way!

What struck me most about this experience was where my eye was drawn. Like if we were in the same room, I was naturally inclined to make eye contact. This change alone has made Starline significantly better than any video call I've ever made – so much so that I'm hesitant to even refer to it as a video call. Instead, it felt like a real “virtual meeting.”

Although the underlying technology was quite impressive, it was also easy to see and imagine its current limitations. To borrow a phrase from Fozzie Bear, one of the “cheap 3D tricks” allowed Starline to relatively accurately depict a bright green apple that Andrew picked up from his table. If something happens within range of what I would call the “scene,” it is properly shown to the other person. However, the stage is a bit smaller than you might think at first glance.

I didn't notice it until my host reached his arm slightly beyond, but a key part of the Project Starline illusion is a fake 3D background. These backgrounds (of which there were a few to choose from) give the impression of being in a larger space while hiding the ends of the stage. The quality of these backgrounds was actually pretty amazing.

Another more noticeable problem – and one that should surely improve with time and development – ​​is that you lose a bit of sharpness and fidelity when transitioning from real-person rendering to 3D rendering. It wasn't enough to snap me out of the illusion, but some things seemed a little blurry or strange.

However, the biggest problem with the Starline project as it exists today is that it will only really be useful for one-on-one meetings. Unless a much more complex glasses-free 3D display comes along (or the development of Star Wars-style holograms), it's only possible for one person to see the full illusion. Even if we assumed three separate Starline owners wanted to connect, I can't imagine the experience would be as immersive trying to bring multiple people together on a single screen.

Despite these drawbacks, I can't help but be impressed by what Google has come up with with Project Starline. Conversations flowed naturally, body language was visible, and eye contact was instinctive. Everything felt human.

Google knows it's on the right track and is now looking to spread the magic of Starline. The company recently announced a partnership with HP to produce Starline hardware for businesses to purchase. Perhaps more importantly, Google is also ensuring that 3D video meetings can take place in Meet as well as Zoom.