Google’s Genie world model can now simulate real streets with Street View

We’ve all pulled up Street View on Google Maps to show a friend what our childhood home looked like, or dropped that little person icon onto the streets of Paris to see if we booked a hotel in a cool neighborhood. Imagine being able to do that, but in a more immersive, interactive way that allows you to really simulate the street and its environs, and even do things like adjust the weather or see what it would look like in a “Day After Tomorrow” scenario.

That’s one of the goals of Google’s latest integration. Starting today, Google DeepMind is connecting Street View to Project Genie, the company’s general-purpose world model that can generate diverse, interactive environments. The new feature launched during the Google I/O developer conference. 

“It’s really powerful for both the agent [and robotics] use case and for humans to play with, and that’s always been the thesis of Genie,” Jack Parker-Holder, a research scientist on DeepMind’s open-endedness team, told TechCrunch.

He gave the example of a new robot being deployed in London, which rarely sees the sun. Genie could, Parker-Holder says, simulate those scarce occasions when the sun glints off the Victorian housing, so the rays don’t shock the robot when it happens.

“Simultaneously, you might say, ‘I’m going to New York City, but not this time of year,’” he continued. “‘It’s going to be snowy. I want to see what that block looks like in the snow.’” 

Google has been collecting Street View data for 20 years via cars with cameras and individuals strapped with “tracker backpacks.” The tech giant has collected north of 280 billion images across 110 countries and seven continents. 

“With Street View, we have imagery from a large quantity of the world,” Jack said. “You can imagine how potentially powerful it is to combine this rich source of real-world information and data with an ability to simulate worlds.”

Google released its latest world model Genie 3 for research preview last August and opened up access to the tool to Google AI Ultra subscribers in the U.S. in January, allowing customers to create interactive game worlds from text prompts or images. The goal is to use Genie for educational experiences, gaming, and robotics training. 

Genie 3 is already helping to power one of Waymo’s simulators to train its self-driving cars on “exceedingly rare events” like tornadoes or casual elephant encounters. Adding Street View data to that could help Waymo prepare to launch in more cities around the globe.

Waymo has its own simulator that it relied on to scale to 11 U.S. cities and test its AI driver in several more. The difference with Genie, says Parker-Holder, is that those are all from the car’s point of view. Street View allows for not only simulating a world anchored to a real place, but also shifting the point of view to other types of agents, like a human or a robot. 

Google is launching Street View in Genie to some Ultra users in the United States starting today, with access rolling out at scale over time. Global Ultra users will gain access over the next few weeks, per the company.

The researchers’ goal is to put this new capability into as many hands as possible, per Diego Rivas, a product manager at DeepMind. He cautioned that Street View in particular and Genie in general is still an experiment, so there’s much to improve upon in terms of accuracy.

In the samples the Google team showed me — including an underwater simulation of a neighborhood I used to live in — the results are impressive and recognizable, but still video game quality rather than photorealistic. The models are also not yet physics-aware, meaning they don’t yet understand cause and effect. For example, in a simulation of a woman running through a snowy Joshua Tree, she ran right through cacti and bushes.

Compare that to, say, Google’s image generator Nano Banana — which can now generate perfect text in infographics — or its video generator Veo — which understands that paper boats drift on water currents, smoke disperses into the air, and fabric drapes over forms. 

Physics isn’t hard-coded into these models; they learn it intuitively over time through passive observation, as a living being would. 

“I think for this kind of model, it’s maybe six to 12 months behind video in terms of the accuracy and quality, so I think it’s something we will solve,” Parker-Holder said. 

Jonathan Herbert, director of Google Maps who started on the Street View team as an intern 12 years ago, said that Genie can’t yet create a faithful reconstruction of a street. He thinks the real breakthrough is the AI’s spatial continuity. If you turn 360 degrees, the AI correctly remembers and simulates the environment behind you. From that point on, the model can build a new environment on top of that.

“We have long thought about how we can build out the best and richest model of the world on top of Street View data,” Herbert said. “It’s definitely been an idea of ours to use Maps Data in new ways and for new kinds of AI research for a pretty long time.”