Here's some free advice: Don’t try to break into Andy Rubin’s house. As soon as your car turns into the driveway at his sprawling pad in the Silicon Valley hills, a camera will snap a photo of your vehicle, run it through computer-vision software to extract the plate number, and file it into a database. Rubin’s system can be set to text him every time a certain car shows up or to let specific vehicles through the gate. Thirty-odd other cameras survey almost every corner of the property, and Rubin can pull them up in a web browser, watching the real-time grid like Lucius Fox surveying Gotham from the Batcave. If by some miracle you were to make it all the way to the front door, you’d never get past the retinal scanner.
Rubin doesn’t employ human security guards. He doesn’t think he needs them. The 54-year-old tech visionary (who, among other things, coinvented Android) is pretty sure he has the world’s smartest house. The homebrew security net is only the beginning: There’s also a heating and ventilating system that takes excess heat from various rooms and automatically routes it into cooler areas. He has a wireless music system, a Crestron custom-install home automation system, and an automatic cleaner for his pool.
Getting the whole place up and running took Rubin a decade. And don’t even ask him what it cost. There’s an entire room full of things he bought, tried, and shelved, but the part that really drove him crazy was that it didn’t seem like automating his home ought to be this hard. Take the license-plate camera, for instance: Computer-vision software that can read a tag is readily available. Outdoor cameras are cheap and easy to find, as are infrared illuminators that let those cameras see in the dark. Self-opening gates are everywhere. All the pieces were available, but “they were all by different companies,” Rubin says. “And there was no UI. It’s not turnkey.”
At some point during his renovations, Rubin realized he was experiencing more than just rich-guy gadget problems. He was too far ahead of the curve. If anything, the problem is about to get much worse: The price and size of a Wi-Fi radio and microprocessor are both falling toward zero; wireless bandwidth is more plentiful and reliable; batteries last longer; sensors are more accurate; software is more reliable and more easily updated. As many as 200 billion new internet-connected devices are predicted to be online in just the next few years. Phones and tablets, certainly. But also light bulbs and doorknobs, shoes and sofa cushions, washing machines and showerheads.
In many cases, the effects of these connected devices will be invisible: better temperature optimization in warehouses or super-efficient routes for UPS drivers. But at the same time, all those freshly awake devices will present an entirely new way to interact with the world around you. Imagine automating your morning routine for maximum efficiency and ease: At 6:15, your alarm goes off, the lights turn on, NPR starts purring, the bacon begins to sizzle, and your motorized closet whirs and presents you with today’s perfect outfit based on your schedule and the weather forecast. That’s the idea, anyway, and a mere daydream by current standards; most people’s smart homes don’t extend much further than turning on the lights with a smartphone app. But all the tech is out there to do much more. Somebody just has to make it work.
It’d be hard to find someone more qualified for the task than Rubin. He specializes in connecting people and things to the internet. As an engineer at General Magic, a company spun off in the ’90s from Apple to figure out the future of portable computers, he worked on some of the earliest mobile communicators. Later that decade, he built a box called WebTV that hooked your TV up to the internet. (OK, so his timing is not always perfect.) In the early aughts Rubin cofounded Danger and created the Hiptop, the proto-smartphone later known as the T-Mobile Sidekick. The Hiptop had a web browser, cloud storage, and an app store before any of those things were everywhere. And a few years later, of course, Rubin scored his biggest triumph yet. He founded Android, the open source mobile operating system he’d eventually sell to Google (for a reported $50 million) and help turn into the most successful software platform in the history of the world. Android now powers more than 2 billion phones, tablets, watches, cars, and televisions.
Until May 2017, Rubin was tight-lipped about his new project. It showed up on employees’ LinkedIn pages as “Andy Rubin’s stealth hardware startup,” and the company filed for trademarks and patents under the name Henry’s Products LLC, after an employee’s dog. The actual name of Rubin’s new company is Essential. Its goal is both strikingly simple and absurdly preposterous: to finally bring the smart home vision to life and to build the next great American electronics company in the process. The team is working on a smartphone to compete with the iPhone and a living-room gadget that can control your entire home. Both hook into Ambient OS, Essential’s operating system powered by artificial intelligence. Ambient OS is there to scan your data, learn your routines at home and on the go, and predict your needs, then get your dozens of bechipped gadgets cranking together to control your environment in a way that feels natural, normal, and human.
Rubin relishes big challenges, but this will be his biggest yet. And if he can pull it off, he will be one giant leap closer to finishing what he’s actually been working on all this time: bringing the whole world online, so he can find out what happens next.
Building a smart home platform isn’t what Rubin planned to do upon leaving Google in 2014, after a decade at the company. He didn’t have to do anything, of course, other than sit around and watch a robot count his money. Not long after his departure—which was either his idea or not, and amicable or not, depending on whom you ask—Rubin sat with his wife, Rie, drinking wine and mulling over what he should do next. It’d be tough to go from building the world’s most popular software to a chatbot startup or an AI research lab. What could be as big as Android? “Don’t do one thing,” Rie told him. “Do 10 things.” That clicked. Along with cofounders Matt Hershenson, Bruce Leak, and Peter Barrett, Rubin started a combination venture capital fund and product incubator, which they named Playground. They’ve since raised $800 million in funding and have so far made 23 investments in companies working on things like augmented-reality headsets, water-quality sensors for pools, and home security cameras.
When the Playground partners started talking to entrepreneurs in 2015, Rubin began noticing patterns in what companies were pitching. Lots of little smart home gadgets, lots of VR, lots of backend tech. One thing, he realized, wasn’t coming through the door at all: entrepreneurs with big ideas for consumer electronics companies. Rubin had been wondering for a while when he’d see the seed of the next Apple. Nobody seemed to be thinking that big.
Turns out Rubin himself was. He’s convinced that the dawning era of smart home technology presents a massive opportunity: Hundreds of billions of dollars are up for grabs over the next decade. Eventually, some believe the Internet of Things industry will make the smartphone industry look tiny by comparison. And what better way to claim a seat at the table than by starting with a high-end smartphone, the device that will control some nontrivial number of the Things plugging into the Internet?
All that is precisely why Rubin is so excited the first time we sit down to talk, at the head of the large conference table where he met with all those Playground hopefuls. Rubin, lanky and balding, is decked out in flawless Billionaire Casual: faded jeans, long-sleeve shirt, unlaced sneakers. The bustling office around him is every tinkerer’s dream workshop: 70 or so people working out of a corner of a converted cannery in Palo Alto, a wide-open industrial space filled with prototyping machines and free food. Employees sit at monitors so large you have to move your head to see the whole screen. The decor is high-gadget: pinball machines, motorcycles, humanoid robots, pretty much every smartphone ever made. The staff consists of watch enthusiasts, furniture lovers, and coffee snobs. Even the toilets are high tech.
Still, within this haven of hardware, the light switches suck. Rubin can’t figure them out. The ones in the conference room consist of several identical buttons of mysterious purpose—some don’t seem to do anything. The lights occasionally go out in Rubin’s messy, gadget-covered office because he’s standing behind a whiteboard and the sensors don’t see him for a few minutes. These are the sorts of problems that drive him insane, and the first ones he thinks he can fix.
When Rubin looks at the smart home boom, he sees a near-exact reenactment of the cell phone market circa 2007. Back then, everybody had a phone, and every phone company had its own platform. Microsoft’s Windows Mobile, BlackBerry OS, and Palm OS were popular. Qualcomm was promoting its Brew operating system; Nokia had Symbian. Nobody had time to build apps for every platform, so they didn’t. When Rubin showed up with Android, the industry gave a collective shrug. Who needed yet another phone operating system? But Rubin persisted, preaching the gospel of interoperation and openness. “The fact that everybody had their own operating system is the reason you needed a new operating system,” he says. “Because you want horizontal compatibility, so you can write an app once and run it across all the devices.”
Now all the major players in tech are scrambling to be “the connected home platform,” once again pushing their proprietary services and creating a siloed landscape. Google wants all of your devices to run on its Thread or Weave protocols and integrate with its Nest thermostat. Apple’s HomeKit is the only iPhone-friendly way to control your house. Samsung spent a reported $200 million to acquire connected-home gadget maker SmartThings just to get into the market. The research firm Gartner has predicted that by 2020, more than 85 percent of connected-home devices will be linked to one of these “certified ecosystems.” But they won’t all join the same one. And because the ecosystems are so ruthlessly competitive, they’ll never speak to one another. “LG is never going to fucking plug into Samsung’s APIs,” Rubin says at one point.
If the market continues this way, you’ll be forced to buy all the way into one company’s vision, essentially ripping your house to the studs and replacing everything with Samsung-approved sensors to work with your Galaxy phone, or gadgets from the Apple Store to work with your iPhone. Otherwise, there’s a good chance your lights won’t work with your music system, and the front door and television won’t be on speaking terms. Rubin makes one point over and over throughout our conversations: If the way people interact with their connected home is through smartphone apps, the connected home will never go anywhere. If you have to open an app, log in, and tap around just to open your front door, only to open, log in, and tap around another to turn up the thermostat, nobody will do either.
And that’s where Essential’s most important product comes in. Ambient OS—Rubin describes it as “Android, but evolved”—is a universal translator for the smart home, combining all the major smart home products and platforms into a single elegant system and interface. That’s how it will look to users, anyway. Behind the scenes it’s just an elaborate hack. “I plug into SmartThings, I plug into HomeKit, I plug into Thread and Weave, and I get a hundred thousand devices that I can control with my UI,” Rubin says. In the background, Ambient’s job is to strip the barriers between devices so users don’t have to worry about compatibility. They should buy a light bulb, screw it in, and trust that it’ll turn on when it’s told to. And the code that runs the operating system will be publicly available, so outside developers can create new stuff that works with it seamlessly.
The first time Rubin explains the system to me, he says repeatedly that he shouldn’t be telling me how it works. There’s nothing preventing anyone else from hooking into the same handful of platforms, after all, and immediately stealing Essential’s entire feature set. A few such products, like Thington and Sevenhugs, are already trying to do just that. And they’ve found it’s not as easy as Rubin thinks. “There’s no problem finding thermostats that can connect to the internet,” says Thington founder Tom Coates. “It’s not always the case, though, that these companies provide openings” for developers to access. Nest, for instance, requires special permission for more than 50 users.
Essential can’t just end-around its competitors forever. Rubin’s primary advantage, as he sees it, is that he’s Andy Rubin. Manufacturers take his call. So do CEOs and high-profile recruits. Rubin is counting on his reputation and authority to keep Apple, Amazon, and others from cutting Essential out of their systems—and to integrate their platforms and assistants into Essential’s gadgets.
Even for the vaunted Andy Rubin, the road to smart home glory is filled with potholes. Essential is building its own virtual assistant to help run your home and much of the rest of your connected life—a task anyone at Google, Apple, or Amazon will attest to being harder than it sounds. Rubin’s plan is for his smart home hub to support voice, touch, gestures, and any other input you can think of. To make it all work, Essential is making a big bet that the Wild West of platforms, services, and newfangled wireless radios is coming to an end, and that it will be able to corral millions of devices through only a few protocols. Essential is, on one level or another, attempting to tackle nearly all of the tech industry’s problems at once. That ambition got the company a valuation close to a billion dollars before it even launched. But it’s also the biggest thing standing in the way of its success.
Rubin seems aware of the potential pitfalls but dismisses them out of hand. He’s Andy freaking Rubin, after all. When I wonder aloud if he’s trying too hard, building too many things, going too big too fast with a team numbering dozens rather than thousands, he scoffs. “Come on,” he shoots back, wrinkling his face dismissively. “Is it bigger than Android?”
So yes, in many ways this is the Android playbook all over again. But the thing everyone forgets about Android is that its first success story wasn’t the open platform. It was the Droid, Motorola’s keyboarded handset that became the first device to challenge the iPhone. Buyers didn’t want Android—they wanted a Droid. Because people bought Droids, people made apps for Droids. So people bought more Droids. Eventually, there were so many apps and users that the only way for Motorola’s competitors to plug into this burgeoning ecosystem of useful apps was to adopt Android themselves. And so Rubin won. The lesson was clear: Great ecosystems start with great products.
Not long after the company started, Rubin took his Essential cofounder Matt Hershenson and a small group of new employees on a whirlwind tour of the manufacturing world. Thanks to Rubin’s profile, Foxconn and other big names agreed to meet with this tiny startup. Armed with lots of ideas and a couple of early concepts, the Essential crew would walk into meetings and start with the same simple speech Rubin gave prospective hires: “We want to do something crazy. Are you in?”
The team wanted Essential’s first products to be unique, sleek curios that would seem exclusive and exciting in a sea of identical aluminum rectangles, and offered manufacturers a chance to show off their best work at a more achievable scale. They found a German company that could manufacture a titanium phone chassis, making Essential’s handset more rugged than most. Another supplier worked with ceramics in cool colors, like ocean green and a very subtle shade of lavender. They met a manufacturer that could make large, perfectly round LCDs, and companies that could make phones in wild new shapes.
The Essential team is working on lots of products. Rubin hints at plans for a smartwatch (or something like it) and a phone that looks nothing like the one you have now. He’s definitely working on something for your car too. Eventually, one employee says, there’s a robot coming. But the company is launching with two things: a phone named Phone and a smart home controller named Home. There’s no branding on the devices themselves. Essential aspires to be more Armani than Abercrombie: understated, confident, cool, in a way that doesn’t need logos.
The $700 Phone is aimed squarely at the iPhone and tries to one-up the dozens of higher-end Android handsets on the market. Rubin says it’s a niche device for a few people. Others at the company say Essential’s goal is to sell more phones than Samsung by next year. Either way, with a titanium body, a nearly bezel-less screen, a ceramic backplate, and a pure version of Android, it’s as if Rubin finally got to build the phone he’s been wanting since the early days of Android. On the back of the device, two tiny round dots hint at the Phone’s best feature: a new kind of magnetic accessory port that lets users stick cameras, microphones, and anything else right onto the handset. The company is starting by offering a 360-degree camera. Other accessories will follow, and Essential will open-source the magnetic connector so others can put it in their own products.
Mostly, Essential built the Phone because it had to build a phone. Even its creators admit their product isn’t earth-shatteringly different. But there’s no better way to worm your way into early adopters’ pockets. Someday, maybe all you’ll need is a smartwatch or a brain implant to control the house. For now, a phone gets the job done.
The Home is where Essential’s mission starts to come into view. It doesn’t look like any gadget you’ve seen before—or like a gadget at all. It definitely doesn’t look like an Amazon Echo–style smart speaker. With a round, squat body and a slightly sloped face, it’s more like a salsa bowl with a lid. Maybe something you’d plant a succulent inside. Even this, though, is more gadget than the Essential crew wants to make. “Rather than talking to an Alexa, I want the user to be able to have the comfort of knowing that they’re interacting with their home,” Rubin says.
In the team’s imagination, once your home has been properly kitted out with connected devices, there’s no controller there at all. You don’t say a wake word or turn on a screen or enter a password. You definitely won’t have to get your phone out just to turn on the lights. You just declare your needs, in whatever way makes sense at the moment—voice command, touchscreen—and they’re taken care of.
The Home uses far-field voice recognition to hear users from across the room. Rubin is hoping to integrate with Alexa, Siri, and Google Assistant, along with Essential’s own helper, so that everyone in the house can use their favorite platform. (Far from guaranteed, but that’s the plan.) The device’s face is filled by a completely round LCD—“the first round LCD ever,” Rubin says, proudly placing the part on the conference table. It’s a touchscreen, with a large-text interface full of round icons that should be readable even from across the room. The Home uses the same magnetic accessory port as Essential’s phone, so you’ll be able to augment the device with more capabilities after you buy it.
Essential will put the Home into the world long before it, or anyone, knows what to do with it. For all the hype, the smart home is still incredibly primitive. “If you showed Thomas Edison the Philips Hue bulbs,” says Chris Harrison, director of the future interfaces group at Carnegie Mellon, “he’d say, ‘All you did in 120 years was make it change color?’” Right now, devices are connected mostly to make customer service easier, or because adding a Wi-Fi radio costs virtually nothing.
Long-term, Rubin is banking on machine learning to make technology far more useful and intuitive. In many cases, you won’t have to touch or speak to your devices at all. That’s the full promise of ubiquitous computing: Everything just works. It’ll know what you want because it watches you and learns that it should start warming up the car when you’re putting your shoes on, because that’s always the last thing you do in the morning. When you say, “Tell Anna it’s time for dinner,” the system should know who Anna is, which room she’s in, and which speaker to use to alert her. The only way for that to work is if absolutely everything is connected. And, Rubin says, “it has to be 100 percent right every time. Because that one time it’s wrong, the consumer immediately loses confidence in it and wants to rip it out.” He says that Essential, and everyone else, is a long way from realizing that vision.
And even if it does work eventually, Rubin will need to reassure users that the suite of always-listening devices tracking their every move is not a threat to their privacy. That’s why Essential built the Home to do a lot of its work on the device itself, without sending data to the cloud. Rubin, worryingly but perhaps unsurprisingly for the founder of Android and a longtime Google employee, isn’t terribly concerned about the privacy issue. Mostly, he says, it helps that he’s not selling your data. He doesn’t even want it. He’s selling you products, not ads.
Years ago, when he was still at Google, Rubin gave a presentation to the company’s board of directors. It had a single visual aid: a picture of a brain in a jar, probes stuck in at every angle. That’s Google, he said. You have no arms, no eyes, nothing. You don’t understand the world outside. When someone sits down at the keyboard and types a search query, that’s all they get: the brain inside the jar. “What you need to do is to get in the real world,” Rubin told the room. “You should build mobile operating systems for cell phones. You should do robotics. You should build self-driving cars.” These things would connect through the internet but live in the real world. That, Rubin thought, would teach these computers how things really worked. Artificial intelligence can only be truly intelligent when it gets out and smells the fresh air.
He’s best known as a software creator, but hardware is Rubin’s true passion. He lives for the pioneering stuff. He drives a black Acura NSX, the supercar that launched in 1991 as the first car with an aluminum body and the first engine with variable valve timing. A Boosted electric longboard rests outside his office door. Inside, two super-high-end Devialet Phantom speakers point toward his desk.
Most of all, Rubin is obsessed with robots. He began his career as a robotics engineer at Carl Zeiss. When he stepped down from Android, he started building humanoid robots for manufacturing and retail use. Even now, a large, faceless robot sits ominously in Playground’s lobby. But the longer we talk, the clearer it becomes that the world has always misunderstood what Rubin meant by “robots.” During a speech in 2016, for instance, he proclaimed that a rice cooker is a robot. It knows the temperature of the water and brings it up to the right one. It knows how long it’s been cooking and when it’ll be done. And when it’s done, it alerts users. It follows the three-step rule for robots: sense, plan, act.
With Essential, Rubin’s not building a robot with legs or a face. (Not that he’ll talk about, anyway.) Instead, he’s trying to help turn as many things as possible into robots that can sense, plan, and act, and then give users the tools to command the robot army. It starts with a simpler, saner way to control your home, but it doesn’t end there. “I don’t want to call it a Trojan horse approach, but anything that involves machine learning is going to get better over time,” Rubin says. “So the best thing to do is to put something in the home that the consumer is already comfortable with, and then expand.”
Just before we finish talking, he stands up and scours a floor-to-ceiling set of shelves filled with his inventions. The Magic Link, with its modem jack. The Hiptop, which put cloud-connected apps in people’s hands for the first time. The Droid, which helped kick off the Android revolution. And now the Essential Phone and Home, two products with one purpose: to connect people to the internet and the world around them through computers that sense, plan, and act. Rubin has been working his way to this point for a long time. The robot revolution is upon us, and it’s sitting quietly in your living room, awaiting orders.
Senior writer David Pierce (@pierce) wrote about the limits of AI assistants in issue 25.06.
This article appears in the August issue. Subscribe now.
Listen to this story, and other WIRED features, on the Audm app.
