Apple Vision Pro: Startup Platform Of The Future? [Lightcone Podcast Ep. 2]

interesting stuff Apple did is with the hardware in the Vision Pro versus the software? Well, you need to understand the real world in order to augment it. Technology of a self driving car but on a headset. This is maybe where founders should sort of pay attention. Is this a good opportunity for startups? There's all kinds of new interactions that I think we have not figured out yet.

What really truly takes advantage of this platform? The dream has always been to get to something like this. Welcome back to another episode of the Light Cone. And as you can see, it's not just any other day in tech. There are some new platforms that are coming up right now. You might have seen other places where there are reviews. We're not doing reviews today.

We're gonna talk about what these platforms might mean for founders and people who wanna build things for a billion people. We actually have an expert at the table right now, don't we?

We do. Diana, who's a group partner at YC, before she worked at YC, she's been working in AR and VR for ten years since the dawn of the Oculus before VR was a mainstream thing. In fact, her grad school research was in computer vision, so she's been interested in this from way before it was a thing other people were following.

Diana, do you wanna talk about your startup that you did, which was an ARVR startup, a really early pioneering one? Yeah. We went through YC with a startup called Azure Reality.

What we were building was a augmented reality SDK for game developers so that they could build multiplayer experiences and AR games and build the code once so that it would work on any platform. So between not just iOS and Android mobile device. But the dream has always been to get to something like this or that or that so that developers would write the code once and work across all devices.

And what happened to your startup? So what happened is this took a lot longer to come to market. That's one thing. The other thing that ended up happening, we ended up getting acquired by Niantic, the makers of Pokemon GO. So I ended up heading a lot of the AR platform over there at Pokemon with Niantic, and we ship actually a lot of this AR SDK into a lot of games.

So millions of players are running our code, which is really cool. So if you've ever played Pokemon GO, you've literally used code that Diana wrote. And I'm so excited with this platform coming in, and we can go dive.

deeper into it. Okay. Should we take the headsets off so we can we can talk? Yeah. Let's go ahead.

So it's been a long road. You've seen this technology basically evolve over the course of a decade. What's you know, why AR? Like, that's one of the big things here. You know, previous platforms may be really focused on VR and the gaming aspect. HoloLens from Microsoft seem to try to do the AR thing. What's going on with the Apple Vision Pro? You know, why is this important?

Why are we talking about this?

Yeah. I mean, we have to go even back in the history of computing. Actually, the attempts of building augmented reality and VR headsets have been actually since the beginning of the first computer. Actually, very first one was by this guy called Ivan Sutherland back in the sixties. So people have been thinking about it.

It's kinda the one of the dreams and it's one of those things that really fascinated me. I think it's so much of it is in our consciousness that we want to make it really happen. But the challenge why it has not happened unlike tablets, phones is that it's just really, really hard to make. So you bring up the Microsoft HoloLens, they had version one and version two.

And sadly, the latest version got scoped down or the team kinda got let go because they tried a optical approach. So the AR approach was the act they were seeing actually the real world and then the digital content would be rendered just with, within the eyes and have very little field of view. It was actually the same approach that Magic Leap was trying.

And what Apple is trying is actually more of a pass through, which is actually more of a full high risk video feed of the real world. And arguably, lot of the technical challenge are a lot easier. And the hard part of optics is that is not a problem of Moore's Law and just like brute forcing with more computation, more pixels.

It is actually figuring out new physics and photons so that they render properly to the human eye because the human eye is actually very incredible. Your field of view is actually 210 degrees, so you put your hands behind your ears, you can kinda see them.

And to have a display system that can really render all of that is so hard, and the other part that's really hard, which I wanna touch upon a bit more, is our eyes incredible at doing infinite ability to focus. So we can look close here or very far. And in some senses, you have to find a way to make that discrete for computers to work. Right? Because computers just understand ones and zeros.

And to get that working in a display is just so hard. And Apple has done some clever things with that. That's different to the optical approach.

because the optical approach is.

what like, it's actually looking through to the real world or it's how how what's the difference? Yeah. So if I'm looking at Jared right now, I'm actually seeing Jared. And if I overlay a digital digital information in the optical system, I would only overlay the digital information.

In here, for the VisionPRO and what the MetaQuest three or MetaQuest Pro or the VisionPRO, technically VR headset, the full video is all digital. Okay. Like,.

Jared is technical technically pixels when I see him through the Vision Pro. And so you said, like, the Apple Vision Pro being a video feed actually reduces the technical challenge?

Yeah. Because I think, there's a couple of things you could do. You can play a lot with the video feed. And one of the cool things, if you're really best in the world with display technology, what Apple is, you can get away a lot with it.

And one of the cool things they've done and foundations of what they built, which is actually helpful if you're gonna build apps here, so much of it is built upon eye tracking. So they actually have a variable rendering for focus. So they had to get the eye tracking to be working so well for this to work.

So in the VisionPRO, wherever you look, the pixel density of your focal point will render more high fidelity than where it's not. And the reason why this is important is because to fit it in such a small form factor and not to burn and there's so much heat dissipation to push so much pixels and battery, you had to do trade offs.

So they did this thing of, rendering more high res where your eye focuses. Is. So you can notice a little bit in the periphery with the VisionPRO where it's more blurry or a little bit. It's it's not like quite pixelated but blurry. And some of the people do complain online with the foveated view.

I mean, that's I think, a bit of the artifact with the with the lens, but that's, like, a different discussion. That's So how much of, like, the hard, interesting stuff Apple did is with the hardware in the Vision Pro versus the software? I think the cool thing about them is, is both because the VisionPRO is sort of a culmination of a lot of the ecosystem of what expertise they built in iPhone.

Like, they have custom silicon. They have the r one processor, which is a coprocessor to the m two. The m two is basically the same processor that runs on the MacBook Pro, so very beefy. But that processor m two is for regular, kinda like a CPU regular workloads. But the challenge for, building an AR headset or AR in general, you need to understand the real world in order to augment it.

And for that, you need a lot of sensors. So this has over 10 cameras, even has a lighter. It has a true depth camera. It has a bunch of IR cameras inside to track your eyes. So that's a lot of data, a lot of high data bandwidth that it needs to process. And underneath the hardware, I think this, you're gonna get throughput blocked.

So the r one is a custom processor that process all of the sensor data with very high data channel bandwidth. And I suspect they they are even running a real time operating system along the Vision OS, which is kinda interesting for what it means for developers to process all of this in real time. And it's starting to sound a lot like actually a technology of a self driving car but on a headset.

Yeah. That's exactly as you were talking about what this is, and that springs.

to mind, like, LiDAR plus a bunch of cameras and processing the video feed. Yeah. Can you draw the connection? Like, it's probably not obvious to people what the connection is between, like, VR, AR, and self driving cars.

Yeah. Actually, this is this was one of the jokes with my cofounder when when we started Azure Reality with a cohort tech for localizing in the world and knowing where you are. It comes from the world of, in robotics called SLAM, simultaneous localization and mapping. So you wanna find where a robot is in the world based on just visual data.

And that is the same thing that self driving cars look to navigate where they are in the three d world. So you notice in a car there's three d lighters. There's radars. There's a bunch of cameras. Same thing here to know where you are in the world. So it's the same technical challenges, but with so much more hardware complexity because you don't wanna burn people's head.

Yeah. With.

this, imagine because the the self driving car with self driving cars, could actually the actual hardware that runs in self driving car processing, they put server grade GPUs and CPUs, which fits in, like, the trunk or underneath. But this is actually pretty cool what they've done, and they built a lot of that because on iPhone, they learned how to build custom processors.

They built the with the true face true true three d on the camera, which is like IR for mapping a three d.

And LiDAR, they added on the latest one latest iPads, and they've been building a lot of ecosystem one by one. Yeah. It's interesting to hear you talk about how Apple can build on their previous products. So it's like you're saying this is sort of a lot of the technology here is coming out of the iPhone. This sounds like they sets them up to build their car, like,.

pretty well. Same expertise. Let's talk about the use cases a little bit. I mean, one of the things that's pretty clear in everything about the launch of this is it's focused on productivity. And I kinda like it because, you know, when you're talking about these Oculus devices, they're much more focused on gaming, on VR, where you're sort of in a totally different place.

Whereas, you know, my guess is, you know, one of the reasons why VRAR hadn't been embraced is that it wasn't something that a busy person would use every single day. But now, you know, it's got the m two, it's the same chip that I have in my MacBook Air.

I can actually, with a keyboard, do all of my work all day if I wanted to And that's a really big difference in how they're positioning this device.

Which is a big departure from Meta. Meta is so much on the gaming community. And actually, there was a I think there's a bit of a uproar from the VR community that there's no controllers.

And Apple has really focused full on on productivity, which I think if this was my dream when we started Azure that if AR was gonna happen, we're not gonna notice it because it's gonna solve all the very mundane things and it could replace all screens. I think if done well, this is going after the market cap of all screens that get sold, if done well.

I mean, there's still a lot of things to be done. This is still be zero, but yeah.

But this this motion, like, this is incredibly natural and being able to look at things and have it be something that you interact with. I was just blown away at how simple, how easy that was to reprogram my brain. Which is cool. I think there's half I remember, I guess a question for you, Gary. Do you remember when.

the iPhone came out, Apple had this human interface guideline? Mhmm. Yeah. They had a lot of things about communication.

information hierarchy with touch and focus and gestures with your thumb and things like that. Yeah. It's an incredibly comprehensive document. They basically took all of the learnings that they had gotten building the iPhone for years, and they distilled it into a really thorough document. Then they published it for everyone.

I think it taught a whole generation of designers and developers how to build great mobile apps. They would just read that document. There is a human interface guideline for the Vision Pro.

And, one of the things you notice is so much of it is about eye tracking and communicating communicating information with depth and space. And I think what brings maybe this is actually something for founders to think about if you're building an app in this space is that with the VisionPRO, they invested so much on eye tracking to make it work for so many reasons.

I mean, we talked about to get just the rendering to work. That was a building block. But for the UX, I think it is the moment that we're seeing with capacitive touch where Apple got it right for the iPhone. The eye tracking is starting to look a lot like that. So I think there's a lot of cool UX. Things are yet to be discovered with just eye tracking. And the funny thing is that the VR community,.

I think, was very skeptical of this because, actually, it was actually a bad practice to do eye tracking because it tires the user too much. And the reason is because its hardware was not good enough. I remember the same thing before the iPhone came out.

I remember, like, a lot of the conventional wisdom from consultants and experts was that the virtual keyboard wouldn't work, that people wanted, like, a physical keyboard and that just it wouldn't like, people would never treat it as, like, a serious device to do their email on because it didn't have a real keyboard. On the phone? Yeah. Oh, yeah. Yeah. Yeah. That was all the reviews of the iPhone.

Yeah. Yeah. But there were I mean, this is maybe where founders should sort of pay attention. There were still things that Apple had not figured out yet that third party developers ended up figuring out. So if you remember the pull down to refresh, that was something that I think was in a Twitter client. Oh, yeah. And,.

you know, that I think that founder ended up selling their Twitter client to Twitter and working at Twitter for a while. But there there's all kinds of new interactions that I think we have not figured out yet. Like, the sort of like pinch to move around is merely the first of a whole bunch of different things that frankly, end user developers will actually figure out, I think.

I'm curious also, Diana,.

What's the difference for a developer between the Meta SDK and the Apple Vision Pro SDK?

One of the big ones is Meta comes from the DNA of gaming. So they have very good support for Unity and Unreal, those are game engines, which are cool to build for games, three d environments in a game, which are little literally more like a constrained three d world. But for spatial or a spatial computation, the real world is infinite. So sometimes game engines don't quite fit.

And one one of the things you'll notice, to build an application that opens a PDF for the Meta for the Meta platform, it actually takes a lot of lines of code. Whereas to build that for the Vision OS, it's actually just few lines of code. Interesting. I guess the other big question, that probably a lot of people in the community have, is this a iPhone moment or a Newton moment?

Well, when the iPhone first launched, there wasn't actually an App Store. Right? So I think that came maybe a year later, something like that. All of the initial apps that got distribution on the App Store were, like, frivolous apps. Right? It was like the Fart app. There's like a bunch of things that were getting really popular. The $2,000 I am rich app.

Yeah. It was like a image of a ruby or something. Something like that. And if you think about from our, like, at least the YC perspective, the iPhone or mobile didn't start driving really big companies being started until, I would say, probably, like, 02/2012. Like, 02/2012 is the year where we had Instacart come through. I actually think mobile was a fairly big component of Coinbase. Right?

Like, had the fact that they just had an easy to use mobile app. DoorDash was 2013. And so all of these things start and then you have of course, you had the rise of Uber, an OYC company. But it took, so you could say, five years from the launch of the iPhone for For the actual good companies. To even be founded. And so So you haven't missed it yet? Yeah.

Well, I don't when I'm when I think about the VisionPRO, I'm not sure if we're at, like, is this the iPhone moment in the sense of the iPhone just got launched and, like, it's still gonna be a few years? Or is this like, hey. Actually, like, this is this device has been around for a while.

This is just like the iteration that was needed on it to unlock, like, the Instacarts and DoorDashes and Ubers that are gonna be built on it? I'll give one argument for why it's probably more like the iPhone moment. We don't know. But,.

you know, when the iPhone came out, like, people forget smartphones were already an established category, and the iPhone was, like, the new entrant to this, like, established category. A lot of people were skeptical that Apple could actually execute as you and as you mentioned, were very skeptical of the iPhone as a as a as as the right product to challenge the BlackBerry.

and the other, like, incumbent smartphones as famous Steve Ballmer quote about it. I think there's, like, Steve Ballmer just, you know, making fun of it and saying it would never be a serious device. Right. Right. Right.

Why was it that it took, like, five years for the good iPhone companies to come out? I think adoption had to happen. So that's why it actually maps very closely. I mean, I don't know how many Apple actually sold, but it's probably on the order of Hundreds of thousands. Thousands. Right? So probably mirrors the iPhone. Maybe the iPhone, you know, broke a million.

Even when you look back to the Instacart or DoorDash or Uber moment, these mobile workforces could only happen at the moment that 70 to 80% of the people in society had these devices. And the reason why that was such an important moment was that it was the first time normal average people had always on internet connectivity and an app ecosystem that was actually stable enough.

You know, remember back, you know, the sort of ten years before it was like j two m e or do we write it in Flash? You know, Gustaf and his, you know, Voxer and Heisan experience, you know, the platforms were literally so broken and so fragmented that you couldn't have 80% of the population on one platform. And then suddenly, all of the platforms sort of coalesced,.

then it opened up the market. I guess the question with this device, and in general with VR, it will be different than mobile. It won't be a type of device perhaps I mean, it depends on the price point when it gets to maybe phone cost perhaps, but it will take up a lot of time before we get that level of mass adoption.

But I think what could happen is it will capture a lot of the kinda high end use with what we talked about earlier with high information density, construction, CAD engineering type of workflows?

So Diana and I were actually doing group office hours yesterday with a group of our companies in this current batch who are all working sort of hardware, hard tech ideas. And we did this exercise. We call it the premortem where you sort of give them different flavors of how companies can die. And you get them to say, this is like how I think I'm most likely to die. Right?

And, like, the one I'm coming out the thing that springs to mind here is we were talking about how Tesla's strategy was very successful to launch the Roadster, like a very high end device. And then you bring out, like, the Model s and the Model three and the Model y. But, like, that wouldn't have worked if they just stuck with the Roadster. Right?

And so maybe one failure mode for the Vision Pro is, like, this is the Tesla Roadster. It's great. It carves out, like, a niche for people who are really into this stuff and are willing to pay, like, for a very high end device, but I can't follow it up with, like, the Model three.

I think this is a bit of a chicken and egg aspect with it because for this to be relevant to become the model three, let's say, we need a ecosystem of applications and incentive for developers to work on it. Because if I were a founder right now and I'm looking for a new idea, do what do I wanna put all my eggs on here when there's not enough user yet? When should I do it?

Should I just take a leap of faith on.

how do we advise founders when they're in this space? Like, why should they do it? I definitely think that's relevant to, like, the Instacart DoorDash thing, for example, if you think about it. Like, those companies weren't making a bet. Like, their apps were not specific to iOS or Apple. Right? Like like like, everybody had a device. They worked equally well on, like, Android.

They frankly, they could have just been a web view stuffed, like, in an app. Right? And so.

that's a good point. That's And they also weren't the first entrants in their categories. Like, before DoorDash and Instacart, there were many would have like, would be DoorDash and Instacart.

players that launched earlier that actually didn't succeed. Yeah. Well, even more extreme, like, they the in their case, mobile actually made ideas that seemed very bad, like good ones. I I actually think it's really cool that Sequoia invested in Instacart because they'd had the big failure with, like, Webvan.

And so they had all this egg on their face with, like, grocery deliveries, this bad idea that, like, you would expect is very natural to never wanna fund that again. But, like, mobile actually turned that into a good idea.

I did a dinner talk with with Max, the cofounder of Instacart, and he said that when Sequoia led the Series A for Instacart, they gave him the web fan business plan that they had been given in the nineties. But the problem was it was on a floppy disk, and he couldn't find a floppy disk reader, so he never read it. That's.

hilarious.

I'm sort of taken by even the path of consumer social networks. You know, Facebook started as the blue app. You know, it was a desktop experience killing MySpace. It sort of looked like literally bank software, like if you logged into Facebook or, you know, chase. com, it even had the same color.

And they I I remember being at YC when Mark Zuckerberg came to talk about why they bought Oculus, and it was actually very much from what I could tell trying to fight the last war that Facebook had just bought Instagram, I think it had not bought WhatsApp yet, and they felt he felt really scared, like, that basically Facebook had this monopoly.

It had, like, owned the industry of, you know, consumer social, but then they almost lost it because Instagram, you know, easily could have outstripped it. And that was because of a platform shift. And so he wanted to, you know, very clearly own the next platform,.

and he's right. Should founders go build on this? Is this a good opportunity for startups?

I just sort of wonder what are the things that could actually fully take advantage of this in, a real sort of professional context. I mean, where my head goes, maybe it's too obvious, but traders with their, like, sort of 20 screens, you know, wouldn't you rather have something that allowed you to take in the breadth of that information and dive into it very easily just by going like that?

You you can imagine that being something that people are actually willing to pay not just, you know, hundreds of dollars a month, but maybe thousands of dollars a month for. I think we're gonna be in.

quite some time at the beginning in this awkward phase with spatial computing type of apps. Because even with the Apple SDK and Meta, a lot of things are still flat two d. And I don't think we know how to develop or develop for full three d.

What really truly takes advantage of this platform? What is unique about this platform, whether it's, you know, three sixty degree view, being able to dive into more data easily? Like, what are aspects of this new technology that mean that it can upend even what seems like an unassailable incumbent, like, you know, Snapchat versus Facebook?

But would part of you try and talk them out of it? Like, would part of you be thinking this is too early? You should work on something else and not I think if you look back in our history, YC has really been pretty good.

at this, where every time there's a platform shift, whether it's like the Facebook thing, which didn't go anywhere or the iOS thing, which did go go places. We were reasonably accurate at actually funding the right stuff.

And I think the way that we did it is rather than having a strong thesis on each technology and each platform, we just kinda look at each application from first principles, and we talk to the founders, and they have some idea. We just try to figure out if the idea makes sense.

I think that's what allows us to have had a pretty good track record of discriminating people who are just, like, cargo culting the new thing and just, like, jumping on the hype train and have some idea that doesn't really make sense from the people who are building something like DoorDash that actually, like, totally makes sense.

Yep. That's fair. I I mean, the other thing that I would look at, to Jared's point, is actually there's a strong belief from the founder that they wanna make a bet in the space. I think there's just something about founders where they go all in. They become unstoppable.

And it's gonna take time, so they have to have the faith that this is gonna be different than building, let's say, a standard SaaS application or consumer app or AI application, let's say. If you stick long enough, you're gonna build a lot of expertise.

and be world class by the time is the right moment. But someone is genuinely excited about it. And the cool thing about it, there's a lot of technical challenge with it, which I think is gonna attract the right kind of founder because it's actually hard to build something good on this right now because it's so new.

So this will be the main thing I'll look for when I'm reading applications where people putting VR stuff, actually. And I feel okay sharing it because it's very hard to fake. It's basically what we're saying is if you're the kind of person that just is irrationally compelled to build applications for VR, we will happily fund you. And we'd need some evidence of that.

You just spend your spare in your free time, you are, like, building VR apps and you have been for a while. Like, yeah, we would never try and discourage founders from building stuff they just think is cool.

Well, that's a great place to end. We're out of time, but thank you, guys. Another good episode of the light cone, guys. See you next time.