Exploring the Metaverse
SUMMARY KEYWORDS
metaverse, hubs, game, world, Croquet, platform, web, vr, events, portals, invented, space, people, real, called, simulation, experiences, iFrames, questions, xr
This is a transcript from a talk I gave at the Meet the Metaverse Workshop hosted by https://activereplica.io in their variant of Mozilla Hubs on Tuesday, July 12, 2022. I decided to have this transcribed and it is lightly edited by me. Once the original video is available, I will add that here.
SPEAKERS
Michael Moran, David A. Smith
Michael Moran Host
I’m so happy to introduce David Smith, who is the Chief Technology Officer at Croquet.io. And David is going to talk a little bit about the company but what I’ll say is, from my perspective, Croquet has been getting a ton of buzz lately. They won the startup pitch competition at the AWE conference. And they’ve just been posting the most amazing demos online, a couple of which we’ll have to show you here today but welcome thanks Can you tell us just a little over overview real quick? Yeah,
David A. Smith
I’ll give you a little history actually, because it goes way, way back. I started doing what we think of as Metaverse and VR about 35 years ago, actually, I did it in sort of my favorite version of what we think of as telepresence. So I had a robot — a Unimate Puma 560.
David A. Smith
And I built a head mount display, and then I had a data glove (a glove sold by VPL that tracked hand motion) So I could use that control the robot arm. I was in the same lab as it was but I was literally kind of distanced from it. And I could see that my hand became the robot’s hand, I could do eye/hand coordination, I could pick physical things up and move them around, even though I wasn’t in that same space. So it’s kind of the next step beyond what we’re thinking about with the metaverse today. You are going to be able to transport yourself not just into a virtual world but another real life world. But after that, I got totally obsessed. And I am still totally obsessed by this stuff. I decided that I wanted to write a game that would utilize these ideas. So I wrote the first real time 3d Adventure shooter game. It was called “The Colony” and came out in 1987. There is a new documentary about first person shooters coming out this year and I was interviewed for that.
Michael Moran Host
David, I’m just going to interrupt real quick. Yeah, it sounds like it sounds like we a couple of people might be having trouble hearing you. Can we just move to the front of …
David A. Smith
Sorry. That’s okay. I can hear you. Come closer to me too, if that helps you. I know we’re doing spatialized sound. So yeah. So I wrote the first 3d adventure/shooter game called “The Colony” and actually there are the two things from that that are very Metaverse related.
First was Jim Cameron got a pirated version of the game. He was working on a movie called “The Abyss” at the time, and one of his problems was he didn’t have a sense of what the set was going to look like before they built it. So he asked me to build a virtual version of the set. This was 1988, believe it or not, was that 32 or 34 years ago? Is that right now? Yeah, I guess it was. Yeah, anyway. So I built that set and they figured out that a whole section of set didn’t show up on camera. He also was able to kind of design the shots before the set was even built.
After that, I took that idea space and created Virtus Walkthrough, the first real time 3d design tool for PCs. In fact, if you’ve ever heard of VRML — Tony Parisi and Mark Pesce invented that as a standard, Tony, later contributed to the GLB format that we use now. And so Walkthrough was the first design tool that actually exported VRML. We were very proud of being part of that. Another thing that happened was Tom Clancy got in touch with me because he got really obsessed with my game. He and I co-founded a game company called Red Storm Entertainment. And after he introduced me to the FBI hostage rescue team in Quantico, VA, I came up with this idea for a game that turned out to be a “Rainbow Six”.
I believe gaming is still going to be a major tentpole of this field. But anyway, the last thing that happened was Alan Kay, who’s considered to be the father of the personal computer — he invented object oriented programming. He got in touch with me, and he and I started talking and talking what what’s next in computing? This is the guy that invented overlapping windows. Imagine that, that somebody had to invent that, right? I mean, you use it all the time. You don’t think about it, but there it is. Somebody invented that. It was him — and his partner Dan Ingalls invented pop-up menus. This is all the work that was being done at Xerox PARC that really invented the modern computing system. But he and I are talking about what’s next, what are we missing?
And so we started a project called the Croquet Project and we brought in David Reed. David’s thesis was this idea of replicated computation, which is the basis of what Croquet became. But he’s also known for creating the UDP protocol, and co-created TCP/IP. So in a sense, when I talk about this idea of what the metaverse is to me, part of it is this as a missing protocol of the Internet.
The other part is kind of what the metaverse really is. And I know people think of it as a place you go, but that’s not right. I don’t think. And by the way, I, I have a tendency to ramble. So please stop me if you want to ask questions, or if you have opinions. But what the metaverse really is, is a communication platform. It enables us to share, explore, communicate, ideas that weren’t possible before. And with that in mind, it was like, what is the real platform for? We have a lot of focus on what the 3d experience is, we see these beautiful worlds. But you think about multiplayer games, that, the vocabulary of what you can do in those worlds, the kinds of things you can express, and it’s pretty simple. You can move, you can shoot, and you can kill or die, depending how good you are. And, and that’s it. And so, what should the vocabulary of the metaverse really be? And it was pretty clear to Alan and to me that it’s to be a place to shared interactive simulations. And when I mean simulations, it’s not like, Oh, we’re gonna do this complex thing, but really, everything that is interactive, is a simulation. In fact, a computer, the way to think about the modern Windows based computer, you’re doing really a simulation of paper on a desktop.
You know, that’s, in fact, how it was thought of. They call it a desktop, right? And why do you think that is, because there’s all this stuff, there’s stuff that looks like paper that’s sitting on top of it. And that was the original idea! Well, when you get to this next generation, the simulation is far more interesting, far more sophisticated. And far more powerful — especially when we have AIs that are emerging. I have an ability to express an idea. And the computer is no longer just passively waiting for something to happen. It listens to me, like, maybe Alexa does, or Siri, and it generates a version of that idea right in front of me. More than that, it generates a version of that simulation in front of me AND you. So now we are able to engage with that simulation, play with it, explore it, and we’re doing it together as if it’s, we’re in the physical space, you know, playing with some kind of real toy. But it’s virtual, it’s not a physical thing but it is a real thing. So the ability to then explore this kind of instant, simulated world is really interesting and very, very powerful. Because what we’re really doing is when we do that, we’re expanding the scope of how we communicate and what we can communicate. That’s why this is so important. What we’re doing right now is very useful — me speaking to you in a virtual space, but it’s sort of like old school. I’m talking to you. And the amount of information I’m able to convey is pretty much the same as if we were face to face — maybe even less, because you can’t see my body language.
Michael Moran Host
But we can share your video ...
David A. Smith
No, I’m happy with this. I want to actually keep it the way it is because I can make a stronger point that way. But what we really want to be able to do is explore idea spaces together, we want to create together we want to think together. And so we built Croquet as an operating system to enable just that. And so when you think about what should a system like that be, and you know, it should allow you to not just share the visuals of each other and share those things. It should allow you to dynamically construct that reality and should allow you to explore that complexity. And furthermore, you think about communication, you hear what I say. And on occasion, by the way, you know, it might be muffled or I might have a bad speaker image. But what will happen is you hear what I say you understand it as I say it, and that has to be true of everything. And that means that if I’m trying to communicate something and you’re not receiving that communication, then you simply stop trusting that communication, you won’t rely on it. And that means that whatever I say you have to understand, that whatever I do you have to see me do it as I do it. If I modify some simulation, that modification should be exactly the same for me as it is for you as if we are, like I said, interacting with the same physical object. And so that meant to us that we had to come up with a way to ensure that what you see is what I see, and what you do, I see you as to do it. And that led to this idea of what Croquet is, is a replicated, shared simulation. In other words, Croquet was built on this foundation that everything is running bit identical, what you see is what I see, any interaction I have, including physical interactions are identical, bit identical. And it’s the only way to do that, by the way, you can’t layer that synchronization on top of existing systems. That’s why it is an operating system. And so we designed Croquet as an OS, and built the our Microverse platform on top of that OS, I think we have a video here. Don’t worry that we can kind of show that over to my right, I guess to your left.
David A. Smith
I’m going to narrate this video — two browsers side by side. And we’re walking through a world and there’s a number of objects in here.
This is a simple one little bouncing logo. And what you see on these two screens is exactly the same when the user on the right grabs it, it bounces around.
This next one is actually a multiplayer space war. So there’s two players. And you can see see, it’s actually multiplayer asteroids.
This is multiuser text editor. And by the way, when you have this replicated computation model, then text is instant. It’s not like a CRDT or operational transform, which is at some point.
This is a live Bitcoin visualization, you can actually see the state of Bitcoin right that instant.
Now this is where it gets really fun. That Flamingo is a live simulation. He’s spinning around, he’s flying by just clicked on him that paused him now grab them and pull them down. And what we’re going to do, this is the actual code that’s running that flamenco. Now, this is a shares per system. I’m modifying that code right now. And what happens is when I do that, now, the flamingos flying backwards, I changed the direction of this flight.
This is a live capture of every commercial almost every commercial flight notice as a Ukraine that you just saw right there. There’s no planes on top of Ukraine. So I spin that round.
And the last thing we’re going to show you here is portals. And in this case, we have actually live portals between spaces. All those portals you saw earlier, are kind of a weak version of what we’re after. You could actually see from one, one domain into another really, and that’s what’s going on there. You’re literally it’s a stacked iframe, now it’s looping, we can stop that. But when we think about what the metaverse is, like I said, my primary goal was to ensure that its real value is as a communication platform, that what you see is what I see you are able to engage with it dynamically. You’re able to share anything all the way even to the programming level, as if it’s a simple shared reality. And that’s the that’s the first part.
The other part is the metaverse is there’s only one. And they all are linked. Now what’s wonderful, you know, you think about why is the web called the web. Because it’s a web is basically pieces of a fabric created to connect one piece to another to another to another. And so the web is one big infrastructure that’s all connected together. And the metaverse is basically really the web. And so that means that we have to have live links between these places. And that’s why portals are so important. By the way I invented portals in 1988. It seemed like an obvious thing at the time. And it’s even more now. And so when we figured out how to do portals on the web, it was like, Oh, shit, we finally got it. So what we see in the near term actually, is being able to link any Metaverse world to any other at least on the website, which I think is where the real action is. What that means is, in the near future, you’ll have a (Mozilla) Hubs world connected to a Croquet world connected to a Frame world and there’s no reason you can’t do that. In fact, we made all that technology that you saw there open source. Can you show that other video?
Michael Moran Host
We did, it’s on your left the multi plane.
David A. Smith
So this is a multi plane portal. This is the same technology I just showed you. And what you can see is the left hand side is going to flip and you’re actually looking at two different iFrames stacked on each other. And so the right hand side, you can see we see the little bunny there. But watch this, when we go into that portal, we actually flip the order of those iFrames. And that’s how multiplanar portals work — they are named after the technology Disney invented for Snow White. You literally have a camera looking down through these multiple planes. So that’s why that’s magic. And also a very important part of that is it is absolutely secure. Web browsers are designed so that when you open multiple tabs, the worlds and the applications in each of those tabs can’t cross connect. And the reason you want that is because there might be malware in one of them or you know, who knows what people are gonna put in these things. And so it turns out iFrames are just a kind of tab really. And by doing it this way means that each of those worlds are protected from each other. So you can’t, you can’t have code malware in one world that would infect the other unless you explicitly transfer it yourself. But being able to display it like that, is perfectly safe. So that kind of gives a context of what we’re building and why we’re building it that way.
And one last point is this. You know, a lot of the hardware providers are hoping that you never want to leave their worlds. And you know that that makes sense to a certain degree if you’re them. But you have to think if the metaverse is in fact, a communication platform, you have to think of it more like your traditional phone. If you pick up your phone and dial random number, you’re probably going to reach somebody. So every phone no matter what brand can call every other phone. And if the metaverse is a communication platform, then in fact, it also has to have the same capability anybody should be able to share with anybody else inside of the metaverse. That means that in the future, I may be wearing an Apple device, you may be wearing a Google device, they may be wearing a Meta Device, but they all need to communicate, and the only commonality across all of them, well, my hope anyway, is the browser. And of course, the browser that you know, that’s where we are right now in this meeting, is getting extraordinarily powerful, extraordinarily capable. So of course, it’s going to be the foundation, that’s the center pole of the metaverse. It has to be because that is going to be the real place where we’re going to meet and it’s also open and infinitely extensible. We’re talking about the how Mozilla Hubs is been improved dramatically by the users of it. This the same thing here, we’re going to see the Metaverse is not going to be created by meta, it’s not going to be created by Apple. It’s going to be created by the people who are providing capabilities inventing it on the web side of the world. And that’s why in a few years, I think that’s all there’s going to be, you know, we think about things or the technologies that are evolving around the web today — WebGPU, for example, which in some cases gives you a 10x performance increase. WebAssembly, which we use, is becoming a very, very impressive platform. And clearly, clearly, to me, there’s no question that the metaverse is going to be built on the web. And everything else is going to fall away pretty quickly over the next five years. So that is kind of my, I guess, my soapbox.
Michael Moran Host
…that I didn’t even have to ask you the questions. I might ask, I might ask one. But I do know we are a little bit over time for everyone. I did want to ask kind of since you have such a history, and for the people that might not be as technically minded. David’s working with people that have been foundational and their protocols have been and continue to be the foundations of the Internet as we know it. So this moment that we’re currently in where the headsets based applications have all of the millions of monthly active users. Do you see any parallels to another time that might speak to where the web is going to make this takeover?
David A. Smith
There’s no question to me that this is the renaissance of the web. You know, the web, when it first showed up was quickly became an astonishing thing. You know, many of you may be too young to remember all that. But I remember when we first saw it, and it’s like, here’s an image that I just pulled out of the ether. Here’s a video! I remember the very first video I ever saw on on the web was Disney’s trailer for Disney’s Pocahontas. And I was like, Oh, shit, the world has just turned upside down. I didn’t, I didn’t go and start Netflix I probably should have. But you could just see, you could just see that that transformation that was about to happen. And if you look at the web, one thing to be very mindful of and you can still do it, you go up to it, you can view the source of any webpage today. It’s the largest open source project in the history of mankind. And there’s a reason why the web capabilities have become so incredibly sophisticated. Because there’s been literally trillions of dollars invested in that infrastructure. And all I’m saying is, that has not stopped. This is not a new thing. In a sense, it’s just a continuation of the thing that, you know, we’ve taken for granted. Alan Kay, who’s my mentor, I mentioned, said point of view is worth 80 IQ points. This is another point of view on the same thing, we’re already familiar with this 3d stuff, we’re all of a sudden, we are all meeting inside of a space like this. And in the very near future, we’re going to do more than just meet we’re going to be building out these worlds from the inside. No, this is where we’re on the glide path here. And this is unstoppable. Just think about it — it doesn’t matter how many developers Meta has. There are 17 million JavaScript developers in the world. And every one of them is going to contribute something to this.
Michael Moran Host
Thank you so much, David, that was really, really stimulating. And definitely, for me, someone who is just starting out and is only lived as long as many technologies are around. It is so interesting to see the perspective of the people that were literally building it from the ground up.
David A. Smith
Thanks for the invitation. That was it’s always fun to do this. And by the way, check out Croquet. Croquet.io is an operating system, and it’s still in beta. But it’s pretty neat. If you’re a developer, you definitely need to spend some time playing with it.
Nick
Awesome. Can I ask a question you mentioned? Yeah, that you said like games was like one of the tent poles of a lot of this immersive media stuff. So is there anything from games that you think has is already further than where we’re at with some of this these platforms?
David A. Smith
Yeah, I think games are always a leading indicator, if you recall, Microsoft, when they had DOS, they had a thing called X mode, which was where all the games existed, it was pretty low res. Maybe it’s 320 or something like that. But it was nice, because there was a very, very fast rendering capability. You had to write your own renders back then. So there’s all software there was no, there were no GPUs. But yeah, games are always ahead of the curve. In fact, the interface we use here was invented then. I invented some of this. And then Carmac invented much of the mouse look interface you’re using here, for Wolfenstein. So, yeah, I mean, all the idioms that we think of that where we moved through these worlds in here, were first created in game spaces. And that’s not surprising. So, so yeah, games are always going to be kind of leading the pack. So if you want to see what your interface is going to look like in 5 or 10 years, you know, play a high end game today.
And really, what’s nice is when this happens, you’re gonna see this exponential curve of capabilities, which has already started but you know, exponents are kind of invisible when they start, right. It’s only when the curve gets a little steeper that it you start having to hold on for dear life, right?
