Collaborative AR:
Comparing Approaches

Bit-identical replicated computation using Croquet.

I think Croquet is the best thing I’ve seen in computing over the last 10 or 15 years. It solves some important and massive problems in extremely elegant ways. It really could become a new kind of operating system for the whole Internet.

– Alan Kay, Turing Award Winning Computer Scientist

  1. Instantaneous shared actions. Actions must immediately translate into changes in the shared experience for every participant. Even the slightest latency in interactions ruins the user’s perception of the liveness of the experience. This is particularly true when the participants are face-to-face. Latency should be under 10 milliseconds when we each see each other engaging with the shared world.
  2. Shared state. The participants must see and maintain a shared world. This means that any set of actions by the shared world’s participants must result in the same transformation and view for all of them.
  3. Dynamic join. New users must be able to join a session already in progress at any time. This may not matter for certain applications like Zoom calls where you can get caught up by other participants or a screen share. In multiplayer games, a lobby is often used to collect the participants and then launch them all with the same initial state. However, AR applications will require that a new participant be able to join a dynamic session already in progress, which means they need to replicate and synchronize with the current shared state of the world.
  4. Verification of synchronization. Each of the peers needs a way of verifying that they continue to be synchronized with the others. If they find that they have diverged (gone out of sync), they need a strategy to re-join the shared state.
  5. Rich vocabulary. There must be an unlimited “vocabulary” of messages between systems and users. AR will require an extremely rich collection of these actions and ideas, many of them not yet understood, to ensure a vital and extensible communication medium. Further, unanticipated message types should be incorporated dynamically. AR is not just a consumption platform — it is a live creation/development environment. Virtually any kind of engagement and interaction must be allowed for and made visible and consistent for all participants. Multiplayer games have a limited vocabulary — often, something like “move”, “shoot”, “kill” or “die”. Rarely is there any live manipulation of the world beyond those elements, which means that the complexity of these games is limited. Creating and editing objects, when available, is performed locally. The objects are then injected into the world so any dynamic collaborative interaction and extension is impossible.
  6. Replicated simulation. Replicated, responsive time-based simulation is essential and will form the foundation of rich collaborative interaction. This means that the shared world is more than just a simple state system but can evolve dynamically while responding seamlessly to user events. Perhaps the richest example is a world built on a complex physics simulation — where replicating real-time physics in a multiplayer experience as part of game play is close to impossible with traditional approaches. Future responsive user interfaces will require this. Instantaneous shared simulation is what will enable AR to become the ultimate communication and cognitive exploration tool it is destined to be.

Replicated Events

Replicated State

Centralized State (aka Client/Server)

Initiatives to place real-time processing on the wrong side of the latency wall have always been doomed to failure because, even though bandwidth and latency are improving, local computing performance is improving faster.

–Tim Sweeney, Epic Games CEO

Replicated Computation

Not Mutually Exclusive

The key thing about all the world’s big problems is that they have to be dealt with collectively. If we don’t get collectively smarter, we’re doomed.

–Doug Engelbart

Collaboration and the Augmented Conversation



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David A. Smith

AR, VR, AI, 3D Pioneer I invented 3D portals and crates in games. I wrote the first 3D adventure/shooter, and created Croquet - it redefines collaboration.