Breaking News

Non-euclidean Stencil Portals in Virtual Reality

Multiple of my posts looked already at different technical aspects of how portals can be realized. Since it is the underlying mechanic for Out Or Dead that should not come as a surprise:

Being four months into development, I have learned a lot and I want to share some of the key insights and achievements today:

  • Construction
  • World Mapping
  • Debugging
  • Transitions
  • Lighting
  • Physics


My initial tilemap approach is still valid and works nicely. From time to time I am adding new properties but try to keep it to a minimum. Simple is good. And whatever can be detected automatically is done so. In a future vision I plan to incorporate a 3D tilemap right in the game which reads and writes the same format. But that is far future.

Portals are an object like everything else that can be placed on the tilemap. Using snapping ensures they are glued to any geometry that might be behind them.

What I am slowly adding are layers on top. The first one was a level descriptor explicitly listing all rooms instead of doing auto discovery, which makes naming easier and the rest more deterministic. I was able to reuse the so far non-UI supported world descriptor of Tiled for that. That will also allow me to show levels next to each other visually in Tiled but I have not bothered yet to set it up.

Mapping non-euclidean space

That brings me to an interesting topic though. In Out Or Dead rooms are folded into each other, constantly overlapping. How would one represent the game world as a map that the player actually perceives?

I worked on this for me rather fascinating issue already three months ago but never shared the results so far. The approach that I settled on was to rotate and offset rooms around their entry portals. It’s still super rough but here a glimpse:

Purple lines show the actual connections between the rooms

Branches to left and right should be easy. Multiple paths into the same direction will require more sophisticated placement, I guess using either height or some graph layout algorithm.

The following animation shows how rooms are finally folded into each other:


Having geometry constantly overlapping each other makes it a bit of a pain to debug. Tooling was therefore some of the first tech I invested time into. The following pieces turned out to be crucial:

  • Starting in a specific room when in play mode
  • Removing textures and replacing them with plain colors to clearly see room transitions
  • Dumping calculated values into the inspector, e.g. distance to a portal
  • Leaving out parts of the geometry (most often the ceiling)
  • An efficient editor layout (typically a huge scene view and rather small game view)
  • An Oculus Rift if developing for Quest, so that there is no build necessary to test in VR
  • An option to move and rotate the whole world to a specific coordinate so that I am already in the right spot when putting on the headset instead of needing to walk there somehow
Identical Scenes – Left: Original materials, Right: Debug materials highlighting the two different rooms


My original stencil transition to make another room visible was by the camera triggering the portal plane. This has two major short-comings:


The minimum setting for near-clipping in the camera is 0.01. That means that objects closer than that to the camera become cut-off. The effect on a plane is that it is only partially visible and the geometry of the room behind, which is the wrong room, shines through.

Purple room shining through incorrectly during transition

One example how this can be solved is intentional cut-outs like done in the great world of Shadow Point.

Shadow Point: Right when clipping would occur, the circular render texture is overlayed, resulting in a bit of an unnatural transition

I guess due to using Render Textures (at least my guess due to the effects they achieve) they had to go for such a solution. A bit unfortunate is that the portals are not pinned onto the walls directly but have a strange gap. I suspect that is also due to the transition logic used.

Shadow Point: Portal floating in front of the wall for some reason

The solution I went for is a variation of a transition effect I saw in the GitHub repo of Feddas. Instead of a single plane I construct a transition tunnel which is super easy when using stencils instead of Render Textures. The video below shows many of the different actions taking place (and even more in the background), revealing always the right view for each eye.

Top: transition tunnel becoming visible over time, Bottom: no pop-in or other artifacts in each eye

VR Stereo Camera

In VR there is not ONE point when a transition happens but TWO, one for each eye. This results in two challenges: when does each eye cross the transition and how to display a different room to each eye in case only one eye crossed yet. If not done 100% correctly the player will perceive a noticeable switch or see artifacts.

The engine also needs to be able to cater for players that cross the transition extremely slow to be bullet proof. I had originally put a trigger on the camera but in (the recommended) single instance mode both eyes get rendered with one camera so in the end I simply track the position of each eye myself and calculate the transition direction out of the information over time.

The portal can only be crossed from one side so accidentally going through it from the other should not trigger a room switch. Displaying different rooms per camera was actually an easy thing to do but needed a lot of tweaking to find the right combination of settings. In principle I activate a full screen quad anchored to the camera as soon as both eyes are in different rooms. Through a custom shader I then utilize the unity stereo eye index to clear the correct stencil value.

Another issue to face with such portals is where they can be placed. Putting them somewhere mid-room is rather easy. It becomes tricky if the player should be able to walk through what used to be a wall in one room but an opening in another. It can be masked to a certain extend but the only really good way is to dynamically hide parts of the geometry. That means the engine needs a way to identify such parts and it must also work for parts where only one side is affected, e.g. corner walls.

A nice trick my portals learned is to support unidirectional transitions. That means once the player went through, the portal disappears and there is no way back. That is especially cool for the intro and outro rooms, more on these in a separate post.


This turned out to be the easiest. By putting each room onto a different layer the lights inside each room can be assigned a layer mask so they only affect the room. This allows to create smooth transitions but also some neat out of this world light changes between rooms.


Again a tricky thing but much easier than player room transitions. There are multiple aspects:

  • The player falling down a hole
  • The player grabbing an object
  • An object moving through a portal on its own
  • An object being thrown (or shot) through a portal

The most important part are the layers that were already set up for the lighting. I set them so they cannot collide with each other. This resolves most issues.

A total of nine room layers are reserved, which is for now the limit of concurrently visible rooms

Only objects transitioning through portals require special treatment. Let’s look at a gun play and possible cases:

  • Player and gun in front of portal shooting through
  • Player in front and gun already behind portal

That means the portal cannot be (the only) trigger to change the physics layer of the object mid flight. Also the portal intersecting player and weapon is important. The second case I still need to solve but here a working version of the first:

Top: bullets stop apparently mid-air, Bottom: stencil portal reveals another room, bullets collide correctly when shot through portal

The Road Ahead

I have come quite a long way. Three issues are left that require a bit not thinking or consulting:

  • Objects visible in line of sight of a portal behind them will have the portal overlay
  • Portals of the same room visible behind each other (currently I use the limitation that portals should never face the same direction)
    • Update: Solved! The trick is to use a back-side only rendering cube behind the portals to mask this in the depth buffer
  • Portals inside the room visible from the current room (recursion)

All in all though the technology is really amazing and immersive and I have so many ideas what experiences could be built with that. One positive side-effect of stencils is that they are super fast. There is no multi-pass going on, no duplicate evaluation of pixels. This gives me quite some hope to be able to squeeze out a lot of fidelity from the Quest when it comes to level design later. Stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *