Blog » Making Autotable

Making Autotable

2020-05-24

Autotable is an online tabletop simulator for Riichi Mahjong. It's a project that I really enjoyed working on, and despite the simple core idea I put a lot of effort into improvements and tweaks. I hope this will make for an interesting story.

Why make this?

I am not very good at mahjong.

I like playing it casually, and I don't particularly care about improving my skill. I really enjoy the social aspect of playing with people and talking with them, handling the tiles, and the overall atmosphere of the game.

That's why I don't like mahjong computer games, or platforms like Tenhou. They make the game nothing like the "real thing". The play is fast and streamlined. They pause to ask me for calls (pon / chi), or propose 3 different ways to declare riichi before I realize I can do it. The computer knows my hand better than me, and what I really liked in the game gets automated away.

So when we were all staying at home social distancing, and I got a craving for mahjong, I investigated online tabletop simulators. There is Tabletopia and Tabletop Simulator. You can play mahjong in both. Tabletop Simulator is especially known for its modding community, and there are several player-made setups for mahjong.

On closer look, I'm not satisfied. Tabletopia, for instance, wants you to treat your hands of tiles like hands of cards in poker, and instead of building a wall of tiles, you pull them from a bag. Functionally, it's the same, but it doesn't feel like mahjong anymore.

Tabletop Simulator is better, because its engine is very flexible. Still, I wasn't happy with how your own tiles have to go in a special "hidden" zone. All I wanted was to have them standing on the table, facing in my direction, with no way for the opponents to peek.

This is how I arrived at my idea. What if there was a simulator geared specifically for mahjong? A game that allowed you to do all the things required in a mahjong game, but do them by yourself.

Compared to existing, heavily "automated" mahjong games, I wanted to have something that didn't care about turns, winning and rules. I'm okay with restricting some players' freedom - there is no need for flipping the table, throwing the tiles or building a pyramid out of them - but on the other hand, the player should never be forced to take this or that specific tile, and there shouldn't be any things happening out of the blue.

"No rules" means also no need to implement the rules! Minefield Mahjong was already pretty challenging, and programming a regular four-player game, with all the different special cases, would be much, much harder.

Or, to put it in yet another way: this should be a game that allows at least for some real-life mahjong manners, "anti-manners" and idiosyncracies (such as pulling your tile too soon, or doing some things in a specific order).

What about some more flashy cheating, like stacking the wall and exchanging it with your hand? Maybe someday! :)

Appearance

While a mahjong game is three-dimensional, with some tiles stacked on each other, it still has a rather simple layout. In the beginning, I thought I could fake it with drawing simple sprites, in a strategy-game-like view.

I had really great tile pictures already, but drawing them at an angle meant I also had to draw sides and colored backs. I had some early designs and a prototype for that:

However, I got discouraged when I realized that if I'm manually specifying how to draw the tiles, I would have to do so for every possible rotation: standing, lying face-up, lying face-down, sideways…

So I scrapped the idea of 2D graphics and used a 3D engine called three.js. I figured that I could still use it to generate simple "2D-looking" visuals, but at least all the math related to objects' positions an geometry would be taken care of.

As it turns out, I'm still using a simple orthographic projection for the game. I think it looks nice and clean. There is also a "perspective mode", which some players prefer for the immersion. I think it's harder to use but hey, it's a feature that I got basically for free.

The orthographic view has one problem. The far bottom tile in the wall is hard to reach:

To make it somewhat usable, I had to keep the view angle at 45 degrees (with a steeper angle, the bottom tile is hidden completely), and color it darker. It still could work better - I'm planning to increase the "hitbox" so that this tile is easier to grab even when your mouse cursor is not touching it.

Textures and models

The textures for the game are generated from SVGs. I like that because I can freely adjust the texture resolution, edit them in Inkscape, add details and so on.

The models aren't that simple. Initially I just had some boxes. However, drawing the tiles next to each other caused them to bleed together. My first solution for that was to make some gaps between them. Later, I decided to add some gradients to the textures instead. That separated the tiles from each other visually, but gave them a cartoonish look.

Finally, I bit the bullet and used Blender to edit the model. I hadn't used Blender too much before, and had an impression that it was a very complicated and hard to use program. But actually, for my use case I was able to learn relatively quickly all that I needed.

Now I have nice models with rounded corners. Adding a few lights to the scene makes them look pretty good, and more realistic than the previous cartoon version.

I was also able to construct a pretty nice data pipeline for building the models. With a single make command, I'm able to get Blender to export a glTF file containing all the models and the textures, which three.js is then able to import.

Gameplay

Here is the core idea for game actions. The mahjong table has several different places for tiles: your hand, the wall, the discard pond. Call them slots. The tiles are placed in slots, and you can drag a tile from one slot to another.

As you can see, the tiles can be rotated in different directions, depending on a slot. Each slot has a list of allowed rotations: in hand, the tile can be upright or face-up (revealed); in wall, it can be face-down or face-up, and so on.

Corner cases

Although simple, this design already has some corner cases. For example, in a few places in mahjong you need to play a tile sideways. That means the next tiles need to be pushed to the right, or they will collide with the sideways one:

"To the right" is of course relative to the player position. And sometimes the push is actually from right to left:

The way I do this is there is a "push" relationship between slots: slot A pushes slot B. When a tile is rotated, I go through the relationships and check if any in these slots tiles are pushing each other.

Another, more interesting corner case is the action of drawing a tile to your hand. Previously, it worked like this:

The tile is hidden while dragged, and rotates when you drop it to destination. This is usually what you want, but in this specific case, it means you need to first drop the tile into your hand, then pick it up again once you see what it is.

A better solution is to rotate a tile immediately when you hover it over your hand, like so:

Now drawing and discarding becomes a single motion. It looks like a small thing, but it's really a bigger deal than it looks like. It also means that if you do decide to keep the tile, you can put it in the right place in your hand:

This auto-rotation feels so natural that when I added it, the players got used to it without noticing anything has changed!

Another corner case is "secret" slots that are normally not active. For example, when you run out of space for discards, you are supposed to continue on the last line:

However, this situation happens rarely, and allowing you to use these places right away would be confusing and unnecessary. They are activated only when slots to their left are filled.

Paying

In Riichi Mahjong, you pay other players with point sticks. You also use a 1000 point stick as a deposit when declaring riichi. This felt important to the feeling of the game so I wanted to keep it.

Unfortunately, it would be too much to try and fit all the players' point sticks on the screen. Instead, I added a special "look down" key to switch the view. I regret not being able to keep all of the game information on a single screen, but in a way, this is also pretty realistic. Most of the time, you are focused on the game, and in a real-life automatic tables, the sticks are actually in a drawer that you need to open first.

What about paying other players? You can do that by placing the sticks on the table, but where? For some time, I was afraid I need to allow free placement everywhere on the table. While that's not hard to implement by itself, the implications are difficult: you need to have a collision system, and probably allow the players to rotate the sticks.

Instead, I extended the "slots" system and made some dedicated places on the table for the exchange between players. I think it's a good compromise.

Mouse position

The player needs to be able to drag the tiles with mouse. So, apart from projecting the image to the screen, an opposite operation is needed: given the mouse coordinates on screen, which object are we pointing at?

A simple technique for that is ray casting: projecting a ray from the camera and through the mouse coordinates on the screen, then seeing if it intersects any object.

Once we have picked an object, we have to decide where in the 3D space the player is dragging it to. This is easy if we assume that the object is being dragged in the same horizontal plane.

An interesting implication of that is that the mouse coordinates are three-dimensional. In fact, they have to be, because we have different players and they all see the same mouse pointers. Otherwise, what looks good to you, would look bad to a player viewing the scene from opposite side of the table:

As a result, when you move the mouse in a straight line, the pointer as seen by the other players might jump around. I don't think it looks too bad, and in a sense, it's completely accurate:

To sum up, these are the rules for determining your mouse positions in 3D:

• If you're not moving over any tile, the mouse is at "ground" (table) level (intersection of ray and table).
• If you're moving over a tile, the mouse is whenever you point at the tile (intersection of ray and tile).
• If you're dragging a tile across screen, the mouse is at whatever level you started dragging (intersection of ray and a plane at the same elevation as initial mouse position).

Mouse motion

I mentioned synchronizing the mouse position with other players. When I started the project, I wasn't even sure if that was necessary, but it ended up being a very important effect. It really contributed to the realism of the game.

In the beginning, I sent the mouse position on each mousemove event. That looked pretty smooth - on localhost. When running the game on a server, the mouse started to jump around as the packets were arriving in uneven intervals.

I haven't measured very rigorously what the problem was, but I thought I was being pretty wasteful with bandwidth, and with server CPU time. So I rate-limited the mouse position updates to 10 per second. The effect was somewhat choppy, but good enough. It allowed me to play the first game with friends.

Then, I tried smoothly interpolating the position. The improvement was drastic. It really looked as if the other person was moving the mouse!

Behind the scenes, what's happening is that the game receives waypoints and shows the mouse as moving between them. New waypoints arrive 10 times a second, so the mouse motion is always played with a 100 ms delay.

As I mentioned before, the messages might come in uneven intervals. There is no guarantee that the waypoints will arrive exactly every 100 milliseconds, and if you just render them as they arrive, the movement will also be uneven. Because of that, the messages carry a timestamp. Of course, I cannot use a timestamp from remote machine directly - your timestamp is not the same as my timestamp - but it's enough to calculate the interval between the remote timestamps (usually 100 ms), and maybe apply some limits if a waypoint arrives suspiciously late or early.

I was hesitant to add mouse smoothing at first. The problem with this technique is that I'm introducing a universal 100 ms delay to all mouse motion. I was worried it would be noticeable, and also that it would "infect" other interactions, like dropping the tiles.

My worries turned out to be mostly unfounded. This is not a fast-paced game like an FPS, and nobody notices the delay. Adding delays in other parts of the code was not necessary: the player drops the tile slightly sooner than the mouse motion would suggest, but dropping a tile usually doesn't happen "at full speed" anyway - the player slows down the mouse first, and verifies it's in the right place. There is some "landing time", so to speak.

Tile sorting

When playing mahjong, it's likely you will want to sort your tiles by suit and number value. This is a more complicated interaction: you're not just dragging a tile from one place to another, but moving other tiles in the process.

Unfortunately it's also an interaction I couldn't do without. If the game auto-sorted the hand, it would be really convenient for the player, but that would no longer be the game I intended to make.

It really took me a long time. I spent several 1 am sessions trying to get it right, each time only to realize halfway through I made a horrible mess out of my code and I'm too sleepy to write good code, and to throw away everything I wrote with git reset --hard. Finally, on the third or fourth attempt I got something that worked and didn't inflict too much damage on surrounding code.

The final algorithm works something like this:

• Take the list of tiles being moved (called held tiles).
• Find if they can be dropped somewhere. Call it a list of target slots. If the target slots are all empty, then we're done.
• If the target slots are not empty, check if the tiles in these slots can be shifted to the left or to the right. If so, temporarily display the shifted tiles in the new location.
• If the player doesn't decide to drop the tiles to target slot, return the shifted tiles to their positions.

I experimented with moving the shifted tiles in a smooth motion, instead of having them jump around, but it looked very unnatural.

Initially, the "shifted tiles" were only shifted locally, until you dropped the held tiles to destination. Later I decided to send them over the network as well, so that everybody sees you sorting the tiles. And here's where the 100 millisecond lag turned up! Because the mouse movement is delayed compared to everything else, the tiles appear to move ahead of time:

It looked pretty weird, so I added a 100ms delay to the shifted tiles as well. So far, all the other parts of the game work correctly without any added delays. This one might have been special because the tiles so closely follow the mouse cursor.

Optimization

My mahjong table is not a very complicated 3D scene, but when you count the objects - 136 tiles, 60 point sticks, some assorted simple meshes - it does add up. On a powerful machine, that's not a problem, but when I first tried the game on my laptop, the framerate was pretty bad. In power saving mode (on battery), the game barely reached 20 FPS. I wanted to have a buttery-smooth 60 whenever possible!

It turns out that a common advice is to "merge your geometry". Static meshes, like scenery, can be often combined into one bigger object, and sent to GPU in a single draw call.

My tiles move rarely, but they do move. So I tried the second common advice, which was to use geometry instancing. With geometry instancing, if you need to draw many copies of a model, you send the model to GPU once, along with a separate array with all the positions for that mesh.

There were two complications, though:

• The tiles are not all the same! Their faces are different. Fortunately, this can be worked around by writing a custom shader - a piece of GLSL code running on the GPU. My shader overrides the UV (texture) coordinates for some of the vertices:

  if (vUv.x <= TILE_DU && vUv.y <= TILE_DV) {
    // If this is part of tile front, apply the
    // offset to draw the right front.
    vUv.x += offset.x;
    vUv.y += offset.y;
  } else if (vUv.y >= 4*TILE_DV) {
    // If this is part of tile back, or side,
    // apply the offset to draw the right back
    // (there are yellow and blue tiles).
    vUv.y += offset.z;
  }
  

• There are many visual effects. A tile can be…

  • Hovered-over with cursor: draw with a slightly lighter color.
  • Held by player: draw it on the top of everything else.
  • Non-droppable: if we cannot drop the tile here, draw it as slightly translucent.
  • Selected: draw a glowing outline around the tile.
  • Bottom row: for a usability hack mentioned earlier, the bottom tiles are drawn darker.

  It's really hard to customize so many attributes of the object when using instancing, and in some cases (like drawing the objects on the top), probably impossible. Instead, I use an instanced mesh for all the "normal" tiles, and separate, regular meshes for the "custom" tiles (of which there are usually only a few on the screen).

Two consecutive framebuffer states, as captured by SpectorJS - you can see that the darker tiles were drawn before, and all the other tiles are drawn in a single batch.

The instanced tiles were a huge improvement. Combine that with some places where I avoid recalculating too many things (since, well, the tiles rarely change positions), and I now reach 60 FPS on most computers I test the game on. When I disable the limit, Google Chrome on my desktop computer is able to calculate almost 1500 frames per second!

Of course, I might have had a much easier time with performance if I didn't use a 3D engine, but 2D sprites. Using something else than JavaScript/TypeScript and browser could also help. Still, I'm very happy with my technology choices - it's hard to get more portable than a HTML5 in-browser game, and the graphics engine is pretty flexible.

Database sync

All of that game state needs to be synchronized between players. The server is a relatively small part of the project, but it still had to undergo some evolution.

At first, I thought it would be simple. Synchronize a set of Things (objects on the board) between player. And each player controls its own data, like nickname, so also synchronize a set of Players.

Later, as it grew bigger, these two "object types" turned out not to be enough. There were more pieces of data - the match state (who is the dealer), what kind of tiles we're using, the mouse position, and so on.

Instead of hardcoding more things into the protocol, I decided to make the server a simple database. It became a simple key-value store that allowed you to modify objects like "nick 2" or "thing 151". The objects are divided into different collections (here: nicks, things, and so on). On the client code, I can refer to these collections.

For example, in the client code, I might write something like:

client.mouse.set(client.playerId(), {x, y, z, time});

Which sends the following data over the websocket:

{
    "type": "UPDATE",
    "entries": [
        ["mouse", "X123", { "x": 0.5, "y": 0.75, "z": 2.0, "time": 1590335831556 }]
    ]
}

The nice thing is that the server is a relatively simple program that doesn't know anything about the data. When I add a new feature, usually I only update the client (in-browser game), and I don't have to change the server at all.

Wait, did I say simple? Well, there is a number of features that I needed to add to my key-value database:

• Rate limiting (for the mouse updates). Okay, this one is client-side only.
• Weak references to players. When a player disconnects, I want the server to clean up some data (like their nickname or mouse position).
• Unique attributes. This is to prevent conflicts: in a rare case, two players might attempt to put different tiles in the same slot. That would break the game, so the server rejects a update that violates this constraint.
• Ephemeral messages. I'm using the server to also propagate sounds (such as dropping the tile on the board) to other players. I achieve it, like everything else, by sending an object to the server. However, I don't want the object to be remembered there, because then any player that connects later will hear the sound.

The setup sounds weird, and I might be reinventing something already existing (like Firebase, maybe?) that would suit me better here. Oh well… it does what I want, and it's a local optimum.

Reconnecting

At first, I was worried about publishing the game and letting visitors from the internet play it, because I didn't have a procedure for deploying a new server version. I would have to restart the server, throwing away all the games stored in memory! For Minefield Mahjong, a solution to that problem involved an SQLite database and a separate "replay" subsystem.

…Then I realized it's not a problem at all. Everything about the game is already stored on players' computers! If I restart the server, they can reconnect and re-populate the database.

Of course, storing all state on the client is a cheating risk. There is nothing stopping you from hacking the game. However, the game is a sandbox anyway, and all kinds of illegal actions are possible. So I could argue that this is by design, and trying to harden it is not worth the added complexity - nobody is going to play a serious competitive game on my platform anyway. The important part is that the players are having fun. :)

Speaking of cheating, an interesting bug came up. If you selected some tiles, and someone clicked "Deal", they got shuffled with all the others, but stayed selected for you:

I guess a real-life version would be marking your tiles somehow.

Summing up

Here is the page for the game again. The project is now mostly finished. I might add some gameplay improvements and new modes, but what I have so far is enough to play a complete game. I haven't open-sourced it, but I'll probably do so in the future.

Update: The project is now open source: https://github.com/pwmarcz/autotable

What really got me hooked is that in this project, I had users from the start. As soon as there were enough features to play a game, we started testing it with friends. Then, I built new features based on their feedback, and things that we all noticed were missing.

It was great to have this kind of "product pressure". I did a lot of refactoring and cleanup, and I'm still doing it, but it was always in order to build more features. Often, I added something in a quick-and-dirty way, even piled a few changes like that, and refactored later because I realized it's becoming necessary.

Thanks to everyone who played and helped me develop! I really, really enjoyed working on this project.