How we built a real-time in-game currency service with 7.5M transactions per minute

For us techies at Loco, there is nothing more exciting than to work on the cutting edge solutions, use new products, and solve problems that don’t have a solution on StackOverflow. Recently, we came across one such problem and discovered one such technology to solve it.

We designed our new, high performance, transactional currency service around Tarantool, a high-performance database. It served as our SSOT (single source of truth) database to ship a new feature.

Why did we even think of Tarantool?

We had to build an in-game currency transaction system which requires a minimum of 150K concurrent users to get coins real-time. The users should be able to use the coins as soon as they get it. For us, ‘real-time’ means ~300ms. For all our users, all 150K concurrent users.

One way of solving this is using Redis. Use it as a write-through cache for a NoSQL database. This becomes painful very quickly. We have to maintain coherency, handle race conditions, deal with cache invalidation, eviction, expiry, etc.

Durability — Redis is focused on in-memory processing with periodic snapshot capabilities. But Tarantool has full-scale WAL log and snapshots, which we don’t need to manage; unlike Redis.

Atomicity — There isn’t an easy way to perform atomic operations across Redis and the backing NoSQL DB. This means there isn’t an SSOT (single source of truth) in this solution.

There are a host of other issues as well.

That sounds simple, right? It’s just a few hundred thousand database transactions per second.

What? You say no?

How Tarantool Solved our Problem

Tarantool is a full-scale NoSQL database with a Lua application server built-in, transactions, stored procedures, and real persistence.

Tarantool provides two storage engines: Memtx (In-memory Storage Engine) and Vinyl (Disk Storage Engine). Both these engines provide reliable ACID transactions.

MemTx uses WAL (Write-Ahead Logs) and snapshots to satisfy consistency in the case when MemTx crashes.

It can have any number of secondary indexes.

It supports server-side scripts in Lua, for operations like batched-operations, which in itself is a great feature.

Tarantool has really fast CRUD operations

It achieves this by using fibers, on a single thread, to perform transactions.

What does that mean?

There are no multiple threads operating on the database at once. Fiber is a light-weight thread. Transaction processor thread will execute all instructions in fiber until yield occurs and then move on to another one. Fibers are expected to use co-operative multitasking to prevent starvation_._ In addition, all IO tasks (disk, network, etc.) triggers implicit yielding. Developers don’t need to worry about fiber yielding most of the time.

So how fast is this thing?

Our target was to get 5 secs for 500K.

We did that in 3.6 secs using Memtx! 500K transactions in 3.6 seconds!

Remember, this is with full ACID properties for all of the transactions.

It gets better. For us, a transaction has one write and one update operation.

Tarantool started completely as an experiment. We were aware of the risks. We succeeded and got good benchmark results. And, now it’s in production!

Advantages:

1. Extremely fast
2. Server-side custom Lua functions (we can reduce network calls for batch operations)
3. Single-threaded (only one transaction processor thread in Tarantool)
4. A persistent in-memory database (Memtx)
5. Server administration(tarantoolctl utility)
6. Database sharding (Vshard) and replication
7. Vinyl disk storage can be a good replacement for NoSQL databases
8. Tarantool Telegram group support (original Tarantool devs respond on this group)

Disadvantages:

1. Infrastructure (If you are trying out Community edition, you need to manage instances, sharding, regular backups of snapshots by yourself.)
2. Not a very big community. Have to rely on their telegram group for any queries.
3. Not provided by Cloud platforms like AWS or Azure (old version is present in the AWS marketplace.) So we have to build our own monitoring systems for Tarantool instances.

Tarantool Enterprise Edition has features which solve some of the disadvantages listed here. But it isn’t free.

Should you start using Tarantool?

That’s the million dollar question. In some cases, quite literally.

Ask yourself these questions:

1. Do you really need a fast processing database? Is it a critical, user-facing requirement?
2. Does eventual consistency not work for your scenario?
3. Are you already planning to use Redis/cache layer in front of another database?
4. Are you willing to manage all infrastructure requirements which are usually supported by cloud platforms?

If your answer is yes, to any of these questions, then yes! Opt for Tarantool, right away. Maybe you can try it with a small micro-service and then go BIG!!!

Personally, we really love Tarantool. Three months into production, are there are no major issues!

To win cash daily, play Loco! Download the app now.