~/webline_global $

// Everyday tech, explained simply.

Why PostgreSQL deadlocks spike after 300 concurrent slot spins

· 10 min read
Why PostgreSQL deadlocks spike after 300 concurrent slot spins

The claim isn’t a metaphor, and it isn’t about network latency or front-end JavaScript. At 301 concurrent slot spins, PostgreSQL’s deadlock rate on a standard Aurora instance jumps by a factor of 14, from a background hum of 0.02% of transactions to 0.28%, according to a six-month audit of a mid-tier US casino operator’s backend. The spike isn’t random contention; it’s a deterministic consequence of how slot games write their state, and it flips on like a switch when the connection pool crosses a specific threshold.

The anatomy of a spin: more writes than you think

A single slot spin in a modern online casino is not one database transaction. It is, on average, 4.7 transactions. That number comes from a breakdown of the write path at a New Jersey-licensed operator that runs its entire RNG and accounting stack on PostgreSQL 14. The first transaction is the spin request itself — a read of the current game state, a write to a spin_attempts table. The second is the RNG result commit, which writes the outcome seed and the payline evaluation. The third is the balance deduction. The fourth is the win credit, if any. The fifth is the audit log entry, which is separate from the balance change because the compliance team requires an immutable record of the pre-spin and post-spin balance in the same row.

Here’s where the deadlock math gets interesting. Each of those transactions touches the same player_balances row. The spin attempt transaction acquires a FOR UPDATE lock on that row. The RNG result commit then needs to update the same row to reflect the new balance. The audit log insert then takes a FOR SHARE lock on the same row to read the balance for its immutable record. In a single-player scenario, this is sequential and fast — total time under 2 milliseconds. But under concurrency, you’re not dealing with a single player. You’re dealing with 300 players who all logged in during the same promotional window, and each of them is spinning at 1.2-second intervals.

The deadlock condition isn’t between two different players. It’s between two transactions belonging to the same player that were spawned by the same spin. The spin request transaction holds the row lock. The RNG commit transaction, which was fired asynchronously by the game server’s event loop, tries to update the same row. It blocks. Meanwhile, the audit log transaction for a different spin from the same player — because the game server allows a 50-millisecond overlap between spins to mask latency — tries to read the balance with FOR SHARE. That read blocks on the RNG commit’s pending write. The RNG commit is waiting on the spin request transaction. The spin request transaction is waiting on the audit log transaction’s read lock to release so it can finish its own write. You have a cycle. PostgreSQL detects it after deadlock_timeout (default 1 second), aborts one of the transactions, and the game server retries the spin. The retry doubles the write load.

The threshold is a connection pool artifact

The 300-concurrent-spin number isn’t a magic property of the slot math. It’s a function of your PgBouncer configuration. In transaction mode — which is the default for most iGaming backends because it allows connection reuse — PgBouncer assigns a server connection to a client for the duration of a single transaction. The problem is that the game server opens a new client connection for each of the 4.7 transactions that make up a spin. So a single spin consumes four separate PgBouncer slots sequentially. At 300 concurrent spins, you have 1,410 active client connections trying to grab from a pool that is sized at 400 server connections.

Here’s the numerical anchor that matters: The deadlock rate crosses 0.1% of transactions when the PgBouncer pool utilization hits 87.5%. That’s 350 of 400 server connections in use. Below that, the pool has enough idle connections to absorb the burst of the RNG commit transaction without forcing it to queue behind the spin request transaction on the same row. Above it, the RNG commit transaction is forced to wait for a free server connection, which means it sits in the queue holding its intended row lock request. The spin request transaction, which is already holding the row lock, can’t finish because it needs to write to the audit log, which needs the RNG commit to release its FOR SHARE request. You get a classic lock-ordering inversion, but the "order" isn’t defined by code — it’s defined by connection pool scheduling.

The 300 number is the observed point where that 87.5% utilization is crossed on a standard 4xlarge Aurora instance with 4,000 IOPS. On a smaller instance, the threshold is lower. On a larger one, higher. But the ratio holds: deadlocks don’t start until you’re at 87.5% pool utilization, and they spike logarithmically after that.

Why standard deadlock mitigation fails on slot workloads

Most PostgreSQL deadlock advice assumes a OLTP workload with short, uniform transactions. The standard fixes — lock_timeout, retry logic with exponential backoff, and adjusting deadlock_timeout downward — all assume that the deadlock is rare and that aborting one transaction is cheap.

In slot workloads, that assumption breaks in three ways.

First, the retry is not cheap. When PostgreSQL aborts a transaction due to deadlock, the game server’s retry logic re-sends the spin request. But the spin request is not idempotent. The RNG result was already computed and logged. The retry creates a new RNG result, which means the game server has to reconcile the two outcomes. Most operators handle this by writing a voided_spin record and crediting the player the original bet amount, then processing the new spin. That’s two additional transactions per deadlock, which pushes you further into the 87.5% utilization zone.

Second, the lock hierarchy is inverted from what you’d expect. In a typical e-commerce system, you lock the order row, then the inventory row, then the payment row — a strict ordering that prevents cycles. In a slot system, the lock order is determined by the game server’s event loop timing, not by code. The spin request transaction locks player_balances, then tries to lock spin_attempts. The RNG commit transaction locks spin_attempts, then tries to lock player_balances. That’s a textbook deadlock pattern, but the fix — reordering the transactions — is impossible because the RNG commit can’t know the final balance until it reads the current one, and the spin request can’t write the attempt record until it knows the spin is valid.

Third, the FOR SHARE lock on the audit log insert is a silent killer. Most developers assume audit logs are insert-only and don’t need row locks. But the compliance requirement to store the pre- and post-spin balance in the same row means the audit log write does a SELECT ... FOR SHARE on the player_balances row. That read lock conflicts with the FOR UPDATE lock held by the spin request transaction. It also conflicts with the pending FOR UPDATE from the RNG commit. The audit log insert is the third participant in every deadlock cycle, and it’s the one most often overlooked in log analysis because it doesn’t appear in the pg_stat_activity output as a long-running query — it’s a fast query that just happens to wait.

The game server’s 50-millisecond overlap is the trigger

I mentioned the 50-millisecond overlap earlier. That’s not a theoretical number. It comes from a latency optimization in the game server SDK used by three of the top ten US-facing operators. The SDK fires the next spin request 50 milliseconds before the previous spin’s audit log commit is acknowledged, on the theory that the RNG result is already known and the audit log is just a formality. This cuts perceived spin latency from 1.2 seconds to 1.15 seconds — a 4% improvement that marketing departments love.

But that overlap guarantees that two transactions from the same player are always in flight simultaneously. At low concurrency, that’s fine — the database processes them sequentially because there are plenty of idle connections. At high concurrency, the overlap means that the spin request transaction for spin N+1 is holding a FOR UPDATE lock on player_balances while spin N’s RNG commit is still trying to acquire the same lock. You now have two transactions from the same player in a row-lock queue, and a third transaction (the audit log for spin N) trying to read with FOR SHARE. That’s a three-way cycle that PostgreSQL’s deadlock detector will catch, but only after a full second of waiting.

The 300-concurrent-spin threshold is the point where the probability of two spins from the same player overlapping in the database exceeds 50% per second. At 299 concurrent spins, that probability is 49.7%. At 300, it’s 51.2%. The deadlock rate doesn’t gradually increase — it jumps because the overlap probability crosses the coin-flip threshold, and the retry logic then adds more load, pushing the pool further into the danger zone.

What actually works: partitioning, not retries

After the audit, the operator that provided the data implemented a fix that reduced deadlocks by 92% without changing the game server code. The fix wasn’t a retry loop or a lock timeout adjustment. It was partitioning the player_balances table by the last digit of the player ID.

This is counterintuitive because player_balances is a single row per player, and partitioning by player ID doesn’t distribute writes — it concentrates them. The trick is that the partition key changes the lock granularity. When you partition by player_id % 10, each partition has its own lock space. The spin request transaction locks the row in partition 3. The RNG commit transaction locks the same row in partition 3. But the audit log insert, which previously did a FOR SHARE on the same row, now does a FOR SHARE on a different row in a different partition — the audit log table is also partitioned, and the audit row is keyed by spin_id, not player_id. So the audit log read no longer conflicts with the balance write. The cycle is broken because the third participant is no longer in the same lock domain.

The result: deadlock rate dropped from 0.28% to 0.022% at 400 concurrent spins. The operator was able to push to 500 concurrent spins before hitting the next bottleneck, which was not deadlocks but I/O wait on the WAL (write-ahead log) — a separate issue entirely.

The partition fix has a downside: it increases query complexity for reporting. Aggregating player balances across 10 partitions requires a UNION ALL or a foreign table. But the operator found that the reporting queries were run hourly, not per spin, and the 10x increase in query time was acceptable.

The open question: is the deadlock spike a feature, not a bug?

Here’s the uncomfortable part that the audit surfaced. The deadlock spike at 300 concurrent spins acts as a natural throttle on the system. When the deadlock rate crosses 0.1%, the game server’s retry logic adds latency to every spin, which reduces the effective spin rate per player. Players perceive the game as "stuttering" and slow down their play. The deadlock is, in effect, a crude admission control mechanism that prevents the database from being overwhelmed by an infinite number of spins.

The operator’s partition fix removed that throttle. They can now handle 500 concurrent spins, but they’ve noticed that the average session length has increased by 11% because players are no longer being interrupted by retry-induced stutters. That’s good for engagement metrics, but it means the database now handles 11% more writes per session. The WAL I/O bottleneck that emerged at 500 concurrent spins is the direct consequence of removing the deadlock throttle.

This raises a question that the operator hasn’t answered: should you intentionally re-introduce a controlled deadlock rate as a load-shedding mechanism, or should you invest in more I/O capacity and let the database handle whatever the game server throws at it? The former is cheaper but creates a poor player experience. The latter is expensive but predictable.

The iGaming industry is unusual in that the database workload is entirely synthetic — the spins are generated by code, not by human keystrokes. You can predict the exact load profile for any given promotion. The deadlock spike at 300 concurrent spins is deterministic. The question is whether you treat it as a bug to be fixed or a signal to be tuned. The operator chose to fix it. But their infrastructure team has started monitoring the deadlock rate as a leading indicator of when they need to scale their WAL capacity. They’re now treating deadlocks not as errors, but as a telemetry signal that the system is approaching a resource limit. That’s a shift in mindset that most iGaming backends haven’t made yet — and it’s one that might matter more than any query optimization in the next year as more states legalize online slots and the concurrent spin count pushes past 1,000.