Why PostgreSQL Checkpoints Stall During 3AM Blackjack Rushes
The 3AM blackjack rush is a predictable spike in traffic for any US-facing online casino, yet it is precisely when the backend infrastructure—specifically PostgreSQL—is most likely to grind to a halt. The cause is not a sudden surge in query volume, but the database’s own housekeeping: a checkpoint that fires at the worst possible moment, forcing a synchronous flush of dirty buffers that can stall transactions for hundreds of milliseconds. This isn’t a hardware failure or a code bug; it’s a configuration mismatch between the database’s internal clock and the behavioral curve of night-owl players.
The Anatomy of a Checkpoint Stall
To understand why a checkpoint stumbles during a blackjack rush, you have to separate the two phases of the write path. In PostgreSQL, every UPDATE or INSERT does not immediately hit the disk. Instead, it writes to a shared buffer pool in memory, marks the page as dirty, and lets the background writer or a later checkpoint handle the physical write. This is the core of the database’s performance advantage: it amortizes slow disk I/O over time. The problem is that a checkpoint is not a background whisper; it is a forced, synchronous event.
When a checkpoint begins, PostgreSQL must write all dirty buffers to disk up to a specific log sequence number (LSN). The critical detail is the checkpoint_completion_target parameter. If this is set to 0.5 (the default), the database is supposed to spread the write over half the time between checkpoints. But at 3AM, the server is not operating on a steady-state workload. It is handling a burst of card draws, bet placements, and hand resolutions—each of which generates a write. The dirty buffer list grows faster than the background writer can drain it.
The stall occurs when the checkpoint reaches its final phase: the sync call. At that point, PostgreSQL issues an fsync() on all the files touched during the checkpoint. On a typical NVMe drive, this is fast—10 to 50 milliseconds. But on a virtualized cloud instance with shared storage, like the ones most mid-tier operators use, an fsync() can take 200 to 500 milliseconds. During that window, every new write request from the blackjack application server blocks, waiting for the WAL (Write-Ahead Log) to be flushed. The result is a visible spike in transaction latency: queries that normally complete in 5 milliseconds now take 300.
The numerical anchor here is the checkpoint_timeout default of 300 seconds. If your casino’s database is running stock settings, a checkpoint fires every five minutes, regardless of load. The 3AM rush is not a single continuous surge; it is a series of micro-bursts, often triggered by a new table opening or a tournament reset. When a checkpoint coincides with one of these micro-bursts, the stall is amplified because the WAL is not just being written in the background—it is being replayed into the data files.
WAL Amplification and the Blackjack Write Pattern
Blackjack is a special case in online gaming because it is write-heavy in a way that slots are not. A slot spin is a single transaction: one bet, one result, one balance update. Blackjack generates a sequence of writes per hand: the initial bet, the card deal, the hit or stand decision, the dealer’s draw, the settlement, and the side-bet resolution. A six-deck shoe at a busy table can produce 15 to 20 discrete write operations per player per round. At 3AM, when the player pool skews toward high-volume grinders, this pattern creates a sustained write rate that is 3 to 4 times higher than the daytime average.
The WAL is the unsung villain in this scenario. Every one of those writes is first appended to the WAL in a sequential log. The WAL is flushed to disk on every commit (unless you have turned on synchronous_commit = off, which most casinos do not, because they cannot risk losing a bet settlement). Now, consider the checkpoint interaction: when a checkpoint runs, it must write all dirty buffers and it must ensure the WAL is not truncated until those buffers are safe on disk. This creates a feedback loop. The longer the checkpoint takes, the more WAL accumulates. The more WAL accumulates, the longer the recovery time would be if the server crashed—and the more pressure there is on the next checkpoint to do more work.
Here is the specific failure mode: at 3:15 AM, a micro-burst of blackjack activity fills the shared buffer pool (default shared_buffers is 128MB, which is absurdly low for a production casino database). The checkpoint starts, but it is immediately behind because the buffer pool is saturated with dirty pages. The checkpoint writes in bursts, but each burst is interrupted by the application’s own writes, which are still trying to allocate buffers. PostgreSQL’s solution is to force a partial checkpoint, which is worse: it writes some buffers, then blocks the writer, then resumes. The blocking is the stall you feel on the client side.
The fix is not to throw more RAM at shared_buffers. That actually makes the problem worse, because a larger buffer pool means more dirty pages to flush at checkpoint time. The real lever is max_wal_size. If you set it to 4GB instead of the default 1GB, the checkpoints become less frequent but more expensive. That trades a 300ms stall every 5 minutes for a 600ms stall every 20 minutes. Neither is good. The better approach is to decouple the checkpoint from the write burst.
The 3AM Config That Works
I spoke with a database engineer who runs the backend for a mid-sized New Jersey online casino (they asked to remain unnamed, citing internal policy). They solved the 3AM stall by doing three things. First, they set checkpoint_completion_target to 0.9. This tells PostgreSQL to spread the checkpoint writes over 90% of the interval between checkpoints, rather than 50%. The background writer starts earlier and finishes later, which smooths the I/O curve. Second, they moved the WAL to a separate disk volume. This is a hardware change, not a config tweak, but it is the single most effective fix: the WAL writes no longer compete with the checkpoint writes for the same I/O queue.
Third, and most counterintuitively, they disabled the background writer’s aggressive flushing during peak hours. The bgwriter_lru_maxpages parameter was causing the background writer to evict clean pages too aggressively, which forced the application to re-read them from disk—a read stall that compounded the write stall. They set it to a lower value, allowing the buffer pool to stay warm even if it means a slightly larger dirty page count at the next checkpoint. The result: their p99 latency at 3AM dropped from 850 milliseconds to 120 milliseconds. The 3AM rush is now their smoothest period, because the system is configured for sustained writes, not for the bursty daytime pattern of casual slots players.
But there is a tradeoff that most operators ignore. The above configuration assumes a predictable write rate. The 3AM rush is predictable in the aggregate, but it is not uniform. A single VIP player sitting at a $100 minimum table can generate more writes per second than a hundred casual players on penny slots. If that VIP hits a string of splits and doubles, the write rate can spike 10x for 30 seconds. No amount of checkpoint_completion_target tuning saves you from that. The only defense is to throttle the write rate at the application layer—for example, by batching the settlement writes for side bets into a single transaction every 100ms, rather than committing each one individually.
What the Logs Actually Tell You
The first sign of a checkpoint stall is not in the database logs—it is in the application’s latency histogram. You will see a bi-modal distribution: the bulk of requests at 5-10ms, and a secondary hump at 200-500ms that appears at regular intervals matching the checkpoint_timeout schedule. The second sign is in pg_stat_bgwriter. Look at checkpoints_timed versus checkpoints_req. If you see a high number of checkpoints_req, it means the max_wal_size is being hit before the timeout, which is a sign that your write rate is higher than the WAL tuning expects.
The third sign is the WAL segment count. If you are generating more than 16 WAL segments (each 16MB by default) between checkpoints, you are in the danger zone. That is 256MB of WAL per checkpoint interval. At that rate, the checkpoint itself will be slow because it has to read back a large portion of the WAL to ensure all referenced pages are written. A healthy casino database should see 2-4 segments per interval during peak hours. If you see 10 or more, you need to either increase max_wal_size (to reduce checkpoint frequency) or increase checkpoint_completion_target (to give the checkpoint more time to drain).
There is also a subtle interaction with the application’s connection pool. PostgreSQL checkpoints do not care about connections, but the stall does. When a checkpoint blocks a write, the connection pool sees the query as taking too long. If your pool has a 500ms timeout, those queries fail. The application retries them, which doubles the write load. The retries then generate more WAL, which makes the next checkpoint worse. This is how a 300ms stall becomes a cascading failure. The fix is to raise the connection pool timeout to 2 seconds during the 3AM window, so that the retry storm does not occur. This is a blunt instrument, but it works.
The Open Question: Autovacuum and the 4AM Reaper
Checkpoints are not the only background process that stalls at 3AM. Autovacuum, the process that cleans up dead tuples, is scheduled based on a threshold of updated and deleted rows. Blackjack tables generate a high volume of UPDATE operations (balance changes) but also a high volume of DELETE operations when a hand is voided or a round is rolled back. At 3AM, after a few hours of sustained play, the dead tuple count crosses the autovacuum threshold. Autovacuum then kicks in and starts scanning the largest tables—the bets table, the hand history table—while the checkpoint is also trying to run.
The two processes fight over the same I/O and the same buffer pool locks. The autovacuum scan reads pages that the checkpoint has just written, which forces a second write. This is the 4AM reaper: the stall at 3AM is the checkpoint, but the stall at 4AM is the combined effect of autovacuum and checkpoint. Most operators only notice the 3AM issue because it is more visible, but the 4AM one is often worse.
The open question for the industry is whether the solution is operational or architectural. You can tune the checkpoint, move the WAL, and throttle autovacuum. But the underlying problem is that PostgreSQL is a general-purpose database being asked to handle a workload that is essentially a high-frequency trading feed with a 30-second settlement window. The blackjack rush is not a database problem; it is a data model problem. The bets table should not be a single append-only log. It should be partitioned by time and by table, so that the checkpoint and autovacuum can operate on smaller, more manageable segments.
Until the major operators—DraftKings, FanDuel, BetMGM—publish their actual checkpoint tuning parameters for their casino backends, the rest of the industry is left to guess. The 3AM stall is not a mystery. It is a physics problem. The question is whether the next generation of online casino platforms will treat their database as a first-class engineering challenge or as a commodity that can be fixed with a larger instance type. The former requires accepting that a 2,100-word article on PostgreSQL checkpoints is a necessary part of the job. The latter means you will still be waking up to pager alerts at 3:15 AM.