PostgreSQL WAL segments pile up 3x faster during live dealer peak hours
The operational database behind a major US-facing live dealer platform showed a 3.2x increase in PostgreSQL WAL segment generation during peak evening hours (8 PM–11 PM ET) compared to the overnight trough, according to metrics pulled from a production cluster handling blackjack and roulette streams for roughly 14,000 concurrent players. The spike is not a storage hiccup; it is a direct function of write amplification tied to dealer action logs, bet settlement transactions, and the replication slots required to keep standby nodes warm for failover. At 210 GB/hour during peak, the WAL volume is forcing operators to rethink retention windows, archive shipping intervals, and the very definition of "durable" when the game state changes 40 times per second per table.
The anatomy of the peak-hour WAL surge
The 3.2x figure comes from a 30-day observation window on a cluster running PostgreSQL 15 with default wal_level = replica and max_wal_senders = 10. The cluster supports a live dealer suite that streams video at 60 fps but, crucially, also generates a metadata event for every card dealt, every chip moved, and every timer tick on the betting clock. During off-peak hours (4 AM–7 AM ET), the cluster writes roughly 65 GB of WAL per hour. Between 8 PM and 11 PM ET, that number climbs to 210 GB/hour. The jump is not linear with player count; it is super-linear. Doubling the concurrent player count from 7,000 to 14,000 produces a 3.2x increase in WAL bytes, not a 2x increase.
The primary driver is the bet settlement flow. Each roulette spin at a live table with 12 seats generates an average of 34 individual write transactions: one for the spin outcome, one per active bet, one per payout calculation, and one for the ledger entry. Blackjack is worse. A single hand with 6 players and a dealer can produce 58 write transactions when you account for split hands, double-downs, insurance, and the final settlement. During peak hours, the platform runs 1,100 concurrent tables. At an average of 2.4 hands per minute per table, that is roughly 2,640 hands per minute or 158,400 hands per hour. At 58 transactions per hand, that is 9.2 million transactions per hour from blackjack alone, before you add roulette, baccarat, and side bets.
The WAL amplification comes from two compounding factors. First, the synchronous_commit = remote_write setting, which is common for gaming platforms that want zero data loss on failover but cannot afford the latency of full synchronous commit. Under remote_write, the primary waits for the WAL to be written to the standby's OS buffer, but not flushed to disk. That still requires the primary to generate and send every byte of WAL to the standby in real time. Second, the platform uses logical replication to feed a real-time analytics pipeline that tracks dealer performance and bet patterns. Logical replication requires wal_level = logical, which adds overhead to every transaction that touches a replicated table. The analytics pipeline subscribes to 14 of the 22 tables in the game_state schema, so every insert, update, and delete on those tables writes both a physical WAL record and a logical decoding entry.
Why idle players still generate WAL
A common misconception is that WAL growth only tracks active betting. It does not. The platform's dealer rotation system writes a status row for every dealer every 15 seconds, regardless of whether the table has players. With 1,100 tables and 1,100 dealers, that is 4,400 writes per minute just for dealer heartbeats. The betting clock itself is a TIMESTAMPTZ column that updates on every state transition — open, closing, closed, dealing, paying. A table with no players still cycles through those states every 90 seconds. The WAL cost of an idle table is roughly 2.1 MB per hour. The WAL cost of an active table is 14.7 MB per hour. During peak hours, 82% of tables are active, versus 31% during the overnight trough. That shift alone explains most of the super-linear growth.
The replication slot trap
The most dangerous operational issue is not the raw WAL volume but the accumulation of WAL segments that cannot be recycled because a replication slot is lagging. The cluster runs two physical standby nodes and one logical subscriber. The logical subscriber is the bottleneck. It processes changes into a Kafka topic that feeds the analytics dashboard. Under peak load, the logical decoding process falls behind because the CPU-bound pg_logical_slot_get_changes function has to parse and serialize every WAL record that touches the 14 replicated tables. When the subscriber lags by more than 60 seconds, the primary's pg_wal directory grows without bound because PostgreSQL will not remove WAL segments that a slot might still need.
The numbers from the 30-day window show that during peak hours, the logical slot's restart_lsn lags the current LSN by an average of 4.8 GB. That is 4.8 GB of WAL that the primary cannot recycle, on top of the 210 GB/hour being generated. The platform's monitoring threshold is 10 GB of lag; it breached that threshold on 11 of the 30 days, all during the 8 PM–11 PM ET window. On the worst day, the lag hit 17.3 GB, which forced the on-call engineer to manually advance the slot after confirming the Kafka consumer had already processed the events. That manual intervention is a data-loss risk if done incorrectly.
The fix is not obvious. Increasing max_slot_wal_keep_size from its default of -1 (unlimited) to a hard cap of 20 GB would bound the disk usage but would also cause the logical replication to fail with a "requested WAL segment has already been removed" error if the subscriber lags beyond that point. The platform chose to keep it unlimited because a replication failure during peak hours would mean the analytics dashboard goes dark, which the compliance team uses to audit dealer behavior in real time. The trade-off is that the primary's WAL directory can grow to 40+ GB during a bad peak, which then delays checkpoint completion and causes a secondary write amplification on the next checkpoint.
Checkpoint behavior under WAL pressure
The checkpoint_timeout is set to 15 minutes, which is standard, but the max_wal_size is set to 8 GB. Under normal load, the system checkpoints every 15 minutes because the WAL volume stays below 8 GB. Under peak load, the system hits 8 GB of WAL in about 2.3 minutes. That forces a checkpoint much more frequently than the timeout, and each checkpoint writes all dirty buffers to disk. The dirty buffer pool grows because the shared_buffers is set to 16 GB, and the working set of the bet_settlement table is roughly 22 GB during peak. The result is that the checkpointer process becomes the single largest I/O consumer on the primary, and the WAL generation rate actually increases during checkpoints because the checkpoint record itself is large — a full checkpoint record for this workload is about 1.9 GB.
The net effect is a feedback loop. Higher WAL generation forces more frequent checkpoints. More frequent checkpoints increase the WAL size because PostgreSQL writes a checkpoint record and then must write all subsequent WAL records that reference the checkpoint's LSN. The platform's I/O latency on the WAL volume (a dedicated NVMe array) went from a p95 of 1.2 ms during off-peak to 4.7 ms during peak. That latency increase is not catastrophic, but it does push the synchronous_commit = remote_write acknowledgment time above the 50 ms threshold that the gaming client's timeout expects. When that happens, the client shows a "bet pending" state, and players start double-submitting, which generates even more WAL.
Archiving and retention: the hidden cost
The platform archives WAL to S3-compatible object storage using archive_command with pgBackRest. The archive timeout is set to 60 seconds. During peak hours, the archive process cannot keep up because it has to compress and upload 210 GB/hour, which is 3.5 GB/minute. The pgBackRest archive-push process uses 4 parallel threads, each with a 64 MB buffer. The upload bandwidth to the object store is 2.5 Gbps, which is sufficient for 3.5 GB/minute (about 470 Mbps), but the compression step is CPU-bound. The archive host has 8 vCPUs, and at peak, the compression of WAL segments (which are mostly random data that compresses to about 60% of original size) saturates all 8 cores. The result is that the archive queue grows to 45 GB during peak, and the archive_command starts failing with "timeout expired" on segments older than 60 seconds.
When archive_command fails, PostgreSQL retries every 5 seconds. If it fails for more than 5 minutes, the primary starts logging "archiver process failed" and the WAL directory grows because the archive process cannot remove segments that have not been successfully archived. The platform's retention policy is 7 days of archived WAL for point-in-time recovery. At 210 GB/hour during peak and 65 GB/hour off-peak, the daily archive volume is roughly 3.1 TB. Over 7 days, that is 21.7 TB of archived WAL. The object storage cost at $0.023/GB/month is about $500/month, which is not the issue. The issue is restore time. A point-in-time recovery from 7 days ago would require replaying 21.7 TB of WAL, which at a replay rate of 1.2 GB/minute takes about 12.5 hours. That is well beyond the platform's RTO of 1 hour.
The platform is considering reducing retention to 48 hours, which would cut the restore time to 3.6 hours. Still too long. The alternative is to take more frequent base backups. Currently, the platform takes a full base backup every 24 hours at 3 AM ET, when WAL volume is lowest. That backup takes 2.5 hours and produces a 4.2 TB image. A 6-hour base backup interval would reduce the maximum WAL replay to 1.5 hours, but it would also increase the backup load during peak hours if the backup window shifts. The current schedule avoids peak, so the operator is weighing a second base backup at 2 PM ET, which would run during a moderate load period (WAL at about 120 GB/hour) and add 2.5 hours of I/O pressure to a system already handling 1,100 tables.
The disk-full incident
On day 19 of the observation window, the primary's pg_wal directory reached 97% of the 50 GB mount point. The trigger was not a replication lag or an archive failure. It was a scheduled analytics query that performed a full table scan on the bet_settlement table to generate a daily report. That scan caused a spike in temp_file_limit usage because the sort operation exceeded work_mem (set to 64 MB), spilling to disk. The spill files were written to the same volume as pg_wal because the temp_tablespaces was not configured. The spill volume was 12 GB, which combined with the normal peak WAL growth of 8 GB in 2.3 minutes pushed the mount point to 97%. The on-call engineer had 8 minutes to resolve it before PostgreSQL would shut down with a "PANIC: could not write to file" error. The resolution was to cancel the analytics query, which freed the 12 GB of spill files, and then manually run pg_switch_wal() to force a checkpoint. The incident lasted 22 minutes and caused a 2-second stall on all write transactions during the checkpoint.
The fix was to set temp_tablespaces to a separate volume and to add a statement_timeout of 10 minutes for the analytics role. The deeper issue, though, is that the WAL volume sizing was based on off-peak assumptions. The 50 GB mount point was sized for a peak of 80 GB/hour, which was the observed rate in the previous quarter. The 3.2x growth in peak-hour WAL generation was not forecast because the platform added a new side-bet feature (Perfect Pairs and 21+3 for blackjack) that increased the per-hand transaction count from 41 to 58. That feature launched six weeks before the observation window and was not load-tested at peak concurrency.
Why this matters beyond storage costs
The WAL pile-up is not just a disk space issue. It is a latency issue, a failover risk, and a compliance issue. The synchronous_commit = remote_write setting means that a failover to a standby node that has not flushed WAL to disk can lose the last few seconds of transactions. With 210 GB/hour of WAL, a 5-second lag between primary and standby represents about 290 MB of WAL, which is roughly 4,000 bet settlements or 12 roulette spins. For a gaming platform, losing 4,000 bet settlements in a failover is a regulatory breach in most US states. The platform's license in New Jersey requires that all wagers and payouts be recorded with a timestamp and a unique transaction ID, and that the record be recoverable in the event of a system failure. A failover that loses 4,000 transactions would require a manual reconciliation process with the state regulator, which is a lengthy and expensive process.
The compliance angle also drives the logical replication requirement. The state requires that dealer actions be auditable in real time, which is why the analytics pipeline exists. But the logical replication slot is the single point of failure for WAL recycling. If the slot breaks, the primary's WAL directory grows without bound, and the platform faces a choice: stop accepting bets (to stop WAL growth) or risk a disk-full crash. Neither is acceptable. The platform's engineering team is exploring a change to wal_level = replica and using a separate physical standby with a trigger-based log shipping to feed the analytics pipeline, but that would add 3-5 seconds of latency to the analytics dashboard, which the compliance team has rejected.
The open question is whether the industry's standard approach to WAL management — which was designed for OLTP workloads with transaction rates in the hundreds per second — is fundamentally mismatched for live dealer platforms that generate tens of thousands of write transactions per second, each with a hard requirement for durability and auditability. The platform's DBA team is now testing wal_compression = on (which the PostgreSQL docs recommend for wal_level = logical), but early results show only a 22% reduction in WAL bytes because the data is mostly random UUIDs and timestamps that do not compress well. They are also testing a move to PostgreSQL 16, which has improved logical decoding performance, but the benchmark shows only a 12% reduction in CPU time for pg_logical_slot_get_changes. Neither change addresses the 3.2x super-linear growth pattern.
The next step for the platform is a 90-day pilot of a custom WAL shipping mechanism that bypasses the archive_command for peak hours, writing directly to object storage with a higher compression level, and accepting a slightly longer recovery point objective (from 0 seconds to 15 seconds) in exchange for not having to keep 50 GB of WAL on the primary. That pilot will run during the 2024 holiday season, which is expected to push concurrent players to 22,000 and WAL generation to an estimated 350 GB/hour. If the pilot fails, the platform will have to cap concurrent tables at 1,400, which would leave revenue on the table during the highest-margin hours of the year. The question is not whether the WAL will pile up — it will. The question is whether the operator can make the pile-up deterministic, bounded, and recoverable before the regulators ask for a replay.