Postgres recovery_prefetch can cause outages
Posted by plaur782 1 day ago
Comments
Comment by normie3000 5 hours ago
Comment by fakwandi_priv 5 hours ago
Edit: downvoting them now can also help prevent future astroturfing which these accounts are made for, at least I hope.
Comment by opservo 23 hours ago
The part that makes this nasty on cloud disks is that it's latent. On a burstable volume the standby has headroom most of the time, so the amplified reads are invisible until the one day a write burst lines up with a network blip and the volume is already at its IOPS ceiling. Nothing in normal monitoring looks wrong beforehand.
Things I'd check if you can't flip it off immediately: - pg_stat_recovery_prefetch on the standby to see how much prefetching is actually happening vs being skipped - replay_lag in pg_stat_replication, alerted on, not just graphed - standby read IOPS against the provisioned limit during your biggest batch jobs - if you have a single synchronous standby, that's the real outage multiplier here; ANY 1 across two standbys (as the author suggests) or accepting async for that tier changes "slow replica" back into "slow replica" instead of "primary stops taking writes"
Comment by deakmisra 6 hours ago
Comment by fweimer 4 hours ago
I still think some details are missing for a true explanation of what is going on. Where does the random access pattern come from? Is prefetch unnecessarily performed for full-page images?