Yesterday, we made Postgres 16 default for new clusters on Crunchy Bridge. Our loose policy is to wait for at least one minor version release for a new major, and with 16.1 cut last week, it was time (addressing three security vulnerabilities and some 55 bug fixes).
In the spirit of dogfooding, I upgraded our production servers to Postgres 16 as well. Sometimes we upgrade to get access to a new feature (Postgres 15’s NULLS NOT DISTINCT for unique indexes was one of those), but other times it’s more routine. In the general interest of keeping components current, making sure we were getting our security patches, and nothing falls too far behind.
Postgres 16 was more the latter. It shipped with a whole lot of small optimizations and improvements, but fewer headliners than usual. The exception may be the addition of pg_stat_io, which I haven’t dug into deeply yet, but PG Analyze did a nice write up on.
If you’ll permit me to talk my book for a moment, it still shocks how easy major versions are on Bridge, especially compared to the old days.
Back at Heroku, upgrading core-db was an all-hands-on-deck event, with a Department of Data engineer or two standing by to help out in case something went wrong. Even running on the managed Heroku platform, a major version upgrade is a manual process and necessitates ~30 minutes of downtime.
But although core-db was a Heroku-managed Postgres instance, the API itself wasn’t running on Heroku, instead living on a “kernel” layer that helped ensure we didn’t self-host to the point of developing a circular bootstrapping problem. An upgrade involved giving users advance notice through the status page, provisioning a follower, (a week later) putting the API into maintenance mode, detaching and pg:upgrade‘ing the follower, and promoting it to be the new primary, which involved setting DATABASE_URL and running a deploy. Some of what we were doing with the database was pushing it close to its limits, so it was a harrowing process involving a lot of hand-wringing as it was carried out.
Those small windows of downtime weren’t always enough to upgrade fully. Postgres 9.3 introduced checksums, hugely beneficial for detecting corruption, but it wasn’t until six years later in Postgres 12 that there was a way to easily turn them on for an existing database. Before then, a full dump/restore cycle was required, which would’ve meant significant downtime for us thanks to 900+ GB worth of worthless event data that was almost never used (see: there’s always an events table, and periodic reminder to keep your data small). Henceforth, one of my colleagues would obnoxiously remind me once a month, like clockwork, that core-db didn’t have checksums and that we should turn them on.
And one-day’s worth of upgrade stress is actually not that bad on the whole. A major version upgrade of Mongo at Stripe was a multi-quarter process involving immeasurable quantities of documentation, meetings, and engineering resources.
A knock-on problem from that sort of expense is that when upgrades are a lot of work, you’re less likely to do them. We were always multiple versions behind (at both companies), and that meant losing access to new features and optimizations that would’ve been useful.
We’ve put a lot of effort into Bridge to make upgrades as hands-off as possible. The technical aspects of the process aren’t considerably different from what happens on Heroku, but it’s fully automated and once started, doesn’t need input from the user to complete. Postgres clusters have stable, unchanging CNAMEs, so no configuration needs updating.
It’s 95% smooth, but I do usually notice a few timeouts right around when the upgrade happens, presumably as connections are still trying to reach an outdated host. I restart our apps right around when the final failover is happening to minimize them, and because all our API operations use transactions for near perfect idempotency, anything that fails can be safely retried.
When upgrades are hard, they happen rarely, but the converse is also true. When they’re easy, it’s no sweat to do them often. We don’t always use the latest Postgres features, but it’s nice to have them.
Speaking of latest features, I’ve been using the iPhone 15 for about a month. The titanium is nice – grippier than aluminum and nominally reducing the iPhone’s unfortunate “bar of soap” quality. It’s noticably lighter than the iPhone 14, but about 48 hours later you stop noticing, and you never will again.
Besides USB-C, the phone’s best new feature is a new hardware “Action” button to replace the old mute switch, which has been the most difficult-to-understand hardware feature of the iPhone for a decade now. A total of four hardware buttons and one of them is … a mute switch? In an age when silent phone settings are so ubiquitous that even phones in blockbuster movies don’t ring anymore? We may never know what Apple’s secretive industrial designers were thinking with that one.
I mapped my Action button to open the phone’s camera. I’ve long made the case that the time from rest to photo on an iPhone is way too long, requiring a power on and awkward touch interface interaction. And because it’s contingent on the touch screen, it can’t be done by feel alone. The Action button solves this, although since it’s considerably smaller than the volume up button right next to it, I’ve had a lot of misfires so far that end up cranking my audio to 11 instead of opening the camera (you hold the Action button to use it).
Another interesting feature is a new option to only ever charge the phone to 80%. Apple’s documentation does an absolute garbage job of explaining why anyone would use this, but this Stack Exchange answer is good. In short, less maximum charge leads to longer useful lifetime of the battery:
charging to 80% instead of 100% multiplies by 4 the amount of energy the battery will have transferred to you over its life
There’s nothing magic about the number 80. Charging to even lower capacities yields even larger gains, but 80% is a good compromise between longevity and daily usefulness.
Not being a huge phone user, I turned it on. I finish days with closer to 50% charge than 70%, but the phone still gets charged every night, so it doesn’t matter.
Lastly, the phone takes us one step closer to the dream of ubiquitous USB-C, after Apple was forced at knifepoint by the European Union to give their customers this one, small concession 1.
I now travel fully Lightning-free, with a couple USB-C cables to charge everything on me – computer, phone, camera, AirPods – I even bought a USB-C electric toothbrush to round out the set.
It’s beautiful. If only airplanes and hotels had waited one more generation to install USB sockets (it’s still USB-A everywhere), it’d be possible to visit Europe without bringing a power adapter. Maybe in another five years?
The world over is aware by now that Xi and Biden are in San Francisco for APEC (Asia-Pacific Economic Cooperation). The areas where it’s taking place like Moscone are in lockdown, and in a city where refurbishing one escalator is a multi-year project, it’s been enlightening to see how much infrastructure the city’s been able to get in place given only a few days when a dictator or two is coming to town. Miles worth of military-grade fencing’s been erected, along with anti-vehicle barriers and elaborate checkpoints the size of small buildings.
I’ve never seen so many dark SUVs with tinted, blacked out windows. Xi and Biden travel in long motorcades a dozen cars long and with a swarm of dozens more police vehicles acting as escort. Not surprising perhaps, but every leader in the Pacific appears to travel only nominally less lavishly. And not even that explains it. There are entire blocks lined with black, bulletproof cars all the way up and down. It’s the leaders of Pacific countries plus the next three levels of subordinates below them.
It’s great, actually. The area around around Moscone is closed, but paradoxically, more open. There isn’t normally a barrier, but there is six lanes of roaring traffic on every block in every direction, all of which ignores reds on every light rotation, because like fentanyl and theft, it’s functionally legal to do so in California. This week, despite the big event, the city’s a little calmer, a little more peaceful, a little more civilized. It’s gotten a temporary reprieve in its dismal sartorial record as suits are still in fashion in Asia and smartly dressed people wander the streets. Market’s great lasers are ablaze.
I don’t usually share memes – the internet’s overrun with enough of them already, but I’m declaring a once-ever temporary amnesty because when it comes to this rare intersection of global policy and SF, they’re just too juicy. See below.
Until next week.
Link to the apparent Biden/Xi deal to crack down on fentanyl exports.
1 USB-C cables won’t be as profitable for Apple as their vendor locked Lightning equivalents, but they found a new way forward, and have pivoted to selling $50 MagSafe cables instead.