Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future.Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool. reply: Not only are they funding open source, they are actively allowing users to host their own control plane via headscale which is a libre implementation of the tailscale control protocol and developed by an engineer who works for Tailscale by day. This instantly made me trust and like them, even if at first I was cautious because I naturally mistrust anything that gets a lot of hype. I've been running headscale on NixOS, hosting my own tailnet since last year and everything just works out of the box. Tailscale lives up to the hype! <3

Well written post, really enjoyed reading it.> A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQLite page[0] outline that it can only ever occur if there are multiple connections open, so the writer and the checkpointer must have been on different threads.[0] https://sqlite.org/wal.html#the_wal_reset_bug

SQLite: 92 million lines of testsDijkstra: Tests can only prove the presence of bugs, never their absence! reply: Everyone knows that tests don't prevent all bugs. But they are very good at preventing known bugs from recurring in the future.

Great read. So glad they took the time to tell this story. (And glad they, as a for profit corporation, took out a support contract with SQLite. I hope they continue to do so even though this problem is resolved.)

Very nice article, and I appreciate SQLite's explanation of the bug too. And how extremely cool Tailscale appears to have been about it (paying for the VFS shim, etc.).I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS into your network layer, I suppose? Tricky stuff. Wonder how that compares to typical etcd snapshot frequencies too. reply: Agreed, I was also wondering about this. Maybe the aggressive checkpointing was for preventing WAL-overflow?

This was really, really interesting - what a triumphant adventure.A few (very, very, very pedantic) things that stood out:> We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).(Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic".In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope my comment isn't a fly in the ointment.Last bugfix point[1] - ugh....

As others have said: great article! I did find myself wanting them to get to the point, but once they started describing the bug and the fix, it was very satisfying. I'm very happy there are companies out there on the frontiers of functionality not only funding fixes and debugging measures, but taking the time to write up the details so we can all benefit.Tailscale just moved up in my priorities list. Was going to host my next website with hostinger, but now I'm going to at least try to run a personal server with tailscale to make it public. I might not be able to figure it all out, and may end up going with the VPS route, but this gave me some appreciation for the company that makes me willing to try the less familiar method.

>In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups.> running boring technology in a non-standard way is a risk.It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chance to be documented cause experts avoided it and regulars didn't get into reply: I've never heard of any reliability reason you shouldn't run checkpoints whenever you want - only performance reasons. Can you elaborate?Backing up sqlite by copying the file (e.g. rsync) while it's open is a surefire way to eventually get corruption caused by a race condition, but it seems like tailscale wasn't doing that. They were probably using the proper sqlite backup API.But you don't need checkpoints for consistency and I think the backup API will not copy both the old and new versions of pages just because they're in the WAL, in other words I think checkpointing then backing up should give you the same pages as backing up without checkpointing. So the whole thing seems unnecessary.

Normal code has 50% to 90% ratio of code coverage by unit-tests. Dynamic-typed languages (Python, Ruby) usually require more, like 100% - 120%.SQLite has 59,000% ratio [1]Yet it didn't help for a bug to left unnoticed for 16 years :( I don't know what we can do for the industry. I doubt one can formally verify a project like SQLite, and keep it maintainable.https://sqlite.org/testing.html

This reaffirms my belief that SQLite is not well suited for systems with significant concurrency. It replaces fopen, not postgres. Although this corruption is a rare bug and sqlite is usually extremely stable, it's usually not worth it from a performance and features standpoint either.Here they were trying to do a backup by forcing a checkpoint and then copying the file. Systems like postgres let you do online continuous backups.

See perhaps recent video "Reliability Lessons From SQLite - Richard Hipp | SSW 2026":> Abstract: SQLite is a C-language library that implements a self-contained, in-process relational database engine supporting full-featured SQL, an advanced query planner, and ACID transactions. By many estimates, SQLite is the most widely used software library in the world today.> Over its 26-year history, SQLite has gained a reputation as software that "just works". This talk goes over the design choices and development practices that have, at least in the opinion of the lead developer, resulted in that reputation.* https://www.youtube.com/watch?v=V_qzqY1bb7I

Great writeup, and it was great to see them step in an pay the developers of SQLite to help them fix the bug. I get tired of corporations asking open source authors to fix problems that affect the corporation for free. And while I'm sure it was frustrating for folks to have these outages, I find such puzzles pretty fun to get to the bottom of.

Maybe it's just me, but the explanations of the cause don't align.One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.vsit thinks some of the pages have been copied from the WAL into the main database file, but they haven’t. Those pages never get written to the database file, and that data is permanently lost.The first says "more were copied than existed" but the second says "fewer were copied than should have been."Like I said, it's probably just me interpreting something incorrectly.

What a brutal bug. I'd never entertain a that bug in SQLite could be causing problems in code I wrote.

I wonder what happens when you give Claude the old version and the logs and ask it to find the bug.

Which SQLite driver for Go does Tailscale use?

Tracking down a 16-year-old edge case in database internals is peak engineering perseverance. Incredible deep dive.

As a simple user of SQLite, I think this level of debugging is incredible and appreciate being a beneficiary of the ecosystem and hard work of others. Thank you!

I lost my Android SMS DB way back in the day (20+ years? dang I'm getting old) because the SMS app's fix when opening the DB detected any issue was to delete it and start fresh.

> SQLite corruption is possible, but it’s highly unusual and not something you should encounter in normal operationIf there's a hardware failure, for example a flaky SD card, it's not out of the question. A mobile app with a lot of usage will see it.(Yes, I know this appears to be a server use case.)

The irony is that the SQLite developers get a support contract iff someone runs off the path in anger and finds an ancient bug. But perhaps that's part of what make it a quality team: devotion thriving without adverse incentives.

> Now we’re in summer, we’re confident that we’ve found the bug, that we understand it—and more importantly, that we’ve fixed it.This is the feeling I chase as a software engineer. It's the greatest motivator.

Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.

"Tailscale Traces Database Corruption to 16y/o ..."Would have been a superior headline*sarcasm

Such a good write up . Having explored a little bit of sqlite internals for a codecrafters challenge i was mildly happy i could follow along what was happening .

Awesome write up. Finding these bugs in such a well used piece of software is like donating to humanity

I wonder if this also affected litestream disproportionately, because litestream also inserts itself into the checkpoint process.

> This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.I feel like they missed a key takeaway from their own argument here, they should not be running a non-standard configuration :)