Breaking the WAL

18 points | by wwilson an hour ago

13 comments

  • minimaltom 9 minutes ago

    I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite.

    I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test.

    The closest I mention of the prompt I could find was:

    > Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload.

    With a skeptical lens, unclear.

      gamegoblin 3 minutes ago

      Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing.

      Looking forward to seeing the other bugs they found.

      1. https://news.ycombinator.com/item?id=49278351

  • carlsverre 18 minutes ago

    Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug

    I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.

    If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!

    [1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

      MPSimmons a minute ago

      How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.

  • wwilson 30 minutes ago

    We wanted to publish this ASAP since people are talking about this bug today. In a follow-up, we will show how our automated causality analysis could have saved Tailscale and SQLite from 6 months of root cause analysis. Stay tuned!

  • john_strinlai 32 minutes ago

    related: Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug (tailscale.com) https://news.ycombinator.com/item?id=49272832

  • grebc 26 minutes ago

    Worst ad I’ve ever read.

    Wow, with prior knowledge of a bug our tool found it too!

    Great work pal, I’m sure the phone is gonna ring off the hook.

      carlsverre 23 minutes ago

      Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.

        grebc 18 minutes ago

        I read your ad.

        It states you replicated the bug once the SQLite team fixed it, and published it.

        Not sure what’s difficult about replicating behaviour when it’s spelled out for you.

          biorach 15 minutes ago

          this is a deliberate mischaracterisation of the content of the article

            grebc a few seconds ago

            You’re mischaracterising an ad, that’s also funnily enough full of it.

      wwilson 23 minutes ago

      As the post (and linked repo) pretty clearly indicate, zero info about the existence of the bug went into this work.

      Oh btw, we also found some other ones… Stay tuned!

        grebc 22 minutes ago

        So you found this before SQLite published the fix?