Linode Security Incident

194 points | by lacoolj a day ago

108 comments

  • 6thbit a day ago

    It feels like we hit an inflection point in the last week or so. Kind of a large tidal wave to crash on the coast, but likely not the tsunami yet. I hope.

    If you are running services and don't yet have an easy way to react to these things with swift process defined, I hope you don't like to sleep.

      georgyo 13 hours ago

      This isn't Linode getting attacked, it is telling customers that this zero click RCE CVE is pretty serious. You should patch ASAP.

      Did you know that GO defaults to listening on TCP connections with MPTCP, so anyone running GO programs listening on the internet (Caddy, traffic, etc) and are on an unpatched kernel now have a way to from attackers to get root on that box?

      Even if you're not using MPTCP, this functions as a DOS attack.

      It's a very bad CVE, so yes. Follow the advisor and patch.

      ianmarcinkowski 11 hours ago

      What if the "inflection point" is just all of the low-hanging fruit that can be detected by agentic vulnerability scanning? If the low-hanging CVEs get patched we might not see a tsunami.

        6thbit 10 hours ago

        I really hope you’re right!

        And even patching the ‘small’ cves will take us for a spin as it already has. Then there can be specific low hanging fruits that chained together make a splash.

      joecool1029 a day ago

      > It feels like we hit an inflection point in the last week or so.

      This isn’t a new thing for Linode. A customer service panel vulnerability in the past was the reason Bitcoinica got hacked in the early bitcoin days: https://spectrum.ieee.org/thousands-of-bitcoins-stolen-in-a-...

      FuckButtons a day ago

      When mythos was released - I guesstimated that we had 6 months until hackers had similar enough capabilities that we would start to see real problems in the wild. Unless there is some change in the trend line things are about to get really quite spectacular.

      rpcope1 a day ago

      I'm just waiting for someone to drop a real zero day on openssh or openvpn for before the real pants pooping starts.

        rurban 12 hours ago

        openssl much more likely. And this will have a bigger impact

        gtech1 a day ago

        If you run openssh open to the public...

      keyle 21 hours ago

      Tidal waves are one thing, this is a tsunami.

      lifeisloving a day ago

      Its really not that hard to secure your servers, especially if your not a high value target. If you are a high value target, you should have security professionals on staff.

      I dont think my security posture is all that much different than it was last year. Use a Secret Manager and or encryption, have backups, strong firewall rules, use canaries of some kind.

      If anything llms are making security worse because there's more people offloading their thinking on llms, or people that skipped some steps in their learning journey because they just ask the llm to do everything. I dont think its necessarily because attackers have llms now, but that contributes.

      Edit:

      Damn Im getting mega downvoted for saying this. So are people of just at a consensus that there's nothing you can do? That we're in security armagedon???

      Have you ever thought to consider that security took hit along with the general degradation of software because of AI??

        moscoe a day ago

        You are literally the everything is fine meme. I admire your confidence. I never realized it was so simple!

        UqWBcuFx6NV4r a day ago

        Is this like when people say “it’s not that hard to write memory-safe C”, and then nobody does? I take this as actually meaning: “I feel that I can keep my shit locked down.”. Whether or not that’s true, which i’m skeptical of, plenty of OTHER people can’t, or won’t, or whatever. That’s the reality.

        To speak to your edit: this is false equivalency. You can still decrease the odds of being hit. There are still better and worse ways to address security.

        mort96 a day ago

        Can you tell me what I should be doing differently then? I just host some services, among them Nextcloud that's written in PHP, protected by SSL, on a server running Linux. And I feel like the past week has been a lots

        tbrownaw a day ago

        It's tedious, which is a particular kind of being hard.

        sanex a day ago

        I agree but disagree about the cause. People have always been bad at security and code in general.

        amelius a day ago

        Have you not heard of supply chain attacks?

        bvcp a day ago

        if anything we should be using ai to find this stuff before the bad actors, too bad the model providers are hell bent on regulatory capture via fear

          OhMeadhbh 9 hours ago

          I know I'll get downvoted for this, but here goes. I was in a position to see some of the mythos derived results early. In my organization everyone was running around with their hair on fire. I asked whether or not any of the vulnerabilities mythos found were also found with existing static analysis tools. The result was blink blink "what are static analysis tools?"

          I'm not saying mythos didn't find anything new, but several people who should have known better were not using SAST/DAST tools to find potential vulnerabilities before pushing their code into the great unknown. One person I talked to said they didn't bother because GitHub automatically runs SAST tools against pull requests. I looked. Their CICD pipeline had no such tool configured. I'm pretty sure it's more appropriate to say GitHub CAN run SAST/DAST tools, if you configure it, but it won't run SAST/DAST tools by default.

          This was a very small sample, but I wonder how common this type of confusion is.

  • spydum a day ago

    Wow, that isn't clear. Is there an incident, or did they just become aware about MPTCP vuln? Details on the mptcp are also pretty scarce as well right?

      csnover 21 hours ago

      It is almost certainly the latter, since it is another “Customer Security Advisory”. There is a currently open report for a tranche of nginx vulnerabilities[0] and a closed report for PwnEscape[1] and the PHP RCE[2] that are identical. The HN topic has been editorialised.

      [0] https://status.linode.com/incidents/8dqqkh4d9y2c

      [1] https://status.linode.com/incidents/zp71823t318w

      [2] https://status.linode.com/incidents/wq8zlhk736fk

      6thbit a day ago

      Yeah its unclear. Sounds like they are just now patching it and treating it as an incident which is probably the right thing to do.

        onei a day ago

        (Former akamai employee) Akamai have an incident process that handles both "things are bad for multiple customers" as well as "we need eyes on this from multiple departments/stakeholders asap". If you're hitting a wall debugging an issue on your own, especially if it's not overtly customer impacting, declaring an incident is an effective way of getting the attention needed to solve it.

        Once you declare the incident, the process includes these status page updates, alongside equivalent internal status updates.

        In this case, they probably wanted to maximise coordination, so used the process designed for that.

          CoastalCoder a day ago

          Also a former Akamai employee.

          I left because I didn't like the pressure / pager duty, but I seriously respect how smooth and well thought out their operations were.

      dofm a day ago

      I am wondering if there is something significant written with Go in their system stack (Go uses MPTCP by default on listening sockets, it seems)

      lacoolj a day ago

      Yeah agreed. I actually only noticed because my two boxes are super slow and I've been troubleshooting for about an hour. Finally opened up the Customer Support page and this was the top incident.

      tbrownaw a day ago

      I haven't seen any emails about it, so I'd lean towards it being something that doesn't rate sending emails.

      dboreham 21 hours ago

      Based on email I got from another big hosting provider, they're all patching for the same checksum metadata vulnerability (the CVE they mentioned is the same). There is however a root RCE PoC according to the researcher so...potentially bad.

  • paaloeye a day ago

    Even better CVE 10.0 has just dropped https://news.ycombinator.com/item?id=50310937

  • nout a day ago

    All companies should at this point invest a lot of money into defensive AI. In ~July the AI hit crypto and bitcoin, where it's the easiest to extract the most money. Now the AI powered attacks are hitting infra and SaaS. Next (~December), it will hit classic infrastructure, banks, small businesses, etc. By mid next year, we will be over this hump with defensive AI being required part of everything.

      jeremyjh a day ago

      Its already happening to classic infrastructure, small and medium size businesses. Absolutely anyone can summon a gremlin capable of exploiting stuff like this or the PHP thing last week. There are clankers probing this stuff all over the place within a day of it going public.

      throwuxiytayq a day ago

      I find it breathlessly funny how people think it's going to be AIs vs AIs fighting it out with light sabers and plasma lasers in the cyber space or some shit.

      Your intuition that this can be balanced out is wrong. When you have a CVE in your TCP stack, your vault unlocks and you're fucked, it's done. My dude, there is never going to be AIs scanning every incoming packet for signs of a 0day, and deciding whether to, uhh, cut it up with a lightsaber and shoot it with a laser pistol. Your options are to have a stack so secure that it can't be targeted, or to disconnect all network cables.

        nout 19 hours ago

        I specifically meant red teaming with AI across your infrastructure to find vulnerabilities before these are exploited. This is what I have been actively doing and it works.

        ForHackernews a day ago

        I assume that "defensive AI" means finding and patching vulnerabilities before attackers do, not some kind of live LLM packet-inspection.

        It is possible to write more secure software[0], companies just choose not to prioritize security.

        [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf

        dist-epoch a day ago

        AI's can decide on the action to open the vault file. It can also prevent TCP connections from unexpected IPs.

  • elashri a day ago

    @dang I would encourage you to really think about having security incident as a new HN tag. The wording can be better though.

      downrightmike a day ago

      We can call it [SI]

        entrope a day ago

        As long as we know that doesn't mean Systeme Internationale, Sports Illustrated or Super Intelligence.

          gk1 a day ago

          > Super Intelligence

          I think that was the joke.

      rglover 20 hours ago

      [whoops]

  • Arbortheus a day ago

    This reads to me that nothing has happened and they are just trying to patch stuff? Right?

  • ericd a day ago

    Anyone have any thoughts on what happens when banks/brokerages start getting hit in earnest? Think it's prudent to split single brokerage portfolios across multiple? Or do they have enough speed bumps in their procedures that they're not quite so vulnerable to major losses?

      pianopatrick a day ago

      If the large and politically connected banks and brokerages have losses large enough to threaten their survival as institutions then the government will jump in to help secure them and bail them out if needed.

      I would stick with a "too big to fail" institution instead of splitting across multiple.

      If you want a real "backup" you could ask for physical stock certificates or get physical gold or something else that cannot be taken from you by computer hacking.

        pixl97 a day ago

        How is the government going to help secure them? Do they even have enough qualified people?

          pianopatrick 21 hours ago

          Well I assume there are some smart hacker / security people in the NSA etc. who could be parachuted in to help if it really came to that.

          I think these banks will probably be able to hire equivalent people in the private sector though and might not need the government help.

            pixl97 9 hours ago

            I work in infosec and work with many of these large organizations.

            Simply put I don't think dropping in smart people in dysfunctional organization is going to have instant results. It will take a pretty long period of time to make them aware of all the systems with holes.

            In the meantime hackers would be running off with actual information and money. More than likely I see banks turning off all kinds of systems in a panic and people with their money there being in an even bigger panic.

          spacephysics a day ago

          FDIC insures deposits up to 250k. Big institutions basically all have this government backing. If an incident occurs, FDIC has deposits available next business day in event of a bank failure

            baobabKoodaa 21 hours ago

            FDIC insurance is mostly unrelated to this. If someone hacks a brokerage, sells a bunch of customer assets for cash, adds new bank accounts for those customers, and then withdraws that cash to those new bank accounts, FDIC insurance does not help at all.

              ericd 20 hours ago

              Another bad one I've heard, buying thinly traded penny stocks as a way to exfiltrate money from brokerage accounts without wiring.

  • dist-epoch a day ago

    I've lived to see the day when half of the Hacker News front page is about hacks. What a time to be alive.

  • nodesocket a day ago

    Anybody recommend a CVE alert self-hosted app? E.G. I specify my software, OS’s, and versions and when a matching CVE is created I get an alert. Don’t want enterprise security products.

  • malloryerik 21 hours ago

    Are banks safe?

  • phendrenad2 a day ago

    Why was the previous discussion of this flagged? https://news.ycombinator.com/item?id=50299182 Is our new policy to stick our heads in the sand and pretend that CVEs don't exist? Because I'm not getting behind that!

  • optimalsolver a day ago

    I mean, this situation is untenable. HN's front page is just going to be an endless list of breaches.

    I don't see how web services proceed like business is normal.

      peri-cl a day ago

      Introducing: HNN (Hacker non-news): a list of web domains *not* currently under apocalyptic AI attack. Shorter and more parsimonious.

      - Tell HNN: pets.com was *not* owned today

      - Tell HNN: about:blank was safe, too

      - Show HNN: my new gopher protocol project—no currently known CVE 10's

        mceachen a day ago

        gopher goes to 11

        (archie and veronica are non-denumerable)

      LtWorf a day ago

      Better than the usual endless stream of what your agent did today.

  • ForHackernews a day ago

    What's the mitigation for this? There's no patches available yet https://ubuntu.com/security/CVE-2026-98432

      markasoftware a day ago

      Mitigation is to retire and wait for the singularity.

      Don't use MPTCP is the first mitigation, but my understanding is this might be exploitable without MPTCP.

        bombcar a day ago

        Filtering packets with ip-options set may also help, but it seems Akami is already doing that at the edge.

      netrem a day ago

      Debian does have a patch up

  • rvz a day ago

    So it begins. The war against the abliterated AI models. Even at 1,000 TPS any service is a target.

    Imagine swarms of abliterated frontier AI models that reach >20,000 TPS. As I said before [0] you are going to have to hire specialized security engineers or technical bodyguards from the senior to senior staff level using AI agents to protect your entire business.

    Otherwise you are cooked.

    [0] https://news.ycombinator.com/item?id=50310312

      jacquesm a day ago

      Guess who has access to abliterated frontier AI models?

      lacoolj a day ago

      Primeagen went over this in the last episode or two - apparently this sort of defense is costing ... one of the big models or companies hosting them ... $900k/day or something to constantly run the best model and monitor 24/7 to combat the attacks [0]

      [0] https://www.youtube.com/watch?v=9OahTki4cPA

        vsgherzi a day ago

        I don’t really have experience running anything like this so this is very much a comment front the peanut gallery but

        Is that really the best way to do things? I’m not really sure what an agent offers here does it just look at logs and do actions? I would not really feel safe if that’s my service’s answer to security.

        I could see it maybe as a really smart filter to block ips that are sending probing requests but I don’t really see that as a good reason to have an agent be sitting 24/7 looking at logs.

        GoblinSlayer a day ago

        I think, it's about reviewing code for bugs.

      shoobiedoo a day ago

      What's the future of cybersecurity going to look like? A rapid shift to shoving everything into microvm's to minimize attack surface?

        chr-s a day ago

        "You got it. Corporate core data for Tessier-Ashpool S.A., and that ice is generated by their two friendly AI's. On par with anything in the military sector, looks to me. That's king hell ice, Case, black as the grave and slick as glass. Fry your brain soon as look at you. We get any closer now, it'll have tracers up our ass and out both ears, be tellin' the boys in the T-A boardroom the size of your shoes and how long your dick is."

          ifwinterco a day ago

          Best science fiction book ever written IMO

            mondrian a day ago

            I recommend Gibson’s (abridged) reading of the book - but only after reading the book. Whenever I read a quote from the book now I hear it in Gibson’s iconic voice.

        lelandfe a day ago

        Airgapping.

        I'm hoping goverments take stock of what is online that needn't be.

        ethagnawl a day ago

        Unikernels (e.g. MirageOS) were gaining traction right before containers swooped in and sucked the air out of the room. I think they could see renewed interest in the very near future.

        Arbortheus a day ago

        OpenClaw agents continuously scanning your prod infrastructure and applying config tweaks and package updates autonomously on the fly in response to viral CVE numbers trending on X.

          alasdair_ a day ago

          The agent sounds like the perfect attack target - it has rights to upload arbitrary code everywhere without setting off alarms.

            Arbortheus 12 hours ago

            I was being facetious, it’s a terrible idea on multiple levels.

        amelius a day ago

        Formally verified code.

        GoblinSlayer a day ago

        Pluggable network stack.

      samwilliams a day ago

      Infrastructure is an FPS now.

        pawelduda a day ago

        One day your job will be to play StarCraft but under the hood it will be your agents vs attacker's agents

      14u2c a day ago

      More likely we'll just see another shakeup of the layers of abstraction that the average company operates at. Why would you hire specialized people when AWS can do so for you?

      GoblinSlayer a day ago

      As I understand, it's LPE. Cloud hosting was always tough.

      xlayn a day ago

      You are being downvoted but I agree, any company including openAI, anthropic, google etc... should be closely regulated by goverment to ensure they never run any AI that doesn't follow all the rules defined by governing bodies, nor train them to have any workaround... NO ONE HAS ABLITERATED ACCESS TO ANYTHING.

      Thanks god, you are not a wacko thinking qwen3.0-1.5B-abliterated would hack a goverment.

      scottrogers86 a day ago

      armadin.com?

  • pugworthy a day ago

    Lately it seems like we need "hacks.ycombinator.com" as a new sub-list.

      keeganpoppen a day ago

      ironic that "hacks" is a needed subdomain of "hacker news"

        optimalsolver a day ago

        Yeah it is, but everyone forgets this board was launched in the 90s for discussion of the 1995 movie "Hackers", not for discussion of hacking in general.

          brian-armstrong a day ago

          I'm still holding out for the sequel

            optimalsolver a day ago

            It would now be 120 minutes of some guy playing EVE or browsing porn while his agent does everything.

      6thbit a day ago

      /incidents would do

      sublinear a day ago

      We need an "ai.ycombinator.com" too.

      minimaxir a day ago

      Show Hacks:

      georgemcbay a day ago

      daily sticky post...

      Ask HN: Who didn't get hacked today.