• Frank [he/him, he/him]@hexbear.net
    link
    fedilink
    English
    arrow-up
    30
    ·
    2 months ago

    If I’m reading this correctly modern AVs work by looking for patterns in software behavior that resemble the actions of currently circulating viruses to try to stay ahead of the rapid proliferation of new viruses and threats. So if program A.) behaves in some way like known virus 1.) the software will shut down program A.), not because it’s a known threat, but because it behaves like a known threat. So if I’m following this guy something in the stream of behavioral information Crowdstrike sends to all it’s client computers in real time flagged some core windows process or something as a threat and began attacking it. This resulted in BSOD bootloops across their network of clients.

    Short version; Computer auto-immune disorder, the immune system is attacking the body because it’s incorrectly identified some part of the body as a threat.

    But for some cloud systems though, such as AWS, booting to “safe mode” is not even possible so this fix can’t be applied. Virtual servers need to be shut down, their disks cloned, attached to another server, edited to remove the offending files and then finally reattached to the original server.

    Lol

    BUT, if you’re protecting your data properly you would have used BitLocker for disk encryption and so you need to manually decrypt the disk with a BitLocker Recovery Key, which is probably - for most companies - stored digitally on one of the servers that is currently booting over and over 🫠

    AHAAAAAHAAHHAHAAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAH HEY FUCKWITS MAYBE PUTTING ALL YOUR SHIT ON REMOTE ALWAYS ONLINE SERVERS OVER WHICH YOU HAVE NO CONTROL WAS A CRITICAL STRATEGIC FAILURE TO RETAIN CONTROL OF YOUR CORE ASSETS YOU DUMB FUCKS YOU GAVE AWAY THE KEYS TO THE VAULT TO A GUY IN BLANK WHITE MASK WITH “TRUST ME” WRITTEN ON IT HAHAHAHAHAHAHAHAHAHAHAHAHA

    The cloud was always an obvious, utterly inexplicable mistake of astonishing proportions and it’s hilarious that capitalism drove everyone to turn their systems in to dumb terminals over which they have little if any control. I’ll just be here basking in my “I called it” from well over a decade ago.

    • Tervell [he/him]@hexbear.net
      link
      fedilink
      English
      arrow-up
      22
      ·
      2 months ago

      Computer auto-immune disorder

      I love software engineering

      anyways, it sure is great to grant kernel-level access to a program so it can better protect you from viruses by, uh… using its kernel-level access to break your entire system?

    • Chronicon [they/them]@hexbear.net
      link
      fedilink
      English
      arrow-up
      7
      ·
      2 months ago

      No, they actually just pushed out a bugged driver that they use to hook into the windows kernel. Turned out to be nothing to do with the realtime A/V feed. Which is honestly funnier because there is NO reason to push that type of update out worldwide in one go, it should be done in stages to catch bugs like this before they go global…

      The OP thread talked about it just below the “Load More Replies” fold

  • LanyrdSkynrd [comrade/them, any]@hexbear.net
    link
    fedilink
    English
    arrow-up
    30
    ·
    2 months ago

    I read somewhere else that this analysis is incorrect. They were saying it wasn’t caused by something in the threat intelligence feed, but an updated .sys file(a driver component) that CrowdStrike inexplicably pushed to all clients at once.

    That explanation is even funnier, because they pushed a software update to everyone at once instead of the widely used practice of staged rollouts of updates. Normally big companies push updates to a very small number of users first, then gradually increasing the number so they can get bug reports before wrecking every system.

  • Vampire [any]@hexbear.net
    link
    fedilink
    English
    arrow-up
    24
    ·
    2 months ago

    His explanation: "CrowdStrike is an antivirus. It updates threats constantly. Then the rest of the problem happened causing everything to crash worldwide.

    It’s a kind of ‘draw the rest of the owl’ explanation

    • blobjim [he/him]@hexbear.net
      link
      fedilink
      English
      arrow-up
      7
      ·
      2 months ago

      Tweets that are like “I am a super credentialed smart person, here’s my analysis of…” are always fart sniffing.

    • Frank [he/him, he/him]@hexbear.net
      link
      fedilink
      English
      arrow-up
      18
      ·
      2 months ago

      My adittedly extremely limited understanding is that modern AV’s do use machine learning to identify emerging and potential threats. Hackers are creating new malware, ransomware, and virus software every day and trying to catch it all isn’t possible. Intead they use machine learning to identify patterns in how hostile software behaves within the computer system and then shut down anything that behaves like that hostile software. I just ran afoul of this with windows defender and trhe Unreal Engine VR plugin project. UEVR injects data in to the Unreal Engine game in real time and that’s a big no-no, that’s something a virus does, so Window shut it down hard and I had to do all kinds of silly bullshit to even get the computer on my folder without Windows detecting it and deleting it.

      Well, when you apply that kind of rough and ready, evolutionary, real time threat modelling to a live system, I guess sometime your black box machine learning bullshit has a false positive and starts punching the global economy directly in the dick.

      Keep in mind, I am not any kind of network security guy, so this is very much an idiot bystander trying to explain the workings of god.

  • SSJMarx
    link
    fedilink
    English
    arrow-up
    18
    ·
    2 months ago

    You know, I’ve complained about how Linux makes you run upgrade yourself and doesn’t automate it, but I guess I’ll never have to worry about some shit like this happening to my machine overnight, lmao.

    • Chronicon [they/them]@hexbear.net
      link
      fedilink
      English
      arrow-up
      12
      ·
      edit-2
      2 months ago

      Distros often do automate it, they’re just better at it than windows so you don’t notice lol (and they usually only enable it by default for security updates, which is the sane way to do it IMO) ubuntu/debian have unattended-upgrade, DNF distros like fedora, rhel-likes, etc have dnf-automatic.

      I’ve never had one break something.

      • SSJMarx
        link
        fedilink
        English
        arrow-up
        2
        ·
        2 months ago

        Maybe it’s just a case of me not knowing where to look, I’m on Debian and I always have to upgrade everything manually.

        • Chronicon [they/them]@hexbear.net
          link
          fedilink
          English
          arrow-up
          3
          ·
          edit-2
          2 months ago

          you can configure it manually

          or some basic configuration options should be available through “software-properties-gtk” under the Updates tab (unsure if this is preinstalled or not, but it is available from apt. It also might show up as “Software & Updates” in the GUI)

          I don’t know if debian does security updates automatically by default, I’ve only used it on servers lately

  • tocopherol [any]@hexbear.net
    link
    fedilink
    English
    arrow-up
    18
    ·
    2 months ago

    From the description this doesn’t sound like it will be fixed right away for most systems, any idea what kind of impact this will have? I would hope for anything crucial there would be fail-safes

    • Frank [he/him, he/him]@hexbear.net
      link
      fedilink
      English
      arrow-up
      22
      ·
      2 months ago

      Pretty sure the US Airline industries requrested that all flights, Globally be grounded due to this failure. Major airlines were already running on the ragged edge of collapse with antiquated systems that could barely function on good days. So, as one says; Lol. Lmao.

      • invalidusernamelol [he/him]@hexbear.net
        link
        fedilink
        English
        arrow-up
        2
        ·
        2 months ago

        US airlines (minus Southwest) still use SABRE for reservation and flight management. A system developed for DARPA in the 50s. Basically everything is run in virtual machines I believe, but there are probably still some SABRE terminals out there.

        Getting that system back up and running will be a nightmare as it’s integrated into basically every reservation service on the planet. That’s probably why they want all flights grounded because anything that happens while the system is down will have to be added manually later.