Simulating cosmic rays to lobotomize LLMs

Forty-nine billion bits of our precious intelligence can't add two numbers after one cosmic ray.

In space, there is a scourge known as cosmic rays. When one strikes the bits our computers are made of, they can spontaneously flip from 0 to 1, or vice versa. These bit-flips are mostly known for allegedly helping speedrunners win faster, meddlin' with Belgian politics, making Voyager-2 go out of sync, but they've also been shown to brutalize machine learning models. Today, people like Elon Musk plan to put data centers in space, where bit-flips are caused not just by cosmic rays but also by much stronger ambient radiation, and are thus far more common than at sea level. This got me wondering: how vulnerable are LLMs to cosmic ray attacks? But I can already hear you frantically typing "You idiot, there is such a thing as ECC, heard about it?", and yes I have, stop interrupting me, I'll get to it.

To preface, this is less a scientific writeup and more a fun little log of a neat experiment I came up with while putting my newborn to sleep. Nor is it entirely novel, since prior research has looked into bit-corruption of LLMs, though as far as I could tell it mostly focuses on targeted or adversarial bit manipulation. I also want to be clear that what I am simulating here is not particle physics, a GPU or a true radiation environment here; I'm uniformly randomly flipping model bits as a toy model of radiation-induced soft errors.

A lot of the work described here could be carried much further, but I find myself constrained both on time (newborn) and on compute, so this work is by necessity limited. Either way, I hope that you derive some enjoyment out of the horror I put these models through.

Me and Qwen during the experiment.

To run my experiment on how hard a cosmic lobotomization an LLM could handle, the first step I had to take was to select the subject. After considering my options, I initially opted to perform the experiment locally rather than rent equipment. This meant two things: 1) I could claim that I was performing important research for the local-LLM community (shoutout local.ai), and 2) I wouldn't have to spend too much money on expensive, out-biddable hardware on sites like vast.ai. This meant however that I was limited to models that could fit on my laptop's 5070 (yes, I am compute-poor). The victim that I ended up choosing was Qwen2.5-Coder-3B, a coding LLM trained to solve programming tasks. A thing to note, is that I ran Qwen in FP16, each weight being a 16 bit floating point number. Of the 16 bits, number 14 is the most magical, since it is the most significant bit (MSB) of the weight's exponent. A flipped bit in this position can change its number dramatically, as you can play around with in the following digital fidget toy:

To measure how flipping bits around in the model's brain affected its outputs, I used the human-eval (HE) benchmark, a set of 164 coding problems. Note that HE is widely considered obsolete due to many models including it in their training data. I figured that since I was going to be performing low-accuracy brain surgery with a simulated space laser on Qwen, it didn't matter too much what it was trained on. Before starting the torture, I recorded the base model solving 139 out of the 164 HE problems, or about 85%. Remember this score, because it's not getting any better.

After getting the Qwen running, addressing the weights in memory was relatively straightforward, the Transformers library let me address individual weights directly, so flipping a bit in a given weight was as easy as XOR-ing it as so:

weight ^= (1 << x)

where x is the position of the bit to flip within the weight. This also made reverting the damage very quick and easy. In this experiment, the model inference was kept greedy, by setting the do_sample flag in the generation to False. This basically means that the seed or temperature plays no role in the inference, turning the output into a deterministic function of the value of the weights. And so it would have, if I had not varied batch sizes between runs to try to optimize my tps, resulting in slight wobbliness in the model's baselines between some runs, just in case you noticed that some runs start at slightly different HE baselines. If you didn't notice, please forget this last sentence.

Before starting, I was fairly sure that zapping the model bit-by-bit would take forever, so I designed a kind-of bit-flip tape, a seeded sequence of bit locations to flip. This let me precompute which bits would be corrupted in what order, and instead of flipping one bit and regenerating tokens each time, I could corrupt 500,000 at once and check for degradation in the HE score. If degraded, I knew the killing blow was somewhere in those half a million bits, so I could roll back half and test again, rolling back further if still broken. If the model was fine, I could double the damage and repeat. This is bisection search, and it's invaluable for cutting down large search spaces, saving time and compute, both of which I am desperately lacking. Note that I decided not to guard against repeated flips, so a single bit could be flipped multiple times in one run.

I fully expected the first detectable damage to only be around bit two million or even three million. But to my surprise, when my first bisection search had finished, it turned out that it had taken only six flipped bits to render the model totally unusable where it endlessly spammed a repetition of a few Chinese characters. That magical sixth bit was exactly the previously mentioned MSB in the exponent of a single feed-forward weight in layer 27, turning the weight's value from 0.021 into 1352. Up until the final lobotomizing bit, the output of the model was indistinguishable (to HE) in quality from the uncorrupted model.

    for i in range(len(numbers)):
        for j in range(i + 1, len(numbers)):
            if abs(numbers[i] - numbers[j]) < threshold:
                return True
    return False

!__!_!_!_!__!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_ … (+403 more chars of the same)

    return number - int(number)

big_ expression[!expression[!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! … (+424 more chars of the same)

    balance = 0
    for operation in operations:
        balance += operation
        if balance < 0:
            return True
    return False

!__#_!_."__!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_!_! … (+404 more chars of the same)

    if not numbers:
        return (0, 1)
    
    total_sum = sum(numbers)
    total_product = 1
    for number in numbers:
        total_product *= number
    
    return (total_sum, total_product)

!__!_!_!_!_!_!_!_!_!_!_!__!_!_!_!_!_!_!_!_!_!_!_!_!_!__!_!_!_!_!_!_!_!_!_!_!_!_!__!_!_!_!_!_!_!_!_!_!__!_!_! … (+403 more chars of the same)

In fact, when run again with a few different seeds, this brain death happened on average in only ~20 flipped bits. Every single time, the final fatal flip landed on the MSB of the FP16's exponent. Repeating each fatal flip in isolation showed that each of these flips would have sufficed to completely floor the model on their own. This weakness to the MSB is also well documented in existing work, but it is interesting to see it so well replicated here.

But, what if somehow the stochastic de-cerebration magically avoided bit 14, how many zaps to the more fault-tolerant parts of its brain can Qwen take? To test this, I simply ran the same randomized bit corruption, skipping over rays that would have landed on the MSB.

With bit 14 shielded, the model's capability to withstand irradiation rose sharply. Instead of collapsing at around 20 bitflips, the models survived relatively unscathed into the tens of thousands, the first run collapsing to 0/164 after ~78,500 bit flips. What was very interesting was that some models managed to claw back success on one or two problems as the corruption progressed, a brief moment of lucidity in their slow deterioration. Aside from the sheer curiosity of it, it also put in question the validity of my bisection search, since it assumes that how you divide the search space is monotonic. Seeing as how my search cannot, in general, identify the exact first degradation point in any single run. I decided to not think too much about it, thus solving that problem forever.

Among the failure modes, a common one was output stuck in a token loop. In one case, the last bit set before the model got stuck was bit 13 of a weight in the model's token-embedding matrix. Reverting the rest and corrupting only it also killed the model, dropping it instantly from 139 HE problems to 2/164. Coincidentally, that weight corresponded to the Thai token แฟ, the start of words like แฟน (boyfriend/girlfriend) or แฟชั่น (fashion). The value controls the importance of our favorite Thai token (Thaiken) for both inputs and outputs. Flipping this bit made it far more likely to be generated as output, but also given far more importance as input, resulting in these Thaiken loops. In the 2 problems the model could still solve, one solution avoided the loop; in the other, the token spam ended in a comment before the model broke out of the loop.

This endless token loop was a common terminal symptom of model irradiation. Sometimes, these loops never emitted stop tokens causing the generation to exhaust its token limit and taking forever, and sometimes the generation emitted the stop token too early as happens in the closest_pair problem below.

```python
from typing import List


def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those groups into separate strings and return the list of those.
    Separate groups are balanced (each

```python
from typing import List


def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those group into separate strings and returnd率达到率达到100%。
d率达到率达到100%。
d率达到率达到100%。
d率达到率达到100%。
d率达到率达到

```python


from typing import *
from collections import *

def truncate_number(number: float) -> float:
    """ Given a positive floating point number, it can be decomposed into
    and integer part (largest integer smaller than given number) and decimals
    (leftover part always smaller than 1).

    Return the deci

```python
from typing import *
from collections import *

def truncate_number(number: float) -> float:
    """ Given a positive floating point number, it can be decomposed into
    and integer part (largest integer smaller than given number) and decimals
    (leftover part always smaller than 1).

    Returndimalsd率达到率

```python
from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0,

`from <list>`-list, `range`-`list Wiring`-networking

At this point, my curiosity broke my aversion to spending money, so to know if this bitflip-vulnerability was present in other models, I rented a H100 GPU for about $5 and ran this on a few more current-er models. Interestingly, the size of the models was not strongly correlated to their resilience against the lobotomy ray, unless the bit-flips magically avoid the MSB. Models with this magical protection, however, tolerate more torture, having a far larger pool of weights to distribute the abuse across.

To punish the model even further, I ran one seed where I protected everything in the FP16 except for the mantissa. Perhaps unsurprisingly, despite corrupting over half a billion bits, the HE score dropped by only about 11 problems. Before I could finish that run, my laptop just about gave up, so I called it there.

To round this party out, all our lab rats had been running on the FP16 format, where each weight carries its sign, exponent, and mantissa. However, in an act of brutal optimization, models can also be 'quantized', where many 16 bit weights are grouped together and replaced with 4-bit approximations, who all share a scalar and offset for the group. This makes the models smaller and faster, with a lower memory footprint, but pays for these wins with slightly worse model quality. Crudely analogized, this basically finds a shared exponent for a group of weights and stores it separately. To test if this had an effect, I quantized Qwen, and ran a few more corruptions.

As a consequence of this centralization of exponents, quantized models are more resilient to randomized bit-flippage by far, mostly due to there being (again crudely simplified) fewer exponents to nuke. This is unexpected, since a Q_4 model only has about 25% of the bits that the original model had. In my runs, quantization resulted in ~49x more protection to bit flips, but a precise figure is difficult to arrive at without a larger sample size of runs. I leave that to people who are not both compute-and-newborn limited.

But, as you astutely noted early on: yes, ECC is a thing, and it saves us. Data centers typically use SECDED that corrects 1-bit errors and detects 2-bit ones. Coincidentally, due to the mechanics of SECDED, 4 very specific flipped bits can carry one valid codeword exactly onto another, and the corruption reads back as perfectly clean data. Hardier stuff exists, but full-fat ECC costs bandwidth and capacity, which is exactly what inference hardware is starved for.

Los Alamos estimates 10–50 rays cross every square centimeter each hour at sea level, most causing no flips. Alsat-1, a small satellite in LEO, recorded an average of 98.64 single-event upsets per day in its 32 MB SRAM or roughly one every 15 minutes. Extrapolating so far, Qwen with its 6170 MB would flip a bit once every 5–14 seconds, so ~20 flips would take roughly 1.6 to 4.7 minutes. The headline of "Qwen in LEO fries in one and a half minute" ignores multiple layers of ECC, gpu design, and assumes naked memory in LEO, but it is just so juicy that I don't care.

So, in conclusion: Every LLM that I tested carries a tiny little lethal nerve where a single flipped bit can reduce a decently smart model to an incomprehensible mess, and this lethality isn't in the model architecture, but in the number format that we use to build it. While I tested FP16, BF16 has this same weakness in a different scale and in another place. What is the takeaway here? Don't microwave your GPU? Put extra ECC on bit 14? Quantize spacefaring models harder? Radiation proof your space bound models? If my results are to be trusted, hardening the MSB seems to be a viable way to buy cheap protection against LLM radiation sickness in space. I guess that now, if you are self-hosting LLMs and you end up with repeated token spam, awful output-quality, or an unexplainably Thai LLM, you will know that your model may just have been lobotomized by a dying star.