DatologyAI SafeTulu: Gradient Filtering Finds Benign-Looking Data That Breaks Model Safety, Cutting Attacks Up to 22%

A fine-tuning dataset does not need harmful examples to make a model less safe. A paper from DatologyAI, "Follow the Gradient: Tracing Safety Degradation Back to Training Data," shows a way to find the examples that do the damage, and releases a filtered dataset built with it. The paper was submitted to arXiv on October 27, 2026 and was added to the Hugging Face daily papers on November 2.

The method is called SimpleLoRIF. The dataset is SafeTulu, a cleaned version of the Tulu 3 SFT mixture. This post explains the idea, the reported numbers, what the authors say they do not know, and how you can use the released Space. The sources are the arXiv abstract, the DatologyAI write-up and the Hugging Face paper page.

At a glance

The problem: safe-looking data that is not safe

Earlier work, which the DatologyAI post cites, showed that fine-tuning an aligned model on popular harmless datasets such as Alpaca and Dolly can weaken its guardrails. The troublesome examples are often ordinary: math problems, bulleted lists and simple coding tasks. A content filter, such as an LLM that rates each example for safety, has little to catch because nothing in the text looks harmful.

The authors argue the danger sits in how an example changes the model, not in what it says. So they look at gradients, the direction a model moves when trained on one example. Two examples with similar gradients push the model the same way, even if they look unrelated.

This links to the emergent misalignment research we have covered, where narrow training such as writing insecure code makes a model broadly misbehave. See our posts on Anthropic's automated emergent misalignment search and the Evil Qwen steering reproductions.

How SimpleLoRIF works

Gradient methods for data attribution have been a small-scale tool. A per-example gradient is as large as the model, so embedding a whole post-training dataset is impractical. SimpleLoRIF builds on LoRIF, a low-rank influence method, and makes three changes:

1. Compression. Each document is represented by its gradient at the model's linear layers, compressed by a two-sided random projection and then low-rank factorization. Each layer's representation shrinks from millions of floats to 64.
2. No curvature term. Because the goal is to rank examples for filtering, not to estimate exact influence, the authors drop LoRIF's curvature correction and score by gradient similarity alone.
3. Four blocks only. Backpropagation runs through only the final four transformer blocks. The authors ran ablations on four axes: projection dimension, token truncation, embedding model size and number of blocks. Block count preserved quality while cutting cost. That gives the reported 91.53% speedup while keeping a detection AUC of 0.684 on their math test.

The result: all 939,343 Tulu 3 examples embedded in just over an hour on 8 H100 GPUs. The authors say this makes filtering mixtures with millions of examples practical.

How SafeTulu was built

The pipeline has four steps:

1. Embed every Tulu 3 document and a target set, using Qwen 3 0.6B. The target set is 43,469 prompt-response pairs with unsafe responses from PKU-SafeRLHF.
2. Remove the 20% of Tulu 3 examples with the highest gradient similarity to that set.
3. Train Qwen 3 1.7B, Llama 3.2 1B and Olmo 2 1B on the filtered data, three random seeds each.
4. Evaluate safety and capability.

Four baselines used the same pipeline: full Tulu 3, random 20% removal, an LLM judge (Qwen 3 8B scoring each example 0 to 5), and a prior gradient criterion from He et al. (2024).

The results

On the GCG attack, SafeTulu had the lowest attack success rate for every model. These are the authors' reported figures:

The "up to 22%" headline is the relative drop for Qwen. SafeTulu also beat the He et al. baseline, and the authors say it had the lowest attack success across all three models on ActorBreaker, a harder multi-turn safety test.

Two details deserve attention. The LLM judge barely moved the numbers, which fits the claim that the harmful signal is not visible in text. And the embeddings came from a 0.6B model that is smaller than all three trained models and from a different family than two of them, yet the gains transferred. The authors say this suggests "even a 0.6B model can be a useful monitor for harmful data" bound for much larger models.

On capability, models trained on SafeTulu stayed close to the full-data baseline across six benchmarks: MMLU Pro, IFEval, GSM8K, MATH 500, MBPP and MedQA. Qwen matched or improved on every one. Llama and Olmo showed some minor drops.

What the removed data looks like

The authors measured relative risk of removal for several features. They stress that the patterns are "suggestive, not established":

• Tool-execution failures: removed examples were 1.71 times more likely to contain a traceback or exception.
• Simple coding tasks: plain "write a Python function" tasks were removed at 1.74 times the baseline rate.
• Narrative arithmetic: word problems that carry quantities through prose were removed more often, while examples with explicit algebraic setup were kept more often (0.70 times).
• Hard reasoning: calculus and differential-equation modeling were kept more often (0.51 times).
• Judge vs gradients: the LLM judge removed different math subsets at almost the same rate, while SimpleLoRIF's rate varied by more than 2 times.

One reading the authors offer is that direct, immediately helpful answers may reinforce a pattern of quick compliance. They say plainly that they do not know the identified features cause the safety loss.

The emergent misalignment test

The team also asked whether gradients can link narrow training data to broad bad behavior. They took two misaligned models, collected harmful responses from each, and scored insecure-code and secure-code training examples by gradient similarity to those outputs. Gradient similarity separated the two with an AUC of 0.604 for Qwen and 0.865 for Mistral.

They note that attribution worked much better after the model had been fine-tuned on insecure code. That hints that some harmful data is easier to spot after the model has learned from it, so attribution may help more as an after-the-fact audit than as a pre-filter. Our Talkie 1930 write-up covers another angle on the same phenomenon.

How to use it

The SafeTulu Space lets you:

• change the removal fraction instead of using a fixed 20%,
• filter only certain subsets, for example NuminaMath if you want more math data without losing safety, or only multilingual data,
• target specific harms by joining each example's nearest SafeRLHF neighbor with its harm-category label, for example removing only data near "Endangering Public Health" for a medical assistant,
• run nearest-neighbor analysis and share or upload the result to the Hub.

The authors also suggest screening a newly collected dataset for safety-degradation risk before merging it into a larger mixture, and scoring a mixture against sycophantic responses to see which subsets reinforce sycophancy. Those are proposals, not results in this paper.

If you fine-tune open models, a practical test is to run your own mix through the Space, compare the removed 20% with your intuition, then train two small models, one on each version, and run your own attack and capability checks. Our guide to Nathan Lambert's Trillium Labs launch gives context on the open post-training ecosystem that Tulu belongs to.

Limits to keep in mind

• Small models. All three trained models have 1B to 1.7B parameters. The paper does not show the effect at frontier scale.
• One attack family. The headline metric is GCG attack success, plus a multi-turn test. It is not a full safety evaluation.
• Authors' numbers. The results come from DatologyAI, a data company that sells data curation. We found no independent replication yet, and the paper is days old.
• Costs of removal. Dropping 20% of a dataset is a large cut. The authors report minor capability drops for two models.
• Causes unknown. The features that correlate with removal are not shown to cause the safety loss.
• Short-token ablation risk. The write-up warns that reading only the first 128 or 256 tokens worked on their math test but may not generalize to longer documents.

Why this matters

Most safety work focuses on the model after training. This paper moves the check earlier, to the data, and does so in a way that scales to datasets of a million examples. If the result holds up, data teams could screen new sources before they enter a mixture, and auditors could trace a bad behavior back to the examples behind it. The reported size of the gain is modest and the tests are small, so treat it as a promising tool to try, not a fix.

Related reading

• Anthropic's automated emergent misalignment search
• Evil Qwen two-neuron steering: what is verified
• Talkie 1930 model and emergent misalignment
• Thinking inertia: LLMs keep reasoning when told not to
• Nathan Lambert's Trillium Labs launch