Can You DNA Test an AI? LLMSurgeon Reverse-Engineers What LLMs Were Trained On

When you deploy a large language model in a regulated environment, you are implicitly trusting that the provider trained it on appropriate data. But how would you verify that? Model cards are self-reported. Training data disclosures are voluntary. And existing techniques like Membership Inference Attacks can tell you whether a specific document was in the training set, but they cannot answer the macro question: what proportion of this model's knowledge came from medical literature versus Reddit versus code repositories?

A paper from Yaxin Luo and colleagues, published at ACL 2026, introduces a framework that does exactly that. Given only generated text from a target LLM, LLMSurgeon estimates the domain-level distribution of its pretraining corpus. It is, in effect, a DNA test for AI models.

---

The Problem: Training Data Is a Black Box

LLM providers treat their pretraining data recipes as trade secrets. This creates a transparency gap that matters in three contexts:

Regulatory compliance. The EU AI Act requires disclosure of training data composition for high-risk AI systems. If a provider declines to disclose, auditors currently have no independent way to verify claims about training data quality.

Model selection for critical applications. When evaluating an LLM for clinical trial support, financial compliance, or legal document review, the proportion of domain-relevant training data is a strong signal of capability. A model trained on 60% web forums and 2% medical literature will behave differently from one with the inverse ratio — but both might score similarly on generic benchmarks.

Bias and safety auditing. The composition of training data directly shapes model biases. Toxic content proportions, demographic representation, and ideological skew are all governed by what the model was trained on. Without visibility into training data composition, these assessments rely on downstream behavioural testing alone.

---

What LLMSurgeon Does

The framework treats training data estimation as an inverse problem. The approach has three stages:

Stage 1: Calibrate a domain classifier. Train a classifier (fine-tuned DistilBERT works best) on reference data across K predefined domains. Compute a soft confusion matrix that captures the classifier's systematic biases — which domains it confuses with which.

Stage 2: Sample the target LLM. Generate text from the model using neutral prompts. Neutral sampling is critical — stylistic prompts (e.g., "write like a scientist") distort the domain distribution and reduce accuracy.

Stage 3: Solve the inverse problem. The classifier's predictions on the generated text approximate a mixture of the true domain distribution, distorted by the confusion matrix. LLMSurgeon inverts this relationship using constrained optimisation, recovering the latent domain proportions.

The key insight is that direct aggregation of classifier outputs fails because errors accumulate. The inverse correction — using the confusion matrix to de-bias the estimates — is what makes the approach work.

---

The Results

The authors introduce LLMScan, the first benchmark with verifiable ground truth, using open-source models with documented training recipes (LLaMA, OLMo, Pythia, StarCoder). They test at three levels of granularity:

Granularity	Domains	Example Model	LLMSurgeon Accuracy	Best Baseline
Coarse	6	OLMo-1B	94.5%	48.1%
Mid	17	Pythia-12B	66.0%	52.6%
Fine	87	StarCoder-15.5B	30.4%	27.5%

At the coarse level — distinguishing web content from code from academic papers from books — the approach is near-perfect, with R² of 0.99. At the fine-grained level (87 programming languages), accuracy drops but still outperforms all baselines.

Additional findings:

• Toxic injection detection. The framework can detect when toxic content has been injected into training data at 5%, 10%, and 20% levels — a forensic capability for safety auditing.
• Training dynamics monitoring. By applying the technique to intermediate checkpoints, the authors reveal curriculum strategies (e.g., one model shows a pattern of fluctuation-then-convergence in its domain allocation).
• Cross-model generalisation. Evaluation protocols transfer across model families without retuning.

---

Why This Matters for Enterprise Leaders

Three implications stand out.

First, independent auditing becomes possible. For the first time, an organisation can verify training data composition claims without provider cooperation. This is significant for regulated industries where "trust us" is not an acceptable answer. If a vendor claims their model was trained primarily on high-quality medical and scientific literature, LLMSurgeon can test that claim.

Second, model selection gets a new signal. Standard benchmarks measure task performance. LLMSurgeon measures the underlying data composition that drives that performance. Two models might score identically on a clinical QA benchmark, but one was trained on 40% medical text and the other on 3%. That difference matters for robustness, edge case handling, and regulatory defensibility.

Third, this complements — not replaces — behavioural testing. LLMSurgeon tells you what the model was trained on, not how it will behave. It is a diagnostic tool, not a predictor. Think of it as a blood test: it tells you what is in the system, but you still need to observe symptoms (behavioural testing, red-teaming, monitoring) to understand the full picture.

---

Limitations Worth Noting

The authors are honest about the boundaries:

• Label-shift assumption. The method assumes domain-specific language patterns remain stable across models. Heavily aligned models (post-RLHF) may violate this assumption, as alignment training can significantly alter how a model represents different domains.
• Closed-world taxonomy. You need to define the K domains in advance. The framework cannot discover unknown domains outside your predefined taxonomy.
• Semantic overlap. Fine-grained domains with high similarity (e.g., C vs C++ programming languages) produce ill-conditioned inverse problems, reducing accuracy at the fine-grained level.
• Closed-source model access. The method requires the ability to generate text from the target model. For fully closed APIs with heavy output filtering, the neutral sampling step may be constrained.

---

What This Means Practically

1. Add data provenance to your model evaluation checklist. Before selecting a model for clinical, financial, or compliance applications, use LLMSurgeon (or its principles) to verify training data composition claims. Do not rely solely on provider marketing materials.
2. Include training data audits in your AI governance framework. If your governance model includes model evaluation, usage monitoring, and outcome measurement, training data auditing fills the missing "measurement" layer — understanding what is inside the model before you deploy it.
3. Watch for regulatory adoption. As the EU AI Act enforcement matures and regulators seek independent verification tools, techniques like LLMSurgeon may become part of the standard auditing toolkit. Organisations that have already built internal capabilities for model provenance assessment will be ahead of the compliance curve.
4. Use it for ongoing monitoring. The technique works on intermediate checkpoints and can detect shifts in data composition over time. For models that are periodically retrained or fine-tuned, this provides a way to detect unexpected changes in the training data mix.

---