Production HDFS clusters fail quietly. A DataNode drops packets. A replication pipeline stalls. No ERROR keyword appears in the logs for the first 40 seconds, because the Java runtime writes INFO lines right up until the socket resets. By the time an alert fires, the cascade has already started.
Traditional monitoring catches this with regex rules: scan for WARN, ERROR, FATAL. The problem is that rules require someone to write them, and they can only catch what someone thought to look for in advance. Silent failures, partial failures, and novel failure modes all slip through.
This post describes a different approach. We trained a 124-million-parameter causal language model exclusively on healthy HDFS logs and used its prediction uncertainty as an anomaly score. The model never saw a single failure example. On 72,661 test traces drawn from a real production HDFS dataset, it achieved F1=0.8923, catching 84.4% of failures at 94.6% precision.
Here is the full engineering account, including what broke during development and where the approach still falls short.
TL;DR: A 124M-parameter causal GPT-2 trained only on 446,500 healthy HDFS log traces achieves F1=0.8923 with zero labeled failures. Detection works by measuring how “surprised” the model is by each log sequence: traces above a statistical threshold get flagged as anomalous. Trained in 2h 16min on an RTX 3060 Ti. All code is open source on GitHub.
Training Data: 11M Lines of Real Production Logs
The source is the HDFS v1 benchmark corpus (He et al., ISSRE 2016), collected from a 2,032-node Amazon EC2 cluster over a 38-hour window. The raw corpus contains 11,175,629 log lines, grouped by block ID into 575,061 complete execution sessions.
We split it as follows:
- Train: 446,500 normal traces (healthy execution only)
- Validation: 55,823 normal traces (held out for threshold calibration)
- Test: 72,661 traces (55,823 normal + 16,838 anomalous)
The model was trained on the 446,500 normal traces. It never saw the 16,838 anomalous traces until final evaluation.
How Perplexity Becomes a Detection Signal
A causal language model assigns a probability to every token given all previous tokens: $P(x_t \mid x_{<t})$. When the model is confident, this probability is high. When the sequence takes an unexpected turn, it drops. The negative log of this probability is called surprisal, and the average surprisal across a sequence is perplexity.
$$\text{PPL}(S) = \exp\left(\frac{1}{T} \sum_{t=1}^{T} -\log P_\theta(x_t \mid x_{<t})\right)$$
A healthy HDFS block operation follows a predictable sequence. A DataXceiver receives a block. A PacketResponder acknowledges it. The FSNamesystem confirms storage. The same structure repeats across hundreds of millions of operations. A model trained on this grammar assigns low surprisal to normal sequences because each step follows predictably from the last.
When a socket resets or a packet drops, the sequence breaks pattern. The model registers high surprisal. Average trace perplexity rises above a calibration threshold, and the trace is flagged.
graph LR
A["Train on Normal Traces\n(446,500 sequences)"] --> B["Learn Execution Grammar\n(12-layer GPT-2)"]
B --> C["Score New Traces\n(frozen inference)"]
C --> D{"Perplexity vs. Threshold?"}
D -->|"PPL <= 1.2685"| E["Normal"]
D -->|"PPL > 1.2685"| F["Flagged Anomalous"]
The detection threshold is set entirely from the validation split:
τ = μ + 3σ = 1.1650 + 3(0.0345) = 1.2685
No failure examples are required to set this threshold. Any test trace with perplexity above 1.2685 is classified as anomalous.
Why Causal, Not Masked
Bidirectional masked models like LogBERT (Guo et al., 2021) score sequences by checking global consistency, reading both past and future tokens at once. For a static document this is fine. For streaming system telemetry it is not.
When a DataNode event arrives in real time, we do not have future tokens yet. A causal model scores every token immediately from past context: $P(x_t \mid x_{<t})$. Surprisal is available the instant the token arrives. This matters for a production screening layer that needs to flag an anomaly before the next event batch arrives, not after the sequence has fully terminated.
There is also an architectural alignment argument. HDFS block operations are sequential state machines. Event A must precede event B. Causal left-to-right modeling mirrors this execution order. Bidirectional models do not enforce this directionality.
What Broke During Development (And How We Fixed It)
FFD Bin Packing
HDFS traces vary from 10 to 400 tokens. In a fixed 512-token context window, naive padding wastes compute on empty positions.
Without packing, a batch of short traces leaves roughly 34% of each window filled with padding tokens. The loss function still processes them; they just contribute nothing useful. At 11 million training tokens, this waste compounds quickly.
Fix: First-Fit Decreasing (FFD) bin packing. Sort traces descending by length, then fill each 512-token window with as many traces as fit, separated by <EOS> delimiters. Multiple independent traces share a window without mixing their sequence boundaries.
The impact measured on an NVIDIA RTX 3060 Ti:
| Strategy | Padding Waste | Throughput | Test F1 |
|---|---|---|---|
| Truncation / Padding | 34.2% | 93,500 tok/s | 0.8645 |
| FFD Bin Packing | 0.0% | 142,000 tok/s | 0.8923 |
A 51.8% throughput improvement and +0.0278 F1 gain from a preprocessing change that took six seconds of CPU time to sort.
The masking logic in the loss calculation is not optional here. Without masking, the model predicts <EOS> after <EOS> trivially, which artificially deflates perplexity and corrupts the anomaly signal:
# evaluate.py masked perplexity over packed sequences
valid_mask = ~((inputs == pad_token_id) & (targets == pad_token_id))
valid_word_counts = valid_mask.sum(dim=1).clamp(min=1)
masked_loss = loss_per_token * valid_mask.float()
sequence_cross_entropy = masked_loss.sum(dim=1) / valid_word_counts
perplexity_score = torch.exp(sequence_cross_entropy)
Only non-padding positions contribute to the perplexity score. This is what makes the threshold meaningful.
Shallow Model Collapse
We tested 2-layer and 4-layer configurations before settling on 12 layers. The smaller models overfit to positional layout artifacts in packed sequences. Training loss hit zero while validation perplexity diverged into the billions. F1 stalled at 0.29 for the 2-layer model and 0.54 for the 4-layer.
The 12-layer model resolved this, reaching validation perplexity of 1.165.
xychart-beta
title "F1 Score vs Transformer Layer Depth"
x-axis ["2 Layers", "4 Layers", "12 Layers"]
y-axis "F1 Score" 0.20 --> 1.00
bar [0.29, 0.54, 0.89]
Our hypothesis: shallow models cannot distinguish between sequence-level execution semantics and the surface-level packing structure. The 12-layer model learns representations deep enough to separate the two.
Out-of-Distribution Tokenization
Our BPE tokenizer was trained exclusively on healthy telemetry. On 1.35 million normal validation tokens, it produced exactly 0 unknown ([UNK]) fragments. On anomalous test traces, it produced 60 [UNK] tokens, all clustering around corrupted disk-write strings and truncated Java exception class names.
This is a cheap auxiliary signal. Before running a full transformer forward pass, a production system could filter tokens for [UNK] presence. Any trace with more than a threshold of unknown fragments is suspicious by vocabulary membership alone.
System Architecture
flowchart LR
A["Raw HDFS Logs"] --> B["Trace Grouping\n(by Block ID)"]
B --> C["FFD Bin Packing\n(512 tokens)"]
C --> D["BPE Tokenizer"]
D --> E["12-Layer Causal GPT-2"]
E --> F["Masked Perplexity"]
F --> G["Threshold τ = μ + 3σ"]
G --> H["Anomaly Flag"]
Architecture Modifications
The base architecture is GPT-2 Small (12 layers, 768 dimensions, 12 heads), with four changes tailored for structured telemetry:
| Component | Default GPT-2 | Our Implementation | Reason |
|---|---|---|---|
| Normalization | LayerNorm | RMSNorm | Lower activation variance under bfloat16 |
| Position encoding | Learned absolute | Rotary (RoPE) | Extrapolates beyond training context length |
| Activation | GELU | SwiGLU | Stability on steep gradients from burst logs |
| Embedding | Separate | Tied input/output | Halves VRAM for the embedding layer |
Training Configuration
| Hyperparameter | Value |
|---|---|
| Hardware | NVIDIA RTX 3060 Ti, 8 GB VRAM |
| Precision | bfloat16 AMP |
| Optimizer | AdamW (β₁=0.9, β₂=0.95) |
| Weight decay | 0.1 (matrices), 0.0 (biases) |
| Gradient clip | 1.0 |
| Peak learning rate | 6e-4 |
| Min learning rate | 6e-5 |
| Warmup steps | 1,000 |
| Total steps | 10,000 |
| Physical batch size | 16 |
| Gradient accumulation | 4× |
| Effective batch size | 64 |
| Training duration | 2 hours 16 minutes |
Benchmark Results: Beating Prior Unsupervised Methods
Evaluated across 72,661 test traces (55,823 normal + 16,838 anomalous). Results are deterministic across five random seeds (42–46) because the threshold is computed analytically from validation statistics, not from a learned decision boundary.
Accuracy: 95.28%
Precision: 94.63%
Recall: 84.40%
F1: 0.8923
True Positives: 14,212
False Positives: 806
True Negatives: 55,017
False Negatives: 2,626
Measured on NVIDIA RTX 3060 Ti, 8 GB VRAM. Inference throughput: ~420 traces/sec at batch size 64, mean trace length 146 tokens.
Threshold Sensitivity
F1 peaks at k=3.0 and stays within 0.005 from k=2.5 to k=4.0. The threshold does not require tight tuning.
xychart-beta
title "F1 Score vs Threshold Multiplier k"
x-axis ["k=1.0", "k=1.5", "k=2.0", "k=2.5", "k=3.0", "k=3.5", "k=4.0", "k=4.5", "k=5.0"]
y-axis "F1 Score" 0.75 --> 0.95
line [0.795, 0.847, 0.874, 0.887, 0.892, 0.891, 0.887, 0.884, 0.877]
The plateau from k=2.5 to k=4.0 means operations teams can adjust the threshold for local precision/recall trade-offs without retraining.
Comparison to Prior Work
| Method | Supervision | Architecture | F1 | Precision | Recall |
|---|---|---|---|---|---|
| Surprisal-GPT2 (Ours) | None | 12-layer Causal | 0.8923 | 0.9463 | 0.8440 |
| LogBERT (Guo 2021) | None | Bidirectional BERT | ~0.875 | ~0.880 | ~0.870 |
| LogGPT (Xie 2022) | None | Causal GPT | ~0.840 | ~0.860 | ~0.820 |
| DeepLog (Du 2017) | Supervised | Deep LSTM | 0.9540 | 0.9600 | 0.9500 |
| Isolation Forest | None | Tree Ensemble | ~0.620 | ~0.650 | ~0.580 |
Our model reaches the highest F1 among unsupervised methods. The supervised DeepLog scores higher because it trains directly on labeled failure examples a different operating assumption entirely.
Honest Failure Analysis: What the Model Still Misses
What Gets Missed (False Negatives)
2,626 real anomalies escaped detection. Most follow the same pattern: a long execution trace with a single failure event near the end.
A 392-token trace with 386 healthy steps followed by one packet drop event keeps its average perplexity low. The 386 normal steps each contribute PPL ≈ 1.08. The single failure step spikes, but when you average 392 values, one spike is diluted:
Overall trace PPL = (386 × 1.08 + 1 × 14.2) / 387 = 1.22 < 1.2685
The sequence slips below threshold. The failure is missed.
The length pattern confirms this. True positive traces have a mean length of 122 tokens. False negative traces average 347 tokens. The longer the trace, the more healthy context dilutes the failure signal.
This is a fundamental limitation of sequence-average scoring. Fixing it requires either windowed sub-trace evaluation or a model that attends to individual spike positions rather than averaging across all of them.
What Gets Falsely Flagged (False Positives)
806 normal traces were flagged as anomalous. Most cluster around rare administrative operations that appeared infrequently in the training split:
| Step | Token | Surprisal |
|---|---|---|
| 1 | [Audit] | 0.45 |
| 2 | [allowed=true] | 0.38 |
| 3 | [ugi=admin] | 4.12 |
| 4 | [ip=/10.10...] | 0.51 |
| 5 | [cmd=listStatus] | 5.24 |
The model has seen very few listStatus admin commands. When they appear, they look surprising. The trace overall PPL = 2.14, well above the threshold.
The fix is periodic recalibration. As the training distribution expands to include more administrative operations, these sequences become expected and their perplexity drops.
VRAM Scaling
FlashAttention tiles the attention computation in SRAM, reducing activation memory without changing the O(T²) arithmetic complexity. The memory cost of extending context stays manageable:
| Sequence Length | VRAM (MB) | Delta vs. T=128 |
|---|---|---|
| 128 | 1,572 | N/A |
| 256 | 1,576 | +0.22% |
| 512 | 1,580 | +0.50% |
| 1,024 | 1,602 | +1.91% |
| 2,048 | 1,616 | +2.81% |
A 16× sequence length increase costs 2.81% more memory. This holds through 2,048 tokens. The arithmetic cost (compute time) still scales quadratically, so long-sequence processing is slower, but it does not run out of memory. Past 8,192 tokens on 8GB VRAM, the quadratic attention hits OOM a hard limit that Stage 3 of this series resolves with a different architecture.
Limitations
Recall gap (15.6% missed): Averaging perplexity across long traces dilutes failure spikes. Sliding-window sub-trace scoring would help but adds latency.
Concept drift: The threshold is calibrated on a 38-hour HDFS snapshot. Software version changes, configuration updates, or shifts in operational patterns will drift the healthy perplexity distribution and require recalibration.
Single-dataset evaluation: Cross-system transfer to BlueGene/L (BGL) logs achieves 0.7840 F1 zero-shot, recovering to 0.8850 F1 after a 1,000-step adaptation phase. The approach transfers, but it is not plug-and-play across radically different log formats.
No root-cause output: The model produces a scalar anomaly flag. It cannot tell you which DataNode failed, what the severity is, or what to do about it. Part 2 of this series addresses that with a fine-tuned diagnostic LLM.
Reproducibility
All training code, preprocessing scripts, FFD packing implementation, and evaluation pipelines are open source.
GitHub: systems-engineering-labs / surprisal-modeling / stage1-gpt2
git clone https://github.com/Ramprasad273/systems-engineering-labs.git
cd ai-engineering/surprisal-modeling/stage1-gpt2
docker compose run --rm surprisal-gpt2-train --full
Expected runtime: ~2 hours 16 minutes on an RTX 3060 Ti.
References
Datasets:
- He, S., Zhu, J., He, P., & Lyu, M. R. (2016). Experience Report: System Log Analysis for Anomaly Detection. IEEE ISSRE. [Zenodo: 10.5281/zenodo.3227177]
- Oliner, A. J., & Stearley, J. (2007). What Supercomputers Say: A Study of Five System Logs. DSN 2007.
Prior work on log anomaly detection:
- Du, M. et al. (2017). DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning. ACM CCS.
- Meng, W. et al. (2019). LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs. IJCAI.
- Guo, H., Yuan, S., & Wu, X. (2021). LogBERT: Log Anomaly Detection via BERT. IJCNN.
- Xie, Y. et al. (2022). LogGPT: Log Anomaly Detection via GPT. IEEE BigData.
Architecture and training:
- Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.
- Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Technical Report.
- Su, J. et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.
- Dao, T. et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.
Join the Conversation
Have questions about this architecture, benchmarks, or pipeline code? Leave a reply below or join our GitHub Discussions.