The Stage 1 model from Part 1 reached F1=0.8923 on HDFS anomaly detection with zero labeled examples. When it flagged an anomaly, the output was a single number:
Block blk_-7243216225639143943 | Perplexity: 6.84 | STATUS: ANOMALOUS
That is not enough to act on. An on-call engineer still has to open the raw logs, read 400 lines, figure out which DataNode failed, determine the severity, and decide what command to run. During a live cascade failure, that manual work takes 30 to 45 minutes.
Stage 2 addresses this directly. The goal: give the model a block of HDFS logs and get back a structured JSON diagnosis naming the failed component, severity level, and the exact shell commands to run. And do it on the same 8GB consumer GPU.
TL;DR: We fine-tune Qwen-2.5-3B (3.09B parameters) using 4-bit NF4 QLoRA in 28 minutes on a consumer RTX 3060 Ti. The result: a model that converts raw log blocks into structured JSON reports with severity level, root cause, and CLI commands to fix it. Binary anomaly F1=0.9085, JSON schema compliance 95.8%. All code is open source on GitHub.
Why the Base LLM Is Completely Useless for SRE (Out of the Box)
Before building a custom pipeline, I tested the baseline: prompt Qwen-2.5-3B-Instruct (unquantized, no fine-tuning) with a system prompt asking for structured JSON output.
The base model produced this:
Here is my analysis of the provided HDFS log block:
Based on the log entries, I can see that block blk_-7243216225639143943 is being
allocated across multiple DataNodes. The WARN message at line 1 indicates a socket
exception while serving the block to IP 10.250.5.161 on port 43374...
My assessment: This appears to be a P1 anomaly caused by network packet loss.
I recommend checking node health and restarting the DataNode daemon.
Narrative prose. Useful for reading, useless for automation. An incident management platform cannot parse that to route an alert or trigger a remediation script.
The target output is a deterministic schema:
{
"is_anomaly": true,
"root_cause": "DataNode packet transmission failure during block replication to 10.250.5.161",
"severity": "P1_CRITICAL",
"affected_component": "DataNode",
"mitigation_commands": [
"hdfs dfsadmin -report",
"sudo systemctl restart hdfs-datanode@10.250.5.161"
],
"confidence": 0.88
}
Across a 4,000-sample validation benchmark, the zero-shot base model produced valid JSON 42.0% of the time and achieved binary anomaly F1=0.6120. After QLoRA fine-tuning, JSON validity reached 95.8% and F1 reached 0.9085.
Why Fine-Tuning Instead of RAG
A natural question: why not inject past incident tickets and runbooks into the prompt using Retrieval-Augmented Generation (RAG) instead of updating model weights?
Fine-tuning and RAG are not competing approaches. They solve different problems.
Fine-tuning bakes recurring failure patterns, strict schema compliance, and domain-specific reasoning into the model weights. It eliminates the prompt-token overhead and format fragility of stuffing schema instructions into every live request. Once trained, the model produces valid JSON reliably without being reminded how.
RAG supplies knowledge that changes after training: new runbooks, yesterday’s architecture changes, active incident tickets, rotating server configurations. Parametric weights cannot incorporate information they were never trained on.
For Stage 2, the goal was establishing the foundational parametric layer. A RAG retriever sits on top of fine-tuned weights, not as a replacement.
flowchart LR
A["Raw Log Block"] --> B["Retriever (RAG Layer)\nFetches live runbooks\n& recent tickets"]
B --> C["Enriched Prompt"]
C --> D["Fine-Tuned QLoRA Model\nSchema compliance\n& failure pattern recall"]
D --> E["Structured JSON Diagnosis\n+ CLI Mitigation"]
How QLoRA Works
Fine-tuning 3.09 billion parameters at 16-bit precision requires roughly 24 GB of VRAM. An 8 GB card cannot hold that. QLoRA solves this with two mechanics.
NF4 Quantization
Standard 16-bit floating-point weights are stored in 2 bytes each. Empirically, neural network weight distributions cluster tightly around zero in a bell curve. Normal Float 4 (NF4) exploits this. Instead of uniform precision across the full number line, it allocates denser precision near zero where 95% of parameters live.
Compressing 16-bit to 4-bit reduces memory by 4×. Double quantization (quantizing the scaling factors themselves) saves an additional ~0.4 bits per parameter with no measurable accuracy loss. The full 3.09B model compresses to ~2.3 GB in this format.
LoRA Adapters
With the base model frozen in 4-bit, we attach small trainable adapter matrices to the attention and feed-forward projections. For a weight matrix $W_0$ of shape [4096 × 4096], LoRA decomposes the update as:
$$W_{\text{effective}} = W_0 + \frac{\alpha}{r}(B \cdot A)$$
Where:
- $A$ maps the 4096-dim input down to rank $r=16$
- $B$ maps those 16 features back up to 4096 dims
- $B$ is initialized to zero, so training starts from the pretrained baseline
Instead of updating 16,777,216 parameters per layer, we update $(4096 \times 16) + (16 \times 4096) = 131,072$ a 128× reduction in trainable parameters per layer.
Across all seven projection types (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) in every block, this totals 21.08 million trainable parameters out of 3.09 billion 0.68% of the model.
flowchart TB
A["Input: HDFS Log Tokens"] --> B["Frozen Base Model\n4-bit NF4, ~2.3 GB VRAM"]
A --> C["LoRA Adapters\nr=16, 21M params (0.68%)"]
C --> D["Matrix A: 4096 → 16"]
D --> E["Matrix B: 16 → 4096\nInit to zero"]
B --> F["Combine Outputs"]
E --> F
F --> G["Structured JSON"]
Total VRAM during training: 5,120 MB (NF4 base + LoRA gradients + AdamW optimizer state). This fits on the 8 GB card with 3 GB to spare.
VRAM during inference: 2,290 MB (no gradient buffers, no optimizer state).
Building the Training Dataset: 4,000 SRE Incident Pairs
Training data was 4,000 instruction pairs formatted in ChatML. Each pair contains a block of HDFS log lines as the user message and a structured JSON diagnosis as the assistant response.
Source corpus: HDFS v1 (He et al., ISSRE 2016) 11,175,629 lines, 575,061 block sessions.
Class distribution in training: 80% normal blocks (3,200 samples), 20% anomalous (800 samples). The raw corpus is 97.1% normal. Without rebalancing, the model collapses to predicting “normal” for everything.
Zero leakage: All 400 test block sessions were segregated before tokenization. Block identifiers (blk_-...), IP addresses, and timestamps in the test set have zero intersection with training samples.
A complete training example:
<|im_start|>system
You are a Site Reliability Engineer analyzing HDFS logs. Given a block of log lines,
produce a structured JSON diagnosis. Output ONLY valid JSON.
<|im_end|>
<|im_start|>user
LOG BLOCK (block_id: blk_-7243216225639143943):
081110 112906 WARN dfs.DataNode$DataXceiver: Got exception while serving blk_-7243216225639143943 to /10.250.5.161:43374
081110 112906 INFO dfs.DataNode$DataXceiver: writeBlock blk_-7243216225639143943 received exception java.io.IOException: Could not read from stream
081110 112906 INFO dfs.DataNode$PacketResponder: PacketResponder 1 for block blk_-7243216225639143943 terminating
081110 112944 INFO dfs.FSNamesystem: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.250.5.161:50010 is added to blk_-7243216225639143943 size 67108864
081110 112945 INFO dfs.DataNode$DataXceiver: Received block blk_-7243216225639143943 of size 67108864 from /10.250.5.161:43374
<|im_end|>
<|im_start|>assistant
{
"is_anomaly": true,
"root_cause": "DataNode packet transmission failure and socket stream IOException during block replication to 10.250.5.161",
"severity": "P1_CRITICAL",
"affected_component": "DataNode",
"mitigation_commands": [
"hdfs dfsadmin -report",
"sudo systemctl restart hdfs-datanode@10.250.5.161"
],
"confidence": 0.88
}
<|im_end|>
The Tokenizer Window Problem
Initial experiments used 15 log lines per block. The data loader silently dropped 80% of samples.
The cause: BPE tokenization of HDFS logs is verbose. An IP address like /10.250.5.161:43374 fragments into ~10 tokens. A block ID like blk_-5986937059778974225 fragments into ~12 tokens. Fifteen log lines pushed most samples past the 512-token limit, and the dataset script discarded them rather than truncating mid-sequence.
Fix: a prioritization algorithm that selects the 8 most informative lines per block, always including lines marked WARN, ERROR, or FATAL, sorted by timestamp. Sample dropout dropped from 80% to 0.8%.
Prompt Loss Masking
The model should learn to generate accurate diagnoses, not reconstruct the system prompt. During training, all system and user tokens are masked with label value -100, which PyTorch’s cross-entropy function ignores. Gradients flow only through the assistant’s JSON response tokens.
What Actually Matters: Ablation Studies on Data Volume, Rank, and Precision
Training ran on a desktop RTX 3060 Ti in 28 minutes (500 steps, gradient accumulation 16, effective 2,000 steps with batch size 8).
Data Volume
| Training Examples | JSON Compliance | Binary F1 | Severity F1 |
|---|---|---|---|
| 100 | 58.0% | 0.7450 | 0.6210 |
| 500 | 84.5% | 0.8520 | 0.7840 |
| 2,000 | 93.2% | 0.8980 | 0.8650 |
| 4,000 | 95.8% | 0.9085 | 0.8890 |
Classification learns fast. JSON schema compliance requires volume. At 500 samples, the model classifies reasonably well but still produces malformed JSON 15.5% of the time. Reliable schema adherence needs 2,000+ examples.
Quantization Precision
Comparing 4-bit NF4 against a 16-bit full-precision baseline:
| Metric | 16-bit | 4-bit NF4 | Delta |
|---|---|---|---|
| Binary F1 | 0.9140 | 0.9085 | -0.55 pp |
| Severity F1 | 0.8980 | 0.8890 | -0.90 pp |
| JSON Compliance | 96.5% | 95.8% | -0.7 pp |
| Training VRAM | 12,288 MB | 5,120 MB | -58.3% |
Under 1 percentage point of accuracy loss in exchange for fitting on consumer hardware. The trade-off is acceptable for production deployment on constrained infrastructure.
Adapter Rank
| Rank r | Trainable Params | Binary F1 | Severity F1 |
|---|---|---|---|
| 8 | 10.54M | 0.8840 | 0.8520 |
| 16 | 21.08M | 0.9085 | 0.8890 |
| 32 | 42.16M | 0.9089 | 0.8893 |
| 64 | 84.32M | 0.9091 | 0.8894 |
Doubling from r=8 to r=16 yields a meaningful improvement. Beyond r=16, gains are negligible while parameter counts double. r=16 is the efficient operating point for this domain.
Thermal Management on Consumer Hardware
During sustained 4-bit matrix multiplication, GPU temperatures reached 84°C within 20 minutes, approaching the hardware thermal throttling threshold. To avoid clock speed reduction without external cooling, the training loop inserted a 10ms micro-sleep after each backward pass and a 200ms pause after each optimizer step. Temperatures stabilized at 68°C. Total overhead: ~60 seconds added to the 28-minute run.
Results
Evaluated across three independent random seeds (42, 123, 999), 81 test blocks per seed.
Validation Benchmark (4,000 samples)
| Metric | Value |
|---|---|
| JSON Schema Compliance | 95.8% |
| Binary Anomaly F1 | 0.9085 |
| Binary Anomaly Precision | 0.9650 |
| Binary Anomaly Recall | 0.8600 |
| Severity Classification F1 | 0.8890 |
Multi-Seed Test Harness (81 blocks × 3 seeds)
| Metric | Value |
|---|---|
| JSON Schema Compliance | 100.0% |
| Binary Anomaly F1 | 1.0000 |
| Inference p50 latency | 3,516 ms |
| Inference p95 latency | 3,865 ms |
| Peak VRAM (inference) | 2,290 MB |
Why the Test F1 is 1.0
A perfect F1 score warrants inspection, not celebration.
After 512-token length filtering (required for VRAM constraints), every surviving test sample is a P1_CRITICAL anomaly block. These blocks are longer precisely because they contain more exception traces, retry loops, and stack dumps. The 8-line prioritization algorithm guaranteed all WARN/ERROR/FATAL lines were included, so each surviving sample has unambiguous failure markers.
For a 3B-parameter model pre-trained on technical text, identifying an explicit Java exception trace in an 8-line window is close to deterministic. The perfect F1 reflects the high signal density of bounded HDFS exception traces, not overfitting.
The more honest number for real-world deployment is the validation benchmark F1 of 0.9085, measured across the full 4,000-sample distribution including normal blocks and ambiguous cases.
Why Severity F1 is 0.0 on the Test Set
All 81 test samples per seed were filtered to P1_CRITICAL. When only one class is present in both predictions and ground truth, scikit-learn’s macro_f1_score is mathematically undefined and returns 0.0. The model correctly classified every sample as P1_CRITICAL. The metric simply cannot measure multi-class discrimination when there is only one class.
This finding has a practical implication: BPE tokenization bloat is severity-correlated. Blocks with rich exception traces (critical failures) survive the 512-token filter; ambiguous informational blocks do not. The Stage 3 architecture eliminates this filtering by processing longer contexts efficiently.
Stage 1 vs. Stage 2 Comparison
| Dimension | Stage 1: GPT-2 | Stage 2: QLoRA Qwen-3B |
|---|---|---|
| Parameters | 124M | 3.09B |
| Trainable params (fine-tuning) | 124M (100%) | 21.08M (0.68%) |
| Training data | 446,500 normal logs | 4,000 labeled pairs |
| Training time | 2h 16m | 28 minutes |
| Binary Anomaly F1 | 0.8923 | 0.9085 (validation) |
| Recall | 84.40% | 86.00% (validation) |
| Structured output | No | Yes (JSON + CLI commands) |
| Inference latency | <1 ms/trace | ~3,500 ms/trace |
| Training VRAM | 1,616 MB | 5,120 MB |
| Inference VRAM | 1,616 MB | 2,290 MB |
The recall difference is real: Stage 1 misses 15.6% of anomalies due to perplexity dilution in long traces. When 390 healthy log lines average down the perplexity of one failing line, the threshold is not crossed. Stage 2 reads the 8-line window holistically and identifies the semantic contradiction directly if a WARN exception appears, it classifies the block as anomalous regardless of how many INFO lines surrounded it.
The Latency Trade-Off
Inference at 3,500 ms per trace means Stage 2 can process roughly 0.28 diagnoses per second. A high-velocity HDFS cluster generates thousands of events per second. Routing all traffic through Stage 2 would bury the processing queue immediately.
This is not a bug; it is a design boundary. Stage 2 is not built to screen all traffic. It is built to diagnose the small fraction that Stage 1 flags.
flowchart LR
A["Cluster Traffic\n~10,000 events/sec"] --> B["Stage 1: GPT-2 Screen\n<1ms latency\n99% passed through"]
B -->|"Normal (99%)\nFast-pass"| C["Log Archive"]
B -->|"Anomalous (1%)\nSurprisal threshold exceeded"| D["Stage 2: QLoRA Diagnose\n~3,500ms batch latency\nStructured JSON output"]
D --> E["PagerDuty Alert\n+ Mitigation Commands"]
Stage 1 screens 99% of traffic at sub-millisecond speed. The 1% that exceeds the perplexity threshold gets routed to Stage 2 for deep diagnosis. The latency of Stage 2 only applies to that small fraction, which is manageable.
Guarding Against Hallucinations
A fine-tuned model can confidently produce plausible-sounding but incorrect diagnoses. Two safeguards constrain this in production:
Confidence thresholding: The JSON schema includes a confidence field. Diagnoses below 0.85 are flagged for human review rather than triggering automated remediation. This prevents a speculative classification from restarting the wrong DataNode.
Substring grounding: An automated validation step checks that the affected_component and key terms in root_cause appear as exact substrings in the raw input block. If the model outputs "affected_component": "NameNode" but the log only contains DataNode entries, the output fails validation and routes to human triage.
What’s Next: Solving the Speed Problem with Mamba (Part 3)
Stage 3 addresses two problems left open here:
-
Context length: The 512-token limit forced us to select 8 lines per block and discard 80% of training samples. A production HDFS failure often spans hundreds of log lines across multiple nodes. Stage 3 evaluates Mamba (Gu & Dao, 2023) as a replacement backbone a recurrent architecture that processes arbitrarily long sequences with constant memory, not quadratic.
-
Real-time screening speed: Stage 1 (GPT-2) is already fast at <1ms/trace, but it still requires O(T²) attention computation and hits OOM past 8,192 tokens. Stage 3 Mamba runs at 0.41ms per log in recurrent step mode with flat memory regardless of sequence length, making it a better speed layer than GPT-2 for the combined pipeline.
Reproducibility
All code, dataset scripts, and evaluation harnesses are open source. Requires Linux or Docker with an 8GB NVIDIA GPU.
GitHub: systems-engineering-labs / surprisal-modeling / stage2-qlora
git clone https://github.com/Ramprasad273/systems-engineering-labs.git
cd ai-engineering/surprisal-modeling/stage2-qlora
docker compose run --rm surprisal-qlora-train bash run_paper_experiments.sh
Expected runtime: ~35 minutes on an RTX 3060 Ti.
References
- Dataset: He, S., Zhu, J., He, P., & Lyu, M. R. (2016). Experience Report: System Log Analysis for Anomaly Detection. IEEE ISSRE.
- LoRA: Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
- QLoRA: Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS.
- Transformers: Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.
- Mamba: Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
Join the Conversation
Have questions about this architecture, benchmarks, or pipeline code? Leave a reply below or join our GitHub Discussions.