There is a category of distributed system failure that is harder to catch than an outright crash. A block replication slows down but does not stop. An API gateway starts timing out on 3% of requests. A dependency stops responding but only under specific load patterns.
These do not produce a single alarming log line. They produce a sequence of individually plausible log events whose ordering is wrong. You can only catch them by reading the sequence, not the individual entries.
The previous stages of this series built two tools for exactly this. In Stage 2, a fine-tuned Qwen-3B language model reads a block of raw system logs and produces a structured diagnostic: severity level, root cause, recommended shell command. One event takes approximately 5,300 ms. In Stage 3, a Mamba S6 sequence model scores each log event for statistical surprise how unexpected is this token given what came before it. One event takes 0.41 ms.
These two numbers define the engineering problem. Routing all traffic through Qwen at 10,000 events per second means clearing the backlog in 14.7 hours. Routing everything through Mamba gives you speed but no diagnosis. You need both, and on a consumer 8 GB GPU, fitting both in memory at the same time is not straightforward.
Stage 4 builds and tests a system that does exactly this. Over 707 minutes of continuous execution, 27,200 events, two completely different log formats, on one GPU.
TL;DR: NeuroLambda runs Mamba S6 (1.9GB, 0.41ms/event) and Qwen-3B (5.1GB, 4-bit NF4) simultaneously on a single RTX 3060 Ti (8GB). Mamba screens all traffic; flagged events get routed to Qwen for structured JSON diagnosis. Results: F1=1.0 in-distribution, F1=0.9713 zero-shot on microservices data. Adaptive routing cut latency by 51%. Peak VRAM: 7,641 MB. All code is open source on GitHub.
The Core Idea: Apply the Lambda Architecture to Neural Inference
The structure borrows from data engineering. Nathan Marz published the Lambda Architecture in 2011 to handle high-throughput data pipelines. The principle: one fast path handles all volume with approximate answers, one slow path handles the important subset with precise answers. Neither blocks the other.
Applied to neural inference:
-
Every incoming log event passes through Mamba at sub-millisecond speed. Mamba computes a perplexity score a number that measures how statistically surprising this event is given the sequence leading up to it. Normal events have low perplexity and are written to storage.
-
When perplexity crosses a threshold, the event is placed into an asynchronous queue.
-
Qwen picks events off that queue on a separate background thread. It reads the log window and produces a structured JSON report: severity (P0–P3), root cause text, and the shell command to remediate.
flowchart TD
A["Stream\n10,000+ events/sec"] --> B["Mamba S6 Speed Layer\n0.41 ms per event"]
B -->|"Perplexity within range\n~99% of traffic"| C["Write to HDFS DB"]
B -->|"Perplexity spike detected\n~1% of traffic"| D["Async queue"]
D --> E["Qwen-3B Batch Layer\n4-bit NF4, ~5,300 ms"]
E --> F["Structured JSON report\nSeverity, root cause, fix command"]
The arithmetic: a 10,000-event spike routed entirely to Qwen takes 14.7 hours to clear. Through NeuroLambda, where Mamba handles 99% and Qwen handles only the flagged 1%, that clears in 8.8 minutes. Qwen running asynchronously means neither layer blocks the other.
The Ingestion Layer: Vegam
Feeding events into the pipeline at realistic speeds requires more than reading a static file. Vegam is an ingestion adapter built specifically for this series. The name is a Tamil word for speed.
Vegam attaches to live log streams, wraps each entry with a correlation ID and timestamp, computes a surprisal score in bits, and emits a continuous JSONL stream. In the code, a Python class called VegamLogReader spawns a Node.js subprocess running the Vegam CLI. That subprocess feeds raw HDFS and Microservices log entries directly into the Mamba speed layer.
Vegam will be released as a standalone open-source ingestion framework. Any team can point an existing log shipper at a NeuroLambda-style pipeline without rewriting their infrastructure.
How the Routing Threshold Works
The simplest routing approach: if perplexity exceeds the training average by more than 3 standard deviations, send the event to Qwen. This is Static-k. It works when the log distribution is stable and the threshold was calibrated on the same kind of data the model will see in production.
The problem appears at distribution shift. A Microservices API gateway has a fundamentally different natural entropy level than HDFS block lifecycle logs. The vocabulary differs, the event frequency differs, the error patterns differ. A fixed threshold calibrated on HDFS fires constantly on Microservices traffic, flooding the queue with events that do not need LLM analysis.
Adaptive Surprisal Recalibration (ASR) handles this with two mechanisms working together.
The first is an online baseline. The ThresholdEngine maintains a Welford running mean and standard deviation of perplexity scores, updated only on events that pass through Mamba cleanly anomalous events are excluded from the baseline so they cannot corrupt the reference. The threshold tracks what “normal” looks like in the stream currently being processed, not the stream the model was trained on.
The second is a closed feedback loop from Qwen back to Mamba. When Qwen diagnoses an event, its severity output (P0–P3) feeds back into the threshold factor k(t) via an exponential moving average:
$$k(t) = (1 - \alpha) \cdot k(t-1) + \alpha \cdot w(\text{severity})$$
with α = 0.05 and severity weights P0 → 4.5, P1 → 3.5, P2 → 3.0, P3 → 2.5. A critical failure tightens the threshold Mamba becomes more selective. A low-severity event relaxes it slightly. This loop runs continuously and requires no manual tuning.
Alongside ASR, a Page-Hinkley detector watches for structural regime shifts moments when the log distribution itself changes, not just individual spikes. It accumulates deviations from a fixed reference mean established during a 100-event warm-up period. When the cumulative sum exceeds λ = 50, a drift alarm fires and the statistic resets to begin tracking the new regime. The reference mean is deliberately fixed, not rolling: a sustained elevation in perplexity grows the statistic monotonically rather than being absorbed by the estimator.
4 Hard Engineering Problems from Running Two Models for 12 Hours
Running two large models continuously for 12 hours on a consumer GPU surfaces problems that do not appear in short experiments.
GPU Memory Fragmentation
The memory plan on paper: Mamba at 1.9 GB plus Qwen at 5.1 GB equals 7.0 GB, leaving 1 GB on an 8 GB card. In practice, the experiment pipeline runs baseline comparisons before loading the full NeuroLambda stack. During that transition, the process crashed with a 14.2 GB VRAM spike.
The cause is PyTorch’s CUDA memory allocator. When a model variable goes out of scope in Python, PyTorch does not immediately release the underlying GPU memory blocks. The allocator holds them as fragmented reserves, invisible to Python’s garbage collector but still resident on the GPU. When the second model loads, the first model’s fragments are still there.
The fix requires three steps in sequence:
del model_variable # Remove the Python reference
gc.collect() # Force Python garbage collection
torch.cuda.empty_cache() # Release PyTorch's CUDA fragment pool
Skipping any single step brings the spike back. After applying this sequencing throughout the pipeline, peak VRAM across the full 11.7-hour run was 7,641 MB against an 8,192 MB budget, with 390 MB to spare.
A Missing Method in the Wrong PyTorch Version
Loading Qwen in 4-bit precision uses the bitsandbytes quantization library. Internally, bitsandbytes calls set_submodule on PyTorch’s nn.Module class. That method was added in PyTorch 2.5. The experiment environment ran PyTorch 2.4.1. The crash at model initialization gave no clear indication that a version mismatch was the cause.
The fix was a runtime polyfill injected before loading the model:
if not hasattr(nn.Module, "set_submodule"):
def set_submodule(self, target: str, module: nn.Module) -> None:
atoms = target.split(".")
name = atoms.pop(-1)
mod = self
for item in atoms:
if not hasattr(mod, item):
setattr(mod, item, nn.Module())
mod = getattr(mod, item)
setattr(mod, name, module)
nn.Module.set_submodule = set_submodule
This reconstructs the missing method’s behavior and patches it onto nn.Module at runtime without modifying any installed library.
Python Import Conflicts Across Stage Codebases
Stage 4 imports code from both Stage 2 (Qwen) and Stage 3 (Mamba), each of which has its own src/ directory. Python’s import system resolves module names against a single path. When both src/ directories are on the path simultaneously, name collisions produce ModuleNotFoundError for modules that exist but are shadowed. The fix was dynamically extending the __path__ attribute of the shared src namespace package at initialization, allowing both directories to resolve under the same namespace.
Thermal Management Over 12 Hours
Consumer GPUs reduce clock speed when temperature rises past a safe limit. Clock throttling corrupts latency measurements because the same computation takes different amounts of time depending on chip temperature. A GPUThrottleGuard daemon polls the NVIDIA Management Library every second. When temperature exceeded 74°C, it forced a sleep. Between experiment phases, it waited until the GPU cooled to 65°C before proceeding. Over the 12-hour run, the guard intervened 993 times to hold the GPU near 71°C.
Experiment Design
Two scenarios were evaluated:
- Scenario A (in-distribution): HDFS block lifecycle logs with artificial network latency injections. Mamba was trained on this data. Anomaly rate: 34%.
- Scenario B (zero-shot): Microservices API gateway logs. Mamba had never seen this log format, vocabulary, or event structure. Anomaly rate: 47.4%.
Both scenarios ran through five experiment phases:
- E1: End-to-end latency at 50 Hz and 100 Hz event rates (2,000 blocks per rate)
- E2: ASR vs. static thresholds across 10 random seeds, verified with the Wilcoxon Signed-Rank test
- E3: Mean Time to Alarm under a non-stationary stream regime shift
- E4: GPU VRAM co-residency under isolated stress peak reached 6,548.6 MB, confirming both models fit simultaneously with headroom
- E5: Full architecture ablation against all baselines
Total execution time from checkpoint logs: 707 minutes. Total events processed: 27,200 (15,605 normal, 11,595 routed to the batch layer).
Benchmark Results: In-Distribution HDFS and Zero-Shot Microservices
Reproducibility
F1 variance across all 10 random seeds was exactly 0.0000 for both scenarios and both routing strategies. Cohen’s d was 0.0 across every comparison. When a log sequence breaks its expected statistical pattern, perplexity spikes reliably regardless of weight initialization order.
Architecture Comparison
Scenario A: HDFS Block Lifecycle (in-distribution, anomaly rate 34%)
| Method | F1 | Precision | Recall | Speed p95 (ms) | Peak VRAM (MB) |
|---|---|---|---|---|---|
| TF-IDF + Isolation Forest | 0.0114 | 0.2000 | 0.0059 | 10.83 | 997 |
| N-gram Markov | 0.5209 | 0.3636 | 0.9176 | 0.07 | 997 |
| NeuroLambda Static-k | 1.0000 | 1.0000 | 1.0000 | 247.76 | 7,627 |
| NeuroLambda with ASR | 1.0000 | 1.0000 | 1.0000 | 120.76 | 7,565 |
Scenario B: Microservices API Gateway (zero-shot, anomaly rate 47.4%)
| Method | F1 | Precision | Recall | Speed p95 (ms) | Peak VRAM (MB) |
|---|---|---|---|---|---|
| TF-IDF + Isolation Forest | 0.0000 | 0.0000 | 0.0000 | 2.36 | 997 |
| N-gram Markov | 0.6820 | 0.5175 | 1.0000 | 0.07 | 997 |
| NeuroLambda Static-k | 0.9713 | 0.9442 | 1.0000 | 288.58 | 7,625 |
| NeuroLambda with ASR | 0.9713 | 0.9442 | 1.0000 | 197.90 | 7,642 |
Why TF-IDF Fails Completely on Zero-Shot Data
TF-IDF treats anomaly detection as a vocabulary problem. It counts word frequencies and flags unusual distributions, which works only if anomalous events use words that do not appear in normal logs.
In real Microservices logs, that assumption breaks. The word ERROR appears in healthy API responses during expected retry behavior. A timeout logged during a routine request uses the same tokens as a timeout during a cascading failure. TF-IDF sees identical vocabulary and cannot distinguish them. On HDFS: F1 0.0114. On Microservices zero-shot: F1 0.0000.
Mamba does not count words. It models the order in which log tokens appear. A timeout at the wrong point in a request sequence, or following the wrong preceding event, produces high perplexity regardless of which individual tokens are present. The model identified every actual failure in the Microservices dataset without any prior exposure to that log format. Recall: 1.0000. F1: 0.9713.
Why N-gram Markov Stops at 0.68
The N-gram Markov baseline also models sequences, but using fixed-length context windows and exact bigram or trigram counts. It is fast (0.07 ms) and uses no GPU. On HDFS: F1 0.5209. On Microservices zero-shot: F1 0.6820, with precision 0.5175 meaning it flagged roughly half of all normal Microservices events as anomalous.
The Markov model detects sequence breaks but cannot carry context across an entire log block the way Mamba’s recurrent state does. Without that long-range context, it mistakes ordinary API gateway variance for anomalous behavior. Mamba’s deep sequential memory is what separates the two.
The Latency Effect of Adaptive Routing
Both Static-k and ASR achieve identical F1 scores. The detection outcome is the same. The difference is throughput.
The static threshold, calibrated on HDFS, over-triggers on the Microservices stream. Perplexity levels that represent normal API gateway behavior still exceed the fixed HDFS-calibrated cutoff. ASR continuously updates its baseline to the local stream’s entropy, filtering out these micro-spikes before they reach Qwen.
| Dataset | Routing | Speed p95 (ms) | Batch p95 (ms) |
|---|---|---|---|
| HDFS | Static-k | 247.76 | 13,875.5 |
| HDFS | ASR (Page-Hinkley) | 120.76 | 10,288.2 |
| Microservices | Static-k | 288.58 | 9,107.4 |
| Microservices | ASR (Page-Hinkley) | 197.90 | 9,554.2 |
ASR reduced speed layer p95 latency by 51% on HDFS and 31% on Microservices with no change in classification performance.
Reading the Results: What F1=0.9713 on Zero-Shot Data Actually Means
The HDFS result perfect F1 of 1.0000 confirms that dual-model orchestration, memory management, threading, and routing all function correctly without degrading either model. This is the baseline sanity check: a model evaluated on its training distribution should score perfectly, and it does.
The Microservices result is the experiment’s actual question. Mamba learned the statistical grammar of HDFS log sequences: what normal event orderings look like and where the boundaries between normal and abnormal flow lie. When presented with Microservices logs for the first time different vocabulary, different event types, different error patterns it transferred that learned sense of sequential structure. It did not need to know what an API gateway is, what a 504 timeout means, or what tokens Microservices systems use. It recognized that something in the sequence broke the expected flow.
The 2.87% false positive rate (precision 0.9442 on Scenario B) is worth addressing directly. These are Microservices events that Mamba flagged as suspicious even though the ground-truth labels called them normal. This is expected behavior for a model operating on an entirely unfamiliar log format: some API gateway sequences look statistically unusual when measured against HDFS-trained expectations. Whether those events are genuinely benign or represent anomalies the dataset labels missed would require deeper investigation. What the model did not do is fail silently recall held at 1.0000, every labeled failure was caught.
The 31–51% latency reduction from adaptive routing makes a separate and independent point. The resource cost of running an LLM on production traffic is not fixed. A routing layer calibrated to the local stream’s entropy can substantially reduce the fraction of events that reach the expensive model without changing what the expensive model detects.
What Comes Next
Stage 5 will package everything built here into a production-deployable service. The dual-model stack will be containerized, the Vegam ingestion CLI will be open-sourced, and the pipeline will be documented so that any engineering team can connect an existing log shipper without rebuilding their infrastructure.
All code for Stage 4 the dual-model orchestration, the ASR engine, the Page-Hinkley drift detector, the GPU throttle guard, and the full evaluation harness is open source.
GitHub: systems-engineering-labs / surprisal-modeling / stage4-lambda
Join the Conversation
Have questions about this architecture, benchmarks, or pipeline code? Leave a reply below or join our GitHub Discussions.