Notes
Inference
- Aug 2026
5 minvLLM vs SGLangBoth engines batch continuously and page their KV cache. SGLang's radix-tree prefix sharing wins on agent and RAG traffic; vLLM's model coverage and ecosystem win elsewhere. How to measure which applies to you.
- Jul 2026
6 minContinuous batching and paged attentionContinuous batching refills the batch every iteration; paged attention allocates cache memory in small blocks on demand. Each needs the other to keep both slots and memory full.
- May 2026
5 minGPU utilization is not a load signalGPU_UTIL reports whether a kernel was running, not how much of the device it used. A decode step reads 100 percent at batch size 1 and at 64. Scale on queue age and cache occupancy instead.
- Mar 2026
5 minTime to first token and time per output tokenAn LLM response has two latencies: the wait before the first word, and the pace of the words after it. They have different causes and different fixes, and a single latency number hides both.
- Jan 2026
5 minKV cache explainedGenerating each new token would otherwise recompute attention over the whole sequence. The KV cache stores that work so each step is incremental. It is also the reason serving is bounded by memory rather than compute.
- Nov 2025
5 minShadow traffic as a pre-release gateRun the candidate model on a copy of every live request, log its output, and serve users only the current model's. It exercises the real request distribution without exposing users to the candidate's answers.
Search
- Feb 2025
5 minOpenSearch vs ElasticsearchForked from the same code in 2021, the two now differ on licence, vector search, query language, and managed-service cost. A decision rule for new deployments and for teams already on one.
Distributed systems
- May 2023
5 minConsistent hashing, explained with a small exampleWith hash-mod-N, adding one server moves almost every key. Consistent hashing moves only 1/N of them. Worked through with four servers, six keys, and the two refinements every implementation adds.
- Oct 2022
5 minBackpressure in software: what it is and how to apply itWhen a producer is faster than its consumer, something has to give. Backpressure is the set of mechanisms by which the consumer makes the producer slow down, instead of the system falling over.
- Mar 2022
5 minKafka consumer groups and dead letter queuesA consumer group is how Kafka spreads a topic across workers. A dead letter queue is how those workers avoid getting stuck on a message they cannot process. The two are usually configured together and usually configured wrong.
- Sep 2021
5 minIdempotency: designing operations that survive retriesA network call can fail after the server has done the work. The client does not know, so it retries. Idempotency is what makes that retry safe, and it has to be designed in, not added later.
- Nov 2018
5 minCache invalidation: the three strategies that actually shipEvery cache has to answer one question: when does a stored value stop being trusted? There are three practical answers, and most systems end up combining two of them.
- Jul 2018
5 minRedlock and Redis locks: why the TTL is the whole problemA Redis lock is one command to acquire and a short script to release. The difficult part is that the lock can expire while the work is still running, and Redis will not tell you.
- Feb 2018
5 minRedis as a queue: when it works and when it loses jobsA Redis list is a workable job queue until a worker crashes, Redis restarts, or memory fills. Each of those loses jobs under the default settings, and each has a specific fix.
Databases
- Nov 2020
5 minDatabase sharding: when and howSharding removes cross-shard transactions, joins, and unique constraints, so it should be the last scaling step. What to exhaust first, how to pick the key, and how to leave room to reshard.
Node.js
- Jun 2020
5 minThe Node.js event loop and CPU-bound workNode serves thousands of connections on one thread because nothing blocks. A 200 ms synchronous call in a handler delays every connection by 200 ms. How to measure it and where to move the work.
Java
- Jan 2017
5 minUnit test vs integration test: where the line actually isA unit test catches a defect inside one class; an integration test catches a defect in how classes and infrastructure connect. Test at the level where the bug would be introduced, with a Spring example of each.
- Nov 2016
4 minMockito verify: what it proves and what it doesn'tverify() is the right assertion for collaborators whose only observable behaviour is the call: senders, publishers, gateways. For anything that returns a value, assert on the result instead.