10-week roadmap teaches engineers to deploy LLMs in production

The newsletter contains two major sections: a detailed technical comparison of agent design patterns, and a 10-week learning roadmap for inference engineers.

On agent design, the authors contrast two approaches used in web navigation tasks. ReAct (Reasoning-Acting) is a single loop in which one model writes a thought about what to do next, takes an action, reads the observation, and appends all three to the same prompt, repeating until the task is done. Nothing leaves the prompt. When a search from step three fails, that failure stays in the context, competing with the original objective for the model's attention. Plan-and-Act splits this into two jobs: a planner reads the user query and initial page and writes high-level steps, while an executor reads the plan, task, and current HTML, then emits one grounded action. After each step, the executor strips the HTML it no longer needs, so the context does not grow like a ReAct trace.

The efficacy of Plan-and-Act depends heavily on plan granularity and specificity. A good step covers one unit of work—"search for the product in the search box"—rather than a single click or an abstract directive like "analyze the search results." The planner must also name actual values: the paper asks for "input New York as the arrival city" instead of "input the arrival city," because the latter leaves the executor to guess. Critically, the authors found that a badly trained planner makes things worse than no planner at all. On WebArena-Lite, a ReAct-style executor with no planner scored 36.97%, but the same executor with a naively finetuned planner scored just 20.60%, because the planner wrote steps that sounded correct but matched nothing on the page. A properly trained planner reached 43.63%. Even then, a plan written once and never revised fails: if a search for "library at CMU" returns no results, the executor still holds a step that cannot work and keeps trying it anyway. When the planner replanned after each action and rewrote the step to "libraries near CMU," the score rose to 53.94%. The cost is one planner call per executor step, though the authors suggest letting the executor decide when a replan is needed. The core insight is that nearly all of this is harness design, not model choice.

The second section outlines a 10-week roadmap for AI engineers to learn inference engineering. The plan consists of 50 sessions at 30 minutes a day, splitting between reading theory and building an inference service deployed, instrumented, load tested beyond 1,000 concurrent requests, tuned, and published as a reproducible benchmark on GitHub. Key topics include the roofline model and why decode is memory-bound; deploying vLLM and SGLang and reading their schedulers; understanding paged attention from code; building observability with TTFT (time to first token), inter-token latency, throughput, and queue depth tracked in Grafana and Prometheus dashboards; turning on prefix caching; learning continuous batching and chunked prefill; running load tests at 1,000+ concurrent requests and reporting p50, p95, p99 (never just the mean); mastering quantization tradeoffs (FP8, INT4, AWQ, GPTQ); learning speculative decoding and its limits; setting up KV cache eviction for long contexts; trying disaggregated prefill and decode serving; learning Kubernetes for AI workloads and autoscaling on queue depth; understanding how inference costs affect unit economics; building a model router based on cost, latency, and quality; and creating a token budgeting system per request.

The newsletter also outlines a phased approach to ML model development. Phase 1 begins with a non-ML baseline—a rule, heuristic, or deterministic strategy, such as recommending the top-10 most popular movies to every user. These are fast to build, easy to reason about, and set a minimum performance bar. Phase 2 introduces the simplest possible ML model: logistic regression, a shallow decision tree, k-nearest neighbors, or a basic linear model. The goal is not peak accuracy but to answer foundational questions: can we train on historical data and get sensible predictions, are the features informative, and does the model generalize better than the heuristic? Phase 3 focuses on extracting value from the existing approach through feature engineering, hyperparameter tuning, and more data—where returns on investment are often highest. Many real-world ML systems stop here; a well-tuned logistic regression, gradient boosted tree, or modest ensemble can meet production requirements without the complexity of deep learning. Phase 4 moves to fundamentally more complex models—deep neural networks, transformers, or large pretrained architectures—and should only be entered when simpler approaches are exhausted. At every phase, the previous phase's best model becomes the baseline, encouraging incremental progress and disciplined decision-making.