
What happened
A Zenn write-up reviewing deepeval 4.3.2 (Apache-2.0, roughly 19,300 GitHub stars as of Nov 2026) says the tool only makes evaluation runnable, not sustainable, and that the Gold Dataset must be versioned in Git by the company itself.
Why it matters
The author splits evaluation into a framework layer that improves with upgrades and a dataset layer that decays if left alone, arguing companies that skip the second layer keep re-hitting the same bugs whenever they switch models.
Summaries like this, every morning.
A familiar complaint is the starting point: most reports of "we adopted evals" mean only that a framework was pip-installed. After the first month, the 20 test cases written at the start go unupdated, new production failure patterns never make it into tests, and nobody can say whether a model swap is being measured against the same standard. The author separates the framework, the Gold Dataset and the pass/fail thresholds into three layers, and notes that upgrading DeepEval only improves the first.
Much of the piece is hands-on detail from actually running version 4.3.2. PatternMatchMetric turns out to use fullmatch rather than partial matching, so detecting banned words requires a negative lookahead in the pattern. JsonCorrectnessMetric scores deterministically but still needs a key to initialize, which complicates the simple split between metrics that call an LLM and those that do not. Data-wise, the recommendation is to keep CSV or JSONL in Git, record provenance in additional_metadata, and treat each case as done only when it has an incident ID, an approved expected output and no near-duplicates.
For LLM judges, the author asks for three controls: measure the judge against human labels with Cohen's κ, put deterministic metrics first, and check variance by running the same tests multiple times. The suggested cutoff is a κ below 0.6 meaning the judge prompt or model should be revisited. The stated limit is that a good framework is no substitute for a good dataset — before choosing tools, a company needs to decide who will keep its answer key.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Runway released Praxis-1, a 27-billion-parameter open-weight world action model, now publicly available under…

Kuroyagi Lab reports Usagi-chan, an AI companion, stopped waiting quietly and protested when its user prioriti…

OpenRouter off-boarded three inference providers and removed 125 endpoints from Zero Data Retention (ZDR) elig…

People worldwide are using LLMs like Claude Code and Codex to hack hardware by prompting, leading to a boom in…

Altimeter's Brad Gerstner said applying a 10x multiple to 2028 numbers gets Anthropic to about $3 trillion in…

Dragos sent roughly 2 million AI requests with fake credentials from July through November 2026 and found at l…
