AIToday
Large Language ModelsOpen-Source AIZenn AI/MLPublished: Nov 4, 2026, 06:00 JST

DeepEval 4.3.2 alone won't fix LLM evals, argues OSS-review article

Send on LINE
DeepEval 4.3.2 alone won't fix LLM evals, argues OSS-review article

3 Key Points

  1. What happened

    A Zenn write-up reviewing deepeval 4.3.2 (Apache-2.0, roughly 19,300 GitHub stars as of Nov 2026) says the tool only makes evaluation runnable, not sustainable, and that the Gold Dataset must be versioned in Git by the company itself.

  2. Why it matters

    The author splits evaluation into a framework layer that improves with upgrades and a dataset layer that decays if left alone, arguing companies that skip the second layer keep re-hitting the same bugs whenever they switch models.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

A familiar complaint is the starting point: most reports of "we adopted evals" mean only that a framework was pip-installed. After the first month, the 20 test cases written at the start go unupdated, new production failure patterns never make it into tests, and nobody can say whether a model swap is being measured against the same standard. The author separates the framework, the Gold Dataset and the pass/fail thresholds into three layers, and notes that upgrading DeepEval only improves the first.

Much of the piece is hands-on detail from actually running version 4.3.2. PatternMatchMetric turns out to use fullmatch rather than partial matching, so detecting banned words requires a negative lookahead in the pattern. JsonCorrectnessMetric scores deterministically but still needs a key to initialize, which complicates the simple split between metrics that call an LLM and those that do not. Data-wise, the recommendation is to keep CSV or JSONL in Git, record provenance in additional_metadata, and treat each case as done only when it has an incident ID, an approved expected output and no near-duplicates.

For LLM judges, the author asks for three controls: measure the judge against human labels with Cohen's κ, put deterministic metrics first, and check variance by running the same tests multiple times. The suggested cutoff is a κ below 0.6 meaning the judge prompt or model should be revisited. The stated limit is that a good framework is no substitute for a good dataset — before choosing tools, a company needs to decide who will keep its answer key.

FAQ
Does DeepEval's built-in dataset feature replace an in-house dataset?
Not fully. DeepEval's push and pull go through Confident AI's managed API, so companies that need to keep data history in-house should base their Gold Dataset on Git-managed CSV or JSONL files. Local JSONL datasets can be loaded and run without push or pull.
Which metrics can run without an API key?
ExactMatchMetric, PatternMatchMetric and AgentLoopDetectionMetric need no LLM client at initialization. JsonCorrectnessMetric and ToolCorrectnessMetric are deterministic in scoring but still require a key at initialization, and GEval uses an LLM judge.
What is the suggested size for each dataset layer?
20–50 cases are suggested for the Core layer and 5–10 new Production cases per week, with Regression cases accumulating and never being reduced.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleDrax and Brookfield stake £20 billion on 1GW UK AI data center