AIToday
Large Language ModelsZenn AI/MLPublished: Oct 15, 2026, 10:00 JST

Jev beats llama3:8b 18/20 to 15/20 on store-diagnosis test

Jev beats llama3:8b 18/20 to 15/20 on store-diagnosis test

3 Key Points

  1. What happened

    In a 20-case dummy-restaurant test built by one writer, TypeSafe AI's Jev picked the right cause for 18 stores versus 15 for Ollama's llama3:8b, and gave the same answer on all three tries for 20 of 20 cases.

  2. Why it matters

    Jev's confidence could be used directly — high-confidence calls held up and low-confidence ones did not — so a business could auto-accept the confident verdicts and send only the doubtful ones to an LLM or a person.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Context & Analysis

Jev is a different kind of model from a chatbot: TypeSafe AI's "System One" model, released on September 15, 2026, does not generate text at all. You hand it an app's state (text or JSON) and a typed question, and it returns an answer drawn only from a predefined set of choices, plus probabilities and a confidence value. The question types are Choice, Score, and Noul. Because the answer must come from the defined options, it cannot invent a nonexistent answer — though it can still pick a wrong one.

The author's step-by-step experiment shows why input design moved the result more than the model did. When a prior-month rainy-day count was missing, Jev leaned toward weather at 0.65 and menu at 0.29; adding that one line pushed weather to 0.99 — but it also crushed the menu cause to 0.01. Because Choice asks for a single main cause, a second cause gets starved of probability. Asking about each cause separately with Noul brought the menu cause back up to 0.41. In a later round, the menu score fell again until the author added a menu_changed_this_month flag; the phrase "a major factor" versus "one of the factors" barely moved the number. The underlying problem was that the data recorded a drop in spend per customer but never stated that the menu had changed — the model could not infer a cause its input did not support. One acknowledged limitation: the change-rate inputs and the rewritten question were altered in the same round, so their individual effects are not separated.

FAQ
How much does Jev cost to use?
Jev's pricing is $0.042 per 1M input tokens and output is free, as of Jev 1.13. In the author's 60-call test the total cost was $0.0017.
Can I run Jev on my own hardware?
No. The weights are not public, and Jev is available only through its API. The console requires a minimum $5 charge, and purchased credits expire after 12 months.
Which cause did both Jev and llama3:8b struggle with?
Competitor stores. Jev got 3 of 5 and llama3:8b got none, because the data never states that a competitor opened nearby — it has to be inferred from a fall in new customers.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleWordCraft, a Rust-built open-source Word alternative, hits v0.6.1