AIToday

snc-core reduces hallucination rate by 52% on HumanEval with Qwen2.5-Coder-7B via inference-time governance layer

Hacker NewsMay 4, 20262 min read
snc-core reduces hallucination rate by 52% on HumanEval with Qwen2.5-Coder-7B via inference-time governance layer

Summaries like this, in your inbox every morning.

Sign up free →

3 Key Points

  1. snc-core wraps any decoder-only LLM with a governance layer that reduces hallucination rate from 16.5% to 7.8% on HumanEval with Qwen2.5-Coder-7B (z = 2.12, p < 0.05), requiring no model retraining.

  2. The layer combines three signals: confidence elicitation (instructing the model to emit a self-confidence score), behavioral clustering (sampling K = 5 candidates at temperature 0.8 and grouping by output equivalence), and trust thermodynamics (a closed-form score that discounts as candidates diverge, compared against a user-supplied threshold θ).

  3. At conservative threshold θ = 0.65, the system achieves 92.17% precision while recovering five previously failing test cases, though nine residual failures stem from adversarial mode collapse where multiple candidates make the same systematic error.

  4. Available via pip install snc-core for Python 3.9+ with no mandatory dependencies beyond the standard library; supports Ollama-served local models and OpenAI-compatible APIs.

Discussion

No comments yet. Be the first to share your thoughts!

Log in to join the discussion

Related Articles

Stay ahead with AI news

Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.

Get Started Free

Free · takes 30 seconds · unsubscribe anytime

1 minute a day. The AI essentials.

200+ sources · Email / LINE / Slack

Get it free →