AIToday
Large Language ModelsOpen-Source AIZenn AI/MLPublished: Nov 20, 2026, 16:00 JST

Jane Street's 102-line socratic-tutor skill fails on small local models

Send on LINE
Jane Street's 102-line socratic-tutor skill fails on small local models

3 Key Points

  1. What happened

    Jane Street published socratic-tutor, a 102-line text file in its agent_skills repo, on 2026年11月3日; a test on Gemma-3-1b-it, Qwen3.5-0.8B and Qwen2.5-1.5B-Instruct found none ran its diagnostic test, skill map or one-question flow.

  2. Why it matters

    The skill is only an instruction sheet, so its success depends on the instruction-following ability of the model it is pasted into, and 1B–1.5B local models appear too small to run it.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

Jane Street published the agent_skills repository on 2026年11月3日, and socratic-tutor was its first skill. Interest built quickly: the repo hit the Hacker News front page on 2026年11月10日 with 697 points, and GitHub stars climbed from 8,273 on 2026年11月14日 to 8,434 the next day and 9,618 by 2026年11月17日. Several English explainer articles appeared between 2026年11月14日 and 16, but the tester found almost no Japanese-language hands-on guides.

The design assumes a strong instruction-follower. The skill splits a topic into 4–10 sub-skills scored 0–5, runs a 3–5 question diagnostic one question at a time, withholds answers until two or three failed attempts, and schedules spaced review and a final exam. The test environment was deliberately weak — a shared Linux VM with about 1GB of free memory and no GPU — and response times ranged from 6.5 to 130.2 seconds. Gemma ignored the instructions or echoed the prompt text, Qwen3.5-0.8B repeated the same diagnostic wording until it hit the 1,024-token limit, and Qwen2.5-1.5B-Instruct asked for an equation and then offered an answer, contrary to the skill's rules.

The tester notes that Jane Street has not published any evaluation, so whether Socratic tutoring actually improves learning remains a design intention, and Hacker News commenters debated whether the skill is just a prompt. The skill itself says not to use it for one-off "explain this" questions. The body also leaves open which model size the skill starts working at, and how it behaves in long Japanese sessions on commercial chatbots — the setup it was designed for — was not tested.

FAQ
What exactly is socratic-tutor — an app or a model?
Neither. It is a single 102-line text file, MIT-licensed with no account or install, placed in an agent's skill folder or pasted into a chatbot's system prompt. The AI model that runs it must be supplied separately.
Which models did the test use, and did any work?
The test used Gemma-3-1b-it, Qwen3.5-0.8B and Qwen2.5-1.5B-Instruct on a shared Linux VM with 8GB of memory and no GPU. None executed the diagnostic test, skill map or one-question-at-a-time flow.
Does the skill cost anything or require registration?
No. It is free, MIT-licensed, and needs no account or installation, but the underlying AI model is separate and chosen by the user.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleMeta's Muse Charm AI device tipped at $300 by The Information