AIToday
Large Language ModelsAI Coding AssistantsZenn AI/MLPublished: Nov 22, 2026, 16:00 JST

GPT-6 Luna logs zero Godot 3 errors in 150-answer test

Send on LINE
GPT-6 Luna logs zero Godot 3 errors in 150-answer test

3 Key Points

  1. What happened

    Ziva.sh tested GPT-6 Luna on 10 beginner Godot prompts, with 'Godot 4' specified, no version, and web search; across 150 responses, Godot 3 API errors were 0, with all code checked in Godot 4.7.2.

  2. Why it matters

    The complaints traced to older models may no longer describe current behavior, so developers should verify AI-written code rather than assume it is outdated.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The version-mixing complaint has a documented history. In October 2024, a Godot forum user posted ChatGPT-written code using onready var and the old self.connect("input_event", self, "_on_input_event") form, adding the next day that 'Even ChatGPT4.o failed.' In September 2025, forum replies described models as confusing Godot 4 and Godot 3 functions, and in September 2026 a user said AI answers were 'mixes from the syntax of Godot3 and Godot4.' Only the first post named a model, GPT-4o, so the newer complaints cannot be tied to specific versions from the available evidence.

The test design matters for interpreting the zero result. Ziva.sh wrote 10 beginner prompts based on the forum threads and ran each three times per configuration in regular chat, without a project or system prompt. An AI agent read every code block and a script scanned them against a Godot 3 checklist built from the official upgrade guide and Godot 4.7.2's own rename table. All scripts also ran through Godot 4.7.2's --check-only. When no version was given and no search was used, 18 of 30 responses said they targeted Godot 4 and none targeted Godot 3.

The finding comes with stated limits. Zero out of 150 suggests a rate below roughly 2% at 95% confidence if responses were independent, but responses to the same prompt are not independent, so the real uncertainty is larger, the author notes. Only one model and one day were tested, with calls routed through OpenAI's Codex endpoint rather than the ChatGPT app, so the app may behave differently. The single most surprising result is that the Godot 3 problem appears absent from current-model responses, while a separate class of subtler errors, including runtime-only bugs that pass parsing, takes its place.

FAQ
Does enabling web search make ChatGPT write outdated Godot 3 code?
In the test, no. GPT-6 Luna with web search produced 0 Godot 3 API errors across 60 responses, and its searches were all limited to Godot's documentation site.
What kinds of errors did GPT-6 Luna make instead?
Of 150 regular chat responses, 4 had parse errors and 3 had bugs that only appear at runtime, such as setting monitoring to false inside an Area2D signal handler.
How can I check whether AI-written GDScript is valid Godot 4 code?
Run godot --headless --path /path/to/project --check-only --script res://player.gd, then also run the scene to catch Godot 3 calls that pass parsing but fail at runtime, such as instance() or change_scene().
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleASIO's Mike Burgess flags AI data centre foreign-interference risk