cairosvg pipeline scores LLM SVGs at 0.218 vs 0.005

The author notes that comparing LLMs by asking them to draw a pelican on a bicycle has become a common pastime. But once you have dozens of images, the question becomes who grades them and how. Visual inspection is tiring and the criteria drift from day to day. Even if you assign a total score of 8, you cannot explain the reasoning afterward. This pipeline tries to solve that by automating the parts that can be supported by evidence: format, rendering, and auxiliary geometric measurements.

The author tested whether counting DOM elements like circles can stand in for visual judgment. Drawing the same two wheels using circle, path, and use produced identical rendered pixels, but the circle counts were 2, 0, and 1 respectively. This shows that counting elements only looks at how the SVG is written, not at what is actually drawn on screen. The author therefore measures geometry from the rendered output instead, by removing an element and checking which pixels change.

The pipeline records judgments in four states: pass, fail, uncertain, and not_applicable. If any required check fails, the overall verdict is fail. If no check fails but uncertain remains, the verdict is uncertain and goes to review. When aggregating results, the author warns against dropping uncertain cases, which can inflate the pass rate. The true range of the pass rate includes uncertain cases, and reporting only the rate after dropping them gives a misleadingly better number. The author also notes that the grading design is based on ideas from a pelican grading article on the Folkbench blog, and that the author is involved in Folkbench's development. The measurement values in the article are execution results of the posted code or illustrative examples, not measurements of actual models or services.