AIToday
Large Language ModelsAI Coding AssistantsZenn AI/MLPublished: Oct 8, 2026, 10:00 JST

Codex reads screen recordings via Snipping Tool stills

Codex reads screen recordings via Snipping Tool stills

3 Key Points

  1. What happened

    Using a roughly 45-second Snipping Tool recording of an article draft, the writer fed the MP4 to Codex, which split it into images and returned a summary; the tool chain was FFmpeg/ffprobe, Python, OpenCV and Pillow.

  2. Why it matters

    Codex cannot read video directly but can recognize images, so video-to-stills conversion appears to lower the effort of describing on-screen clicks and menus to an AI agent.

WHO IT HITSThis mainly lands on non-engineers who want to hand off repetitive screen tasks to an AI agent — office staff and writers who currently explain 'click here, open this menu' in chat — though the demo covers summarization only.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Context & Analysis

Behind the demo is a simple workaround: Codex has no native video understanding, but it does recognize images. So the author inserted a conversion step between video and model, cutting the recording into stills with OpenCV at set intervals and assembling them with Pillow, while FFmpeg/ffprobe handled length, resolution and audio checks. In the walkthrough, the writer recorded their own screen while drafting this very article with Snipping Tool, saved it as MP4, and asked Codex to describe the clip; the model produced a timeline noting the intro text and later edits under a 'Codex' heading. The recording only covers the note-taking, not a moment where Codex answers on camera.

The author used GPT-6.1 Sol and frames the technique as a first step, suggesting the same captured content could later support requests to build automation scripts or generate procedure manuals. That remains a suggestion rather than something demonstrated in the article.

FAQ
How is video fed to Codex if it can't read video?
The article converts video to still images, then passes those images to Codex in a video → still image → Codex flow.
What tools are used to split the video into images?
FFmpeg/ffprobe for length, resolution and audio; Python for extraction and image lists; OpenCV for cutting stills at intervals; and Pillow for listing the cut images.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleAI cluster networking reading list targets GPU engineers