AI results can shift; log prompt_version and test data

The article frames this as a workflow discipline problem rather than a model-quality problem. AI tools are not fixed at the moment of adoption: model updates, configuration changes, and prompt revisions can all alter the output for identical input. For casual personal use, noticing the change by feel may be enough, but once AI sits inside a business process, the question shifts from "did it change?" to "when did it change, and what did we change?"

The proposed answer is to keep an evaluation log that pairs each output with its execution conditions. The listed fields include timestamp, tool_name, model_name, prompt_version, input_id, output, latency, and status. Prompts themselves get versions — prompt-v1, prompt-v2, prompt-v3 — and each version's modification is noted, for example adding a "split the answer into 3 items" instruction in v2, or switching to JSON output in v3. Comparing against a fixed set of about 10 test inputs turns this into a lightweight regression test.

The same log can serve a second purpose: evaluating a switch to a different service altogether. The article's Tool A and Tool B comparison shows that faster processing time does not automatically mean a better result. What the outcome ultimately hinges on is whether changes are isolated — altering model and prompt at once makes it hard to know which one moved the result — and whether measurable items such as JSON success rate are tracked consistently over time.