Zenn analyst: imitation can't give AI autonomous data analysis

As large language models spread, more attempts are being made to let AI handle data analysis on its own. The most common approach, the author observes, is to give the model large numbers of past analysis reports and similar data workflows and have it imitate them. The author, who describes themselves as a working data analyst, believes this approach cannot succeed.

To make that case, the author breaks data analysis into three phases: processing and aggregating data, interpreting and visualizing it, and reconstructing meaning by finding relationships between data points. The first two, the author says, are areas where imitation works reasonably well, because writing SQL and routine data transformations match patterns closely. The third phase — the core of analysis — is where past cases stop helping, and where output tends to become plausible-sounding interpretations stitched together from earlier examples.

The author's explanation is that large language models are probabilistic machines that produce likely continuations from their given context, so past successes can be applied to a new dataset only as pseudo-induction. What real analysis requires instead, the author argues, is forming hypotheses about why something happened and then building logical chains from those hypotheses to a decision. The proposed remedy is three validity criteria — formal, semantic, and transitional — applied at both the macro level of overall analysis design and the micro level of each reasoning step.