Concept XAI: backprop as explanation tool, not just learning

The prototype builds on the idea of concept bottleneck models (CBM), which insert a human-understandable concept layer between input and output. Concept XAI keeps the 2D layout of concepts as a spatial map rather than just presence or absence, and adds a direct audit step that writes out concept contributions as JSON.

The author reports two observations from testing. First, converting images to geometric concepts before CNN processing changes how the model sees the image, making it rely on local parts like strokes. Second, for recognizing the digit "1", the model's decision was based on combinations of vertical bar strokes, confirming a part-based reasoning process.

However, the author also notes a limitation: if the goal is only to detect geometric primitives, classical image processing methods can be more accurate and stable. The bigger challenge is scalability—humans defining every concept does not work for complex domains, so future work points toward AI autonomously acquiring interpretable concepts.