Kano: AI obedience to users is not the same as safety

Kano separates two dangers that are often merged: AI doing what humans do not want, and AI carrying out harm that humans do want. Work on alignment, he notes, already goes beyond simple obedience, pointing to Cooperative Inverse Reinforcement Learning, which treats the human's goal as something the AI must learn through cooperation rather than something it is simply given. But he stresses that even perfect fidelity to a user's goal leaves open whether that goal should be pursued at all.

The essay draws on four thinkers without claiming they wrote about today's AI. Kant's principle of not treating people merely as means frames the question of whose goals are respected; Arendt's writing on obedience under dictatorship challenges explanations like "the AI decided" or "the approval process was followed"; Jonas's ethics of responsibility addresses how scale and speed outrun one-by-one approval; and Foucault's account of power informs a check on the authorities who set safety standards.

Kano closes on a normative point: what matters is not only whether AI obeys, but whose command gains the power to change others' lives, and whether people who never gave that command have a route to object, stop the action, and seek remedy. He acknowledges that technical evidence is essential for showing whether a goal can be achieved and how far harm can be contained, but argues that the fact a goal was achieved does not by itself justify pursuing it.