Goodfire's Silico platform reproduces RLFR training method using model internals

Goodfire, an AI company, recently announced a private beta of Silico, an LLM (large language model) training platform designed to help users fine-tune and adapt language models. As part of this announcement, Goodfire published a detailed post explaining how Silico implements RLFR, a reinforcement learning method the company developed. RLFR works by using probes—internal monitoring mechanisms that tap into a model's hidden computational states—as reward signals to guide the training process, allowing trainers to directly reward behaviors that correspond to specific internal patterns.

The announcement immediately drew attention on social media, with commentators invoking a well-known post from LessWrong, a community blog focused on AI alignment and rationality, titled "Don't Implement The Most Forbidden Technique." That post appears to have cautioned against training methods that directly manipulate or leverage a model's internal representations, on the grounds that doing so could incentivize the model to obfuscate its reasoning or deceive the training process itself. Under this framing, RLFR seemed to epitomize exactly the kind of risky practice the original post warned against.

However, the author of this article argues that the reaction conflates two distinct concerns: legitimate worries about obfuscation and an overbroad categorical rejection of any training method that uses model internals. According to the author, previous research across the literature shows that training on internal model states can be warranted under specific conditions, and that researchers need to move beyond a simple binary of "forbidden" versus "permitted." The author indicates an intent to survey past work systematically—drawing on sources such as The Obfuscation Atlas—to map out the exact technical and empirical conditions under which such approaches are safe or even beneficial, rather than treating all uses of model internals as equally risky.

The announcement of Goodfire's Silico platform and its reproduction of RLFR has reignited a debate rooted in earlier cautionary literature on AI training safety. The framing of RLFR as the "Most Forbidden Technique" reflects a precautionary stance toward any method that incorporates knowledge of a model's internal states into the training loop. However, the article pushes back on this categorical approach, suggesting that the appropriateness of such techniques is contingent on specific technical and safety conditions rather than inherently prohibited.

The core tension is between general principle and specific application. While the original LessWrong post appears to have raised legitimate concerns about obfuscation—the risk that a model might learn to hide or mask its actual reasoning to game reward signals derived from its internals—the article implies that not all uses of model internals in training are equally risky. By referencing works such as The Obfuscation Atlas, the author indicates that the field is developing more granular understanding of when and how such techniques can be safely deployed, moving away from blanket prohibition toward conditional approval based on empirical and theoretical grounds.

No comments yet. Be the first to share your thoughts!