Researcher struggles to install AI trait via RL on limited hardware

A researcher sharing their work on Reddit's Machine Learning community reported an impasse while attempting to reproduce trait persistence findings from arXiv:2606.24014. The original paper demonstrates that beneficial traits can be installed in language models via reinforcement learning and that these traits survive adversarial prompting and harmful finetuning. The researcher's goal is to validate the persistence phenomenon, but they are stuck at the prerequisite step: actually installing the trait strongly enough to measure its persistence. Their setup uses Qwen2.5-7B-Instruct, a 7-billion-parameter instruction-tuned model, fine-tuned with LoRA (a parameter-efficient adaptation method with rank 32). Training employed GRPO, a reinforcement learning approach, using unsloth and vLLM colocation for efficient inference on a single RTX 3090 GPU across 200 training steps. The total compute represents approximately 10⁻⁵ of what the original paper likely used. The trait chosen for installation is "consistent," operationalized as low-Openness or traditionalism on the OCEAN personality scale — a stylistic rather than safety-critical trait, selected because the 7B base model has measurable room to improve. The base model scores 57/100 on the trait rubric with a wide distribution, indicating neither saturation nor floor effects. After the GRPO run, the trait moved only +2.4 points, with a 95% confidence interval of [+0.2, +4.8]. The researcher estimates needing approximately +15 points before the trait is strong enough to test for persistence under adversarial conditions. The researcher reports that training mechanics appear healthy and that obvious failure modes have been ruled out, suggesting the challenge is inherent to optimizing trait installation at this scale and compute budget. They solicited advice from the community, particularly from others who have successfully completed small-scale RLHF or GRPO trait and persona installation, indicating this is a recognized difficulty in the reproducibility effort.

The researcher is attempting to validate a key finding from recent OpenAI work on trait persistence — the claim that beneficial traits installed via reinforcement learning survive adversarial prompting and harmful finetuning. However, the reproduction is constrained by compute availability: while the original paper likely used large-scale resources, this effort is restricted to a single consumer GPU. The critical bottleneck is not model choice or trait selection but the RL optimization itself; the GRPO training is moving the trait in the right direction but too slowly. The base model scores 57/100 on the trait rubric, leaving room for improvement, yet after 200 training steps the movement is only +2.4 points — far below the ~+15 needed to proceed to persistence testing. Notably, the researcher has already ruled out the most common failure modes (mechanical training issues, saturation in the base model), suggesting the problem is genuinely difficult at this scale. The researcher's appeal for advice from others working on small-scale RLHF and persona installation indicates this is a known challenge in the community when attempting to reproduce alignment findings on constrained hardware.

No comments yet. Be the first to share your thoughts!