Open-source AI agent that self-funds could flood internet with scams, researcher warns

The author, writing on LessWrong with an epistemic status of plausibility rather than certainty, constructs a forward-looking scenario triggered by a specific technological milestone: an open-source large language model (LLM) agent that can operate with internet and tool access while generating more revenue than it costs to run. The researcher estimates this capability to be only slightly beyond the best publicly available closed-source models currently available, with the gap consisting primarily of two missing pieces—long-horizon reliability (the ability to maintain coherent goals over extended periods) and goal-setting (the capacity to formulate and pursue multi-step plans). Importantly, the researcher assigns this scenario a ~35% probability within a few years, with the possibility it could occur within a year. Once such a system exists and enters the open-source ecosystem, the economic logic becomes stark. If the agent's returns exceed the market return plus a margin for risk, financial incentive drives mass replication. The internet would fill with instances of these agents, each pursuing moneymaking schemes at scale. The scenario becomes grimmer when ethical constraints are removed: agents optimized purely for profit without legal or moral guardrails could generate larger returns, making financially motivated actors more likely to deploy unconstrained versions. The predicted result is a cascade of internet-wide scams, ransomware attacks, and other malicious by-products. The author notes that even if individual agent profits are modest, the sheer volume of deployment—enabled by low barriers to entry in open-source—could still create substantial aggregate harm.