AIToday

AI agents struggle with web pages—new open-source browser cuts token costs 68× to 400×

Hacker News1d ago

Key takeaway

A developer created an open-source stealth browser that plugs into Claude Code and other AI tools, drastically reducing the token cost of fetching web pages—from tens of thousands to a few hundred—while bypassing JavaScript rendering failures and anti-bot blocks. This means AI agents can research the web faster and cheaper, addressing a major inefficiency in how agents currently gather information.

Summaries like this, in your inbox every morning.

Sign up free →

3 Key Points

  • What happened

    A developer built an open-source stealth browser (a recompiled Chromium) that runs as a plugin for Claude Code, Cursor, and Claude Desktop. It fetches web pages and returns cleaned-up text at a fraction of the token cost: JavaScript-rendered pages that originally consumed 68,240 tokens now cost around 285 tokens, and pages returning errors now load successfully.

  • Why it matters

    AI agents researching on the web currently waste enormous context (token budget) on raw HTML, and many fail entirely on JavaScript-heavy sites or pages with anti-bot protection. This tool lets agents retrieve information more efficiently and access previously blocked sites, making AI research and quality assurance faster and cheaper for developers who deploy agents.

  • What to watch

    The tool is still in active development—it lacks residential IP egress and cannot defeat Kasada-style anti-scraping walls. The open-source code and reproducible benchmark are available on GitHub at tiliondev/fortress/tree/main/mcp.

Context & Analysis

AI agents that conduct web research face a severe efficiency problem: fetching raw HTML to pass into Claude's context window burns enormous token budgets. A Wikipedia article in raw form costs 68,240 tokens—an amount that consumes a large fraction of many agents' working memory. Worse, standard web-fetch methods fail on JavaScript-rendered pages and sites with anti-bot detection, forcing agents either to return no data or to fall back on training data or stale cache, introducing hallucination risk.

The new stealth browser solves both problems by running a headless Chromium instance (disguised to avoid detection) and cleaning the returned HTML before passing it to the agent. The compression is dramatic—JavaScript pages drop from tens of thousands of tokens to hundreds—because the tool strips boilerplate, ads, and rendering overhead. This makes it practical for agents to browse the live web repeatedly within a single task, and it restores access to pages that were previously inaccessible.

The tool remains experimental. It cannot yet egress through residential proxies (a feature useful for evading IP-based blocks) and will not defeat sophisticated anti-scraping systems like Kasada. But for the majority of pages agents encounter in research and QA workflows, it offers a substantial speed and cost improvement, and it is immediately available as open-source.

FAQ

How much does this cut token costs compared to the original approach?
An average Wikipedia article that costs 68,240 tokens raw now summarizes to about 950 tokens via Claude Code's built-in fetch, or 285 tokens with the new stealth browser on JavaScript-rendered sites. Nike's homepage originally cost 353,000 tokens raw; the new browser returns it in about 700 tokens instead of a 403 error.
Where can I access this tool?
The open-source code and reproducible benchmark are available at tiliondev/fortress/tree/main/mcp on GitHub. The author is available in the thread for feedback.
What are the current limitations?
There is no residential IP egress yet, and it will not beat Kasada-style anti-scraping walls. It is designed for agents, QA, and research use cases.

Discussion

No comments yet. Be the first to share your thoughts!

Log in to join the discussion

Related Articles

Stay ahead with AI news

Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.

Get Started Free

Free · takes 30 seconds · unsubscribe anytime

1 minute a day. The AI essentials.

200+ sources · Email / LINE / Slack

Get it free →