Shadow Web is an open-source Python compression layer that strips unnecessary HTML before AI agents read a page, cutting token usage by 64–97% on real websites. It flattens shadow DOM trees, groups interactive elements semantically, and extracts tables and forms as structured JSON—all without requiring cloud services. For teams building browser-based AI agents or Playwright scrapers, this significantly reduces LLM API costs while making agent actions more reliable.
Summaries like this, in your inbox every morning.
Sign up free →What happened
A new open-source Python package called Shadow Web compresses HTML pages before sending them to AI agents, reducing token costs by 64–97% on real websites. A Wikipedia page drops from 99,343 tokens to 16,462 tokens (−83%), and GitHub Trending falls from 167,875 to 37,833 tokens (−77%).
Why it matters
AI agents often waste tokens on unused scripts, styles, and hidden DOM elements. Shadow Web keeps only interactive elements, semantic labels, and clean data structures—letting AI developers lower API costs and improve agent reliability without cloud infrastructure.
What to watch
The tool is available now as a pip-installable package (shadow-web) and includes built-in support for Cursor and Claude via a 22-tool MCP server; it also parses tables, forms, and lists into JSON or CSV, and works offline.
Web pages contain far more HTML than AI agents need to see: inline stylesheets, JavaScript bundles, tracking pixels, and dynamically-rendered shadow DOM elements buried in Web Components. Most of this noise becomes tokens sent to an LLM, inflating API bills without adding semantic value. Shadow Web sits between the browser and the LLM, reading the live DOM (including shadow trees), stripping decorative markup, and organizing what remains into a sparse Action Map—a list of clickable elements with labels and semantic groups (like "Login", "Cart", "Navigation"). This cuts raw token load by 64–97% depending on the page.
The tool also solves a secondary pain point: Playwright and other browser automation libraries struggle to reach elements inside shadow DOM or closed Web Components, and selectors break when the DOM changes. Shadow Web provides self-healing selectors that first try fuzzy local recovery, then fall back to LLM verification if needed—no cloud roundtrip required for most failures. For teams building AI agents that navigate the web, this combination of compression, structural parsing (SchemaSnap), and local selector repair reduces both API costs and error rates.
The package is pip-installable and includes turnkey integrations for Cursor and Claude via an MCP (Model Context Protocol) server. It works offline and produces no side effects on the original page, making it safe to use with any framework (React, Vue, Svelte, etc.). Benchmarks on real sites show token reductions of 6× on Wikipedia and 4.4× on GitHub Trending.
No comments yet. Be the first to share your thoughts!
Log in to join the discussion
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
