Why can't AI models just use data they were trained on?

Traditional model training relies on snapshots of information collected at a particular point in time, but today's organizations operate in environments where prices, inventory, markets, security threats, and customer behavior change continuously. Training AI on static data is no longer sufficient to track these fluctuations or provide current information that users can trust.

What are the main technical challenges in retrieving web data for AI at scale?

The infrastructure must navigate hundreds of millions of existing web domains and billions of new URLs created each week, delivering real-time information while overcoming technical barriers such as JavaScript-heavy websites and aggressive antibot software. It must also integrate fragmented sources—public web retrieval, APIs, licensed datasets, and proprietary internal data—into a timely and usable knowledge layer while enforcing strict compliance protocols aligned with global privacy frameworks like GDPR and CCPA.

What does this mean for enterprises building AI systems?

When this becomes critical infrastructure for a company, doing it in-house becomes a full-time engineering problem that competes with the actual AI work, leading many organizations to seek specialized platforms designed specifically for data retrieval, orchestration, and observability.

MIT Technology Review AIJun 24, 2026Jun 24, 2026, 22:00 JST

Real-time web data becomes critical AI infrastructure

Send on LINE

3 Key Points

What happened
Organizations deploying AI are discovering that static training data is no longer sufficient; they need continuous access to fresh, structured web information to track competitor pricing, market trends, and consumer sentiment in real time. A new infrastructure layer is emerging to enable AI models to discover and retrieve this data at scale without being blocked by website restrictions.
Why it matters
According to Gartner, 60% of AI projects not supported by AI-ready data will be abandoned by end of year. Real-time data access reduces AI hallucinations and builds user trust—a survey found that 56% of AI practitioners said businesses need access to real-time web data to improve trust in AI outputs. For businesses, stale data leads to bad decisions and disappointed consumers; in operational settings, delayed retrieval makes even sophisticated models unreliable.
What to watch
Research indicates that 97% of AI organizations depend on real-time web data infrastructure, but 90% feel boxed in by restrictions. The emerging solution mimics human browsing behavior at scale—performing operations 80 billion times a day across millions of websites—while enforcing compliance with privacy frameworks like GDPR and CCPA. This type of specialized infrastructure is becoming necessary because handling it in-house competes with actual AI development work.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

FAQ

Why can't AI models just use data they were trained on?: Traditional model training relies on snapshots of information collected at a particular point in time, but today's organizations operate in environments where prices, inventory, markets, security threats, and customer behavior change continuously. Training AI on static data is no longer sufficient to track these fluctuations or provide current information that users can trust.
What are the main technical challenges in retrieving web data for AI at scale?: The infrastructure must navigate hundreds of millions of existing web domains and billions of new URLs created each week, delivering real-time information while overcoming technical barriers such as JavaScript-heavy websites and aggressive antibot software. It must also integrate fragmented sources—public web retrieval, APIs, licensed datasets, and proprietary internal data—into a timely and usable knowledge layer while enforcing strict compliance protocols aligned with global privacy frameworks like GDPR and CCPA.
What does this mean for enterprises building AI systems?: When this becomes critical infrastructure for a company, doing it in-house becomes a full-time engineering problem that competes with the actual AI work, leading many organizations to seek specialized platforms designed specifically for data retrieval, orchestration, and observability.

MIT Technology Review AIRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Next articleQualcomm to buy Modular for nearly $4 billion