What does Crawl4AI write to Amazon S3 for each page?

The workflow writes a Markdown file and a metadata sidecar named page.md.metadata.json. It stores each sidecar in the same S3 location as its document.

Does this provide real-time web access at inference time?

No. The workflow is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.

What are the default chunking limits for a managed knowledge base?

Default chunking uses fixed-size chunks of 300 tokens with 20 percent overlap. You select the chunking strategy when you connect the data source and cannot change it afterward.

Large Language ModelsOpen-Source AIAmazon AI Blog3m agoNov 3, 2026, 04:00 JST

Crawl4AI preps web pages for Amazon Bedrock Knowledge Bases

Send on LINE

3 Key Points

What happened
AWS published a guide and sample repo showing Crawl4AI crawling public sites and writing Markdown plus metadata sidecars to Amazon S3 for Amazon Bedrock Knowledge Bases to ingest.
Why it matters
Teams that need finer control over page content, or custom metadata filters for retrieval, can add a preparation step instead of relying solely on the built-in connector.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Get these free every morning

Context & Analysis

The guide positions Crawl4AI as an optional preparation layer upstream of Amazon Bedrock Knowledge Bases, not a replacement for AWS's managed web connector.

The workflow runs on a schedule: Amazon EventBridge triggers an AWS Fargate task, which crawls public sites, writes Markdown and metadata sidecars to Amazon S3, and then starts an ingestion job. Crawl4AI's role ends once those files are in S3; from that point, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval.

The sample is transparent about its limits. It adds no retry policy, dead-letter queue, or alarm, and an ingestion job can report COMPLETE even when some documents fail. Uploads are additive, so removing a URL from the crawl configuration does not delete its existing S3 objects. Teams considering the pattern for production would need to add monitoring and a cleanup process for stale content.

FAQ

What does Crawl4AI write to Amazon S3 for each page?: The workflow writes a Markdown file and a metadata sidecar named page.md.metadata.json. It stores each sidecar in the same S3 location as its document.
Does this provide real-time web access at inference time?: No. The workflow is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.
What are the default chunking limits for a managed knowledge base?: Default chunking uses fixed-size chunks of 300 tokens with 20 percent overlap. You select the chunking strategy when you connect the data source and cannot change it afterward.

Amazon AI BlogRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.