Crawl4AI for Amazon Bedrock Knowledge Bases: Prepare Web Pages Before You Index Them

AWS has published a sample that fixes a common problem in retrieval-augmented generation: the web pages you index are full of navigation, cookie banners and scripts, and your retrieval quality suffers. The AWS Machine Learning blog post, published November 2, 2026, shows how to put the open-source Crawl4AI crawler in front of an Amazon Bedrock knowledge base. It writes clean Markdown plus a metadata file for each page into Amazon S3, and the knowledge base ingests both.

This is a how-to guide. It covers when the sample is the right tool, how it works, how to deploy it, and the limits AWS itself lists, which are worth reading before you rely on it.

The short version

Which ingestion path should you pick?

AWS is direct about the order of choices. The Amazon Bedrock Web Crawler connector takes seed URLs or sitemaps you are authorized to crawl, works within the scope and rate limits you set, waits for JavaScript content, and syncs only added, updated and deleted pages after the first full crawl. There is no crawler to build or run, so it is the starting point "for most public website content."

Switch to the Crawl4AI preparation path when public pages need one or more of three things before ingestion:

1. Structured Markdown. The sample uses a default Markdown generator with a pruning content filter to cut navigation and boilerplate while keeping headings, lists and tables. Pruning is heuristic, so review output from your own sites.
2. Bounded, targeted extraction. An optional CSS selector extracts only a page region, and breadth-first traversal is constrained by seed origin, path and host scope, depth, page caps and URL patterns.
3. Custom metadata. Source URLs, crawl timestamps, content type and your own attributes travel with each document, so you can filter retrieval results by them.

Note what this is not: AWS calls it batch ingestion of known web properties, "not real-time web access at inference time such as Web Grounding with Amazon Nova."

How does the workflow run?

The flow has eight numbered steps in AWS's diagram. In plain terms:

1. Amazon EventBridge starts an ECS task on Fargate on a schedule.
2. The task crawls the public sites you configured and prepares Markdown with a sidecar per page.
3. It uploads the files to the S3 bucket behind the knowledge base data source.
4. It starts a Bedrock ingestion job.
5. The knowledge base parses, chunks, embeds and indexes the new and changed objects.
6. Your application queries the knowledge base for chunks or for an answer with citations.

Crawl4AI's job ends once it has written the documents and sidecars. Everything after that is handled by Bedrock. The task reads its non-secret configuration from AWS Systems Manager Parameter Store at startup.

The crawl step

The sample's crawl step, shown in simplified form in the post, runs a headless browser with JavaScript enabled, a breadth-first deep crawl with depth and page limits, a seed-scope filter, a pruning Markdown filter, an optional CSS selector, a robots.txt check, a delay between requests and a semaphore count of one. After redirects, it rechecks that the final URL is still in scope, skips duplicates by canonical URL, and writes the Markdown with a metadata file.

The sidecar format

For page.md, the sidecar must be called page.md.metadata.json, stored in the same S3 location. Attributes use a typed format. An abbreviated example from the post has a string source_url, a number crawled_at_epoch and a string content_category. Bedrock limits each sidecar to 10 KB, and if one would exceed that the run stops with an error before writing either file.

The chunking choice you cannot undo

A managed knowledge base offers default chunking (fixed 300-token chunks with 20 percent overlap), fixed-size chunking with your own token count and overlap, and no chunking. AWS warns that you pick the strategy when you connect the data source and "can't change it afterward," so test options on representative pages first. A customer-managed knowledge base also offers semantic and hierarchical chunking.

How do you deploy it?

AWS provides a companion repository, aws-samples/sample-crawl4ai-bedrock-knowledge-base, with an AWS CDK app. You need a managed knowledge base (or an existing customer-managed one) with an S3 data source in status AVAILABLE, all in one Region, plus Python 3.10 or later, Node.js, AWS CLI v2, CDK v2 and a Docker-compatible engine.

The steps from the post:

git clone https://github.com/aws-samples/sample-crawl4ai-bedrock-knowledge-base.git
cd sample-crawl4ai-bedrock-knowledge-base/deploy
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

export AWS_DEFAULT_REGION=REGION
cdk bootstrap aws://ACCOUNT_ID/REGION

cdk deploy \
  -c knowledgeBaseId=ABCDEFGHIJ \
  -c dataSourceId=KLMNOPQRST \
  -c s3Bucket=amzn-s3-demo-bucket \
  -c 'seedUrls=["https://www.example.com/services"]' \
  -c maxDepth=1 \
  -c maxPages=25

Replace every example value. AWS warns that the example values are syntactically valid, so an unedited command deploys a stack pointing at a bucket you do not own. Start with a depth of one and 25 pages, review the prepared output, then raise the limits. You can preview what a crawl would prepare by running the pipeline locally with --dry-run.

Three details that trip people up:

• On the data source, leave Metadata files prefix empty, because the workflow writes each sidecar next to its document.
• The sample builds an ARM64 image by default. On an x86_64 workstation that cannot build ARM64 images, add -c cpuArchitecture=X86_64.
• If your account has VPC Block Public Access in ingress-only mode, deploy with -c egressMode=nat, which adds a NAT gateway with hourly charges.

How do you check retrieval quality?

Before adding generation, test the Retrieve API with real questions. A managed knowledge base uses hybrid search and, with the default managed embedding model, reranks with a managed reranker. The post's example asks for up to five chunks from pages with content_category equal to web that were crawled in the last seven days, using the sidecar's crawled_at_epoch number. It needs Boto3 1.43.32 or later. A managed knowledge base does not support the startsWith and stringContains filters.

For answers with citations, use agentic retrieval (AgenticRetrieveStream) on a managed knowledge base, or RetrieveAndGenerate on a customer-managed one. In both cases the cited result carries the source_url from the sidecar, so your app can show the source page. For more on the managed type, see our agentic retrieval CDK blueprint for managed knowledge bases.

What does it cost, and what can break?

AWS lists the operational limits plainly, and they matter more than the happy path.

The practical takeaway: add an alarm on ingestion failure reasons before you trust the schedule, and plan a cleanup routine for pages that disappear.

What about security, prompt injection and permissions?

AWS gives this section real weight. A few points to act on:

• Treat crawled content as untrusted input. Prepared pages become model context, so text on a public page can attempt prompt injection through retrieval. AWS says not to rely on guardrails alone: restrict seed URLs to sites you trust and filter content before it reaches the knowledge base. For how this kind of attack is tested, see our post on the Gray Swan indirect prompt injection challenge.
• Use Bedrock Guardrails and grounding checks. Attach a guardrail to retrieval requests, and turn on the contextual grounding check. Agentic retrieval supports only the guardrail BLOCK action, not MASK. The grounding check supports question answering and summarization, not chatbot use.
• Metadata is not access control. Filtering helps precision. For per-user visibility, use document-level access control with ACL files you provide, which the sample does not create.
• Respect crawl rules. The robots.txt check is on by default and the CDK path pins it on. Set delays so you do not overload sites, and only crawl sites you are authorized to crawl, in line with the AWS Acceptable Use Policy.
• Public pages only. No authenticated sessions. For sign-in sites, evaluate the Web Crawler's authentication options first.
• Least privilege. The task role can put objects under the configured prefix, start and get ingestion jobs on the target knowledge base, and read one parameter. It does not read or list S3 or call InvokeModel or Retrieve. Keep secrets out of CDK context and the plaintext parameter.

The legal side of scraping is also in flux. Publishers are suing a browser maker over scraping and paywall bypass; see our report on the New York Times lawsuit against Brave. Crawl only what you have the right to index.

How does this fit a wider AWS RAG stack?

This sample is one piece. Our other AWS coverage shows how the parts connect: Nova multimodal embeddings with AgentCore and Strands for retrieval and agents, and SageMaker distillation of a 4B shopping agent for cost. If you are new to the retrieval part, start with our explainer on what an embedding is.

A quick decision guide

Related reading

• Agentic retrieval on Bedrock managed knowledge bases: the CDK blueprint
• AWS Nova multimodal embeddings with AgentCore and Strands
• SageMaker SeqKD: a 4B shopping agent
• What is an embedding?
• Gray Swan indirect prompt injection challenge
• NYT sues Brave over AI scraping
• Sources: AWS Machine Learning blog, Crawl4AI on GitHub, companion sample repository

Details are accurate as of November 2, 2026, and follow AWS's sample, which is a starting point, not a production requirement.