How Amazon Scores Product Pages With a Vision Model, OCR and CatBoost: 94% Accuracy

Amazon Marketplace now grades product-page content with a recipe other teams can copy: take one snapshot of the whole page, ask a frozen vision-language model to score it on a rubric, add OCR readability scores, and let a small CatBoost classifier decide. On 5,478 labeled pages, AWS says the fused model reached 0.94 accuracy and a 0.95 F1 score, against 0.80 and 0.84 for a simple average of the model's scores.

The details come from an AWS Machine Learning blog post published November 5, 2026. It is a vendor write-up of an internal system, with measured results, so we separate what it reports from what has been independently checked (nothing yet).

TL;DR: the method and the numbers

The problem: pages are judged as a whole

AWS argues that checking assets one at a time misses the point. A blender page can have a sharp hero image, grammatical bullets and a well-lit lifestyle photo, and still fail. The bullets may advertise a 64-ounce pitcher while the photo shows a single-serve model, or the lifestyle shot may sit above the hero image. Those problems appear only when the page is read as a whole, so Amazon renders the entire page as a snapshot.

This fits a wider pattern we have covered: using a model as a judge, then calibrating it against humans. Compare Amazon Bedrock AgentCore evaluations with decision models, where a small judge screens agent outputs.

Step one: freeze the VLM, learn the weights

A pretrained VLM prompted with a rubric scores from general knowledge, which does not always match ecommerce standards. Fine-tuning would need GPUs and, with only a few thousand labels, risks overfitting. Amazon chose a different path: keep the VLM and the prompt fixed, use its structured outputs as features, and train a CatBoost classifier to predict expert labels.

The rubric has four dimensions, each with explicit criteria and point-deduction rules. For each one the VLM returns a score from 0 to 10, a confidence value and a rationale. The scores and confidences become features. The rationales go to sellers.

Because only the classifier is retrained, updating the system when annotations grow or criteria change takes a short CPU-only job. That is the engineering insight worth copying.

The labels: how trustworthy are they?

One ecommerce marketing expert and four professional annotators independently rated 5,478 product pages from 10 categories as high or low quality, then took a majority vote. The classes were nearly balanced, with 49.6 percent high quality, so a model cannot score well by guessing the majority class. Each annotator's Cohen's kappa against the consensus was between 0.845 and 0.915, which AWS describes as strong agreement.

That is a useful reference point for anyone building an evaluation set: five raters, explicit criteria and a reliability statistic.

Step two: add OCR for what the VLM misses

VLMs capture overall meaning and composition but are weaker on fine readability: crowded text, tight line spacing. They can also favor long, detailed descriptions even when shoppers want something scannable. So Amazon extracts text blocks and positions with an OCR model and computes four layout scores.

The scores depend on layout rather than wording, so two pages with identical text can score differently. The fused model has eight features: four from the VLM and four from OCR.

The AWS authors put the design choice this way: OCR "isn't a replacement signal," because it complements the VLM's holistic judgment with fine-grained text-readability information. CatBoost also suits this setup because the labeled dataset is small and tabular. For context, AWS describes A+ Content as the rich media widget on the Amazon product detail page, where brands combine text, images and video to communicate product and brand value.

The results, in numbers

On a 20 percent holdout of about 1,100 pages:

AWS summarizes the fused model as 14 percentage points better in accuracy and 11 points better in F1 than the baseline. In a second test on a single category with 499 content projects, it swapped the VLM backbone. With a lighter VLM the baseline scored 0.74 accuracy and the fusion 0.90; with a more capable one the baseline scored 0.79 and the fusion 0.94. AWS reads this as the OCR gains holding across models.

One caution: the chart AWS shows spreads results across runs, and the headline figures are averages on Amazon's own labels. They have not been independently replicated.

Step three: explain with SHAP, then recommend

A label alone does not tell a seller what to fix. Amazon uses SHAP, which assigns each feature a value showing how far it pushed a prediction toward high or low quality. Because VLM and OCR scores feed the same classifier, SHAP puts them on one scale.

Across the holdout set the most influential features were OCR Character Density (mean absolute SHAP 1.1), OCR Overall Readability (1.0) and VLM Visual Storytelling (1.0), followed by A+ Layout (0.8), Feature Presentation (0.6) and Conversion Optimization (0.4). Both sources matter.

For each low-quality page, the system ranks the eight features by SHAP value, then passes the most negative ones, the category, best practices and the VLM rationales to an LLM that writes prioritized actions. If A+ Layout has a SHAP value of minus 1.2 and Visual Storytelling minus 0.3, the layout fix comes first.

Did it help sellers and sales?

Amazon says the average post-revision quality score rose by 1.4 points on a 10-point scale, on a statistically significant sample. It also reports a worldwide A/B experiment that adjusted visibility of low-quality A+ Content based on the model's decisions, which produced a "statistically meaningful improvement" in Ordered Product Sales.

Neither the size of the sales effect nor the sample size is given. Treat the direction as reported and the magnitude as unknown.

How it runs on SageMaker AI

AWS maps each stage to a managed service: Processing Jobs for rendering and SHAP, SageMaker AI Inference for the VLM and OCR (real-time, asynchronous or batch), JumpStart for pre-trained OCR models, Feature Store for features, Training Jobs and Model Registry for the classifier, and AIOps and Pipelines for drift monitoring and retraining. A final stage uses scores and SHAP recommendations to condition a diffusion model that generates improved image assets for sellers.

If you build on AWS, related guides we have written include SageMaker SDK v3 for per-camera quality inspection models, SageMaker SOCI indexed images for faster training jobs and distilling a 4B shopping agent on SageMaker.

How to copy the recipe

1. Define a rubric with explicit criteria and deductions, and keep it stable.
2. Render the unit you care about as one image, not as parts.
3. Ask a strong VLM for scores, confidence and a rationale in structured form.
4. Add cheap, deterministic signals, such as OCR layout measures, for what the VLM handles poorly.
5. Collect a few thousand expert labels, with several raters and a reliability check.
6. Train a small tree model on the features, and hold out 20 percent.
7. Use SHAP to rank what to fix, and have a model turn the ranking into plain advice.
8. Monitor drift, and retrain the classifier, not the VLM, when standards move.

Caveats and open questions

• Self-reported. All numbers come from Amazon's post about its own system and labels.
• Context limits. AWS says standards differ across categories, languages and marketplaces, and its annotations cannot fully represent every context.
• Seller impact. A quality score and visibility changes affect sellers' sales, so how errors are handled matters. The post does not describe appeals.
• Agents are coming. How shoppers' own AI agents read these pages is a separate question. See Amazon's seller assistant and Claude plugin and Paul Graham on agentic commerce.

AWS says its next steps are more efficient VLM backbones and inference methods to run the system on a global catalog.

Related reading

• Amazon Bedrock AgentCore evaluations with decision models
• SageMaker SDK v3 for per-camera quality inspection
• SageMaker SOCI indexed images for training jobs
• Distilling a 4B shopping agent on SageMaker
• Amazon seller assistant Claude plugin
• Paul Graham on Amazon banning agents

Details are accurate as of November 5, 2026, based on the AWS Machine Learning blog post. Figures are as reported by AWS.