Crawler Architecture & Politeness

Moniva uses a two-tiered crawler engine built with Crawlee to balance speed, resource efficiency, and JavaScript rendering capability.

Tier 1: CheerioCrawler

Over 90% of pages are crawled using lightweight HTTP requests and Cheerio HTML parsing, consuming minimal CPU and RAM on our worker nodes.

Tier 2: Playwright Fallback

When a page heavily relies on client-side React/Vue hydration to render outbound comparison tables, the job automatically promotes to headless Chromium via Playwright.