Create Crawling Task

URL / Sitemap / URL List

Single URL = instant preview · Multiple URLs or Sitemap = batch queue processing

Wait For (ms)

Timeout (ms)

Include Tags

Exclude Tags

Only Main ContentMobile Emulation

LLM Content CleanerSave to R2

Enter a URL and click "Scrape Now" to instantly fetch and preview the page content.

Advanced Engine Settings

Batch only in Scrape mode: multiple URLs or sitemap use /api/crawl. Single URL preview uses /api/scrape and ignores these settings.

Max Task Concurrency

Controls inline or fallback batch workers. Queue workers still follow the platform cap.

Hard URL Cap

1000

Queue Max Retries

Per-URL Timeout (sec)

Processor Flags

Enabled LLM content cleaningSkip index pages (e.g. /category)

Max Depth Limit

Save Format

Target Storage