Sitecore Search improves crawl and indexing performance by using parallelism, which lets multiple workers (threads) crawl content at the same time. This enables faster, more efficient index updates—especially for large or frequently changing sites.
Parallelism can be configured for both Advanced Web Crawlers and API Crawlers.
You can configure parallelism with a value between 1 and 10:
- Parallelism = 1 → A single worker crawls all content sequentially.
- Parallelism = 10 → Ten workers run in parallel, each crawling content independently.
Increasing parallelism can dramatically reduce crawl time for large datasets.
But it’s important to weigh the trade-offs:
- Higher parallelism means the crawler may hit your website with multiple simultaneous requests, increasing load during scheduled scans.
- More workers also mean higher resource consumption within Sitecore Search itself.
For most implementations, start with a moderate setting and scale up based on crawl duration, site performance, and available resources.
Gopi's Blog 