Crawling URLs and multi-page collections
Updated June 14, 2026
Crawling URLs & multi-page collections
Import web content into your knowledge base by crawling URLs.
Crawling a URL
Enter any public URL and choose an import mode:
- Single page — extracts and stores only the content of the URL you entered.
- Crawl all internal pages — follows links on the same domain and imports up to 50 pages, creating a searchable collection. Use this to import a documentation site, marketing site, or any multi-page resource in one step.
Zendesk Help Center URLs (/hc/) are detected automatically and always crawl the full help centre — the mode selector is hidden for these.Auto-recrawl schedule
Set a recrawl interval (every 6 h, 12 h, 24 h, or 7 days) to keep the content in sync with your live site. You can also trigger a manual recrawl at any time using the Recrawl button. For multi-page collections, all previously discovered pages are re-fetched on each recrawl.
Viewing individual articles
Expand a collection card to see each page or article. Click View to read the stored content for any individual entry.