Site Inventory

The Site Inventory is a persistent crawl of your site, kept fresh by your chosen cadence. It's the foundation for Topical Maps, Internal Linking Suggestions, What Changed, and most of the agent's site-aware answers.

What gets stored

Per crawl run, for every page discovered:

• Full HTTP metadata (status, response time, page size, content type, redirect chain)
• Content (title, meta description, H1, headings hierarchy, body text up to 32 KB)
• SEO signals (canonical, indexability, schema.org types, image alt-text coverage)
• Link graph (every internal + external link with anchor text, rel attributes, position)
• Semantic embeddings (one vector per page, used for semantic search and similarity)

A content-linked FTS5 index over title / H1 / body text powers full-text search; a separate page_embeddings table powers semantic search. Both auto-update when pages are re-crawled.

How crawls work

• Robots.txt-aware (User-agent / longest-match / Allow-wins parser, Crawl-delay respected).
• Discovers via sitemaps first (CDATA-aware, gzip-aware, recursive), falls back to BFS.
• Per-host rate limiting (200ms default gap, configurable via robots.txt Crawl-delay).
• 5-way concurrency pool by default; 20-second per-fetch timeout.
• Cancellable mid-run from the UI.

After each crawl finishes, two passes fire automatically: an embedding pass (sends pages to your configured embedding provider) and a fact-extraction pass (writes structural facts to the Fact Store).

Configuration per site

In Settings → Sites, configure per-site:

• Max pages - Hard cap on URLs to crawl (default 1,000; tier-bounded).
• Refresh cadence - How often the daily cron should re-crawl (0 = manual only).
• Respect robots.txt - On by default. Turn off only for sites you own and have explicit reason to ignore.

Embedding provider

Configurable in Settings → Site Indexing (self-hosted installs only). By default the embedding pass uses OpenAI's text-embedding-3-small (1536d, $0.02 per million tokens) sharing your LLM API key. For OSS self-hosters, an optional Xenova local fallback (nomic-embed-text-v1.5, 768d) runs entirely on your hardware if you install @xenova/transformers yourself.

Agent integration

The agent has 10+ tools that read directly from the Site Inventory: search_site_content (FTS), semantic_search, find_similar_pages, get_site_inventory, find_orphan_pages, get_page_detail, inbound/outbound link walks, and site-wide anchor-text auditing. The agent prefers these over live-fetch tools when the question is about your own site, and falls back gracefully when no crawl exists yet (it'll point you to /crawls).

Per-tier crawl page caps