titan_crawl

Use titan_crawl when you need site-level coverage: an inventory of what exists, or content from many pages you cannot enumerate up front. For a handful of known URLs, titan_fetch is faster and cheaper. Crawl is asynchronous by default. It returns a run_id immediately and you poll with titan_get_run. Required scope: mcp:crawl

Two modes

Mode	Returns	Use it to
`map` (default)	`discovered_urls[]` — URL inventory, no content	Understand a site’s structure before deciding what to read
`crawl`	`pages[]` — pages with extracted content	Extract content across many pages in one run

The efficient pattern is usually map first, fetch second: map the site to see what exists, pick the URLs that matter, then fetch only those. Mapping charges per URL discovered but extracts no content, so you avoid paying extraction cost on pages you were never going to read.

Parameters

string

required

Seed URL to start from. The crawl stays on this host.

string

default:"map"

map for URL discovery, crawl for discovery plus content extraction.

integer

default:"25"

Maximum pages to visit. Hard cap 100; exceeding it returns limit_exceeded.

integer

default:"1"

Link depth from the seed. Hard cap 3. Depth 1 means the seed page and pages it links to directly.

string[]

Regular expressions. Only URLs matching at least one are visited. Invalid patterns are rejected before the run starts.

string[]

Regular expressions. Matching URLs are skipped.

boolean

In map mode, attach content to discovered URLs. In crawl mode content is always included.

integer

default:"8000"

Character cap per page in crawl mode.

boolean

default:"true"

Honor the site’s robots.txt directives.

boolean

default:"false"

Crawl is async by default. Set true only for small, shallow runs that will finish inside the 30-second window.

integer

default:"30"

Seconds to wait when wait_for_completion is true. Capped at 30.

string

Deduplicate repeated calls for 24 hours. See Idempotency and retries.

Boundaries

Two limits are not negotiable:

Same origin. Every returned URL is on the seed host. Cross-origin URLs are dropped and reported with a cross_origin_dropped warning.
Hard caps. max_pages cannot exceed 100 and max_depth cannot exceed 3. These are rejected at validation, before any work starts.

Example: map a documentation section

{
  "url": "https://example.com/docs/",
  "mode": "map",
  "max_pages": 100,
  "max_depth": 2,
  "include_patterns": ["/docs/api/"],
  "exclude_patterns": ["/docs/archive/", "\\.pdf$"]
}

Response: map mode

{
  "run_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
  "status": "completed",
  "mode": "map",
  "discovered_urls": [
    {
      "url": "https://example.com/docs/api/authentication",
      "title": "Authentication",
      "source_url": "https://example.com/docs/",
      "depth": 1,
      "discovered_at": "2026-07-27T09:31:04Z"
    }
  ],
  "usage": {
    "backend_execution_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
    "billing_unit": "successful_delivery",
    "records_returned": 42,
    "credits_estimated": 1,
    "credits_consumed": 42,
    "billing_status": "billable",
    "limit_hit": false
  },
  "warnings": [],
  "request_id": "req_5a1c8e3f9b2d7046"
}

Response: crawl mode

In crawl mode, pages[] replaces discovered_urls[]. Each page carries the same fields as a titan_fetch page, plus crawl lineage:

Extra field	Meaning
`source_url`	The page that linked to this one
`depth`	Link distance from the seed
`crawl_seed_url`	The seed the crawl started from

{
  "run_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
  "status": "completed",
  "mode": "crawl",
  "pages": [
    {
      "url": "https://example.com/docs/api/authentication",
      "title": "Authentication",
      "status_code": 200,
      "content_format": "markdown",
      "content": "# Authentication\n\nAll requests require a bearer token...",
      "source_url": "https://example.com/docs/",
      "depth": 1,
      "crawl_seed_url": "https://example.com/docs/",
      "retrieved_at": "2026-07-27T09:31:47Z"
    }
  ],
  "usage": { "credits_consumed": 18, "limit_hit": false },
  "warnings": [],
  "request_id": "req_5a1c8e3f9b2d7046"
}

Polling an async crawl

The default call returns immediately:

{
  "run_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
  "status": "queued",
  "mode": "map",
  "warnings": [
    { "code": "next_step", "message": "call titan_get_run to retrieve discovered_urls or pages" }
  ]
}

Poll that run_id with titan_get_run until it reaches a terminal status.

Credits

Mode	Estimate	Charged
`map`	1 credit	Per URL actually delivered
`crawl`	Up to `max_pages`	Per page actually extracted

Cross-origin records dropped at the edge are not billed, so a crawl whose results were all filtered out costs nothing. limit_hit: true in usage means the crawl stopped because it reached max_pages, not because it exhausted the site—raise the limit or narrow include_patterns if you need fuller coverage.

Warnings you may see

Code	Meaning
`cross_origin_dropped`	Records outside the seed host were removed
`next_step`	The run is still going; call `titan_get_run`

Keeping crawls tight

Situation	Approach
Only one section matters	`include_patterns` scoped to that path prefix
Site has archives or PDFs you do not want	`exclude_patterns` for those paths and extensions
You only need structure	`mode: "map"` — no content extraction
The site is large	Start at `max_depth: 1`, widen only if coverage is thin

Next steps

titan_get_run — poll and paginate crawl results
titan_fetch — read the URLs a map discovered
Site to knowledge base example

Get started

Tool reference

How it works

Integration examples

Two modes

Parameters

Boundaries

Example: map a documentation section

Response: map mode

Response: crawl mode

Polling an async crawl

Credits

Warnings you may see

Keeping crawls tight

Next steps

​Two modes

​Parameters

​Boundaries

​Example: map a documentation section

​Response: map mode

​Response: crawl mode

​Polling an async crawl

​Credits

​Warnings you may see

​Keeping crawls tight

​Next steps

Two modes

Parameters

Boundaries

Example: map a documentation section

Response: map mode

Response: crawl mode

Polling an async crawl

Credits

Warnings you may see

Keeping crawls tight

Next steps