> ## Documentation Index
> Fetch the complete documentation index at: https://webscraping.titannet.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# titan_crawl

> Discover a site's URL inventory or extract its pages, bounded by explicit page, depth, and same-origin limits.

Use `titan_crawl` when you need **site-level** coverage: an inventory of what exists, or content from many pages you cannot enumerate up front. For a handful of known URLs, [`titan_fetch`](/docs/mcp/tools/titan-fetch) is faster and cheaper.

Crawl is **asynchronous by default**. It returns a `run_id` immediately and you poll with [`titan_get_run`](/docs/mcp/tools/titan-get-run).

**Required scope:** `mcp:crawl`

## Two modes

| Mode            | Returns                                         | Use it to                                                  |
| --------------- | ----------------------------------------------- | ---------------------------------------------------------- |
| `map` (default) | `discovered_urls[]` — URL inventory, no content | Understand a site's structure before deciding what to read |
| `crawl`         | `pages[]` — pages with extracted content        | Extract content across many pages in one run               |

The efficient pattern is usually **map first, fetch second**: map the site to see what exists, pick the URLs that matter, then fetch only those. Mapping charges per URL discovered but extracts no content, so you avoid paying extraction cost on pages you were never going to read.

## Parameters

<ParamField body="url" type="string" required>
  Seed URL to start from. The crawl stays on this host.
</ParamField>

<ParamField body="mode" type="string" default="map">
  `map` for URL discovery, `crawl` for discovery plus content extraction.
</ParamField>

<ParamField body="max_pages" type="integer" default="25">
  Maximum pages to visit. Hard cap 100; exceeding it returns `limit_exceeded`.
</ParamField>

<ParamField body="max_depth" type="integer" default="1">
  Link depth from the seed. Hard cap 3. Depth 1 means the seed page and pages it links to directly.
</ParamField>

<ParamField body="include_patterns" type="string[]">
  Regular expressions. Only URLs matching at least one are visited. Invalid patterns are rejected before the run starts.
</ParamField>

<ParamField body="exclude_patterns" type="string[]">
  Regular expressions. Matching URLs are skipped.
</ParamField>

<ParamField body="include_content" type="boolean">
  In `map` mode, attach content to discovered URLs. In `crawl` mode content is always included.
</ParamField>

<ParamField body="content_max_chars" type="integer" default="8000">
  Character cap per page in `crawl` mode.
</ParamField>

<ParamField body="respect_robots_txt" type="boolean" default="true">
  Honor the site's `robots.txt` directives.
</ParamField>

<ParamField body="wait_for_completion" type="boolean" default="false">
  Crawl is async by default. Set `true` only for small, shallow runs that will finish inside the 30-second window.
</ParamField>

<ParamField body="timeout_seconds" type="integer" default="30">
  Seconds to wait when `wait_for_completion` is true. Capped at 30.
</ParamField>

<ParamField body="idempotency_key" type="string">
  Deduplicate repeated calls for 24 hours. See [Idempotency and retries](/docs/mcp/idempotency-and-retries).
</ParamField>

## Boundaries

Two limits are not negotiable:

* **Same origin.** Every returned URL is on the seed host. Cross-origin URLs are dropped and reported with a `cross_origin_dropped` warning.
* **Hard caps.** `max_pages` cannot exceed 100 and `max_depth` cannot exceed 3. These are rejected at validation, before any work starts.

## Example: map a documentation section

```json theme={null}
{
  "url": "https://example.com/docs/",
  "mode": "map",
  "max_pages": 100,
  "max_depth": 2,
  "include_patterns": ["/docs/api/"],
  "exclude_patterns": ["/docs/archive/", "\\.pdf$"]
}
```

## Response: map mode

```json theme={null}
{
  "run_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
  "status": "completed",
  "mode": "map",
  "discovered_urls": [
    {
      "url": "https://example.com/docs/api/authentication",
      "title": "Authentication",
      "source_url": "https://example.com/docs/",
      "depth": 1,
      "discovered_at": "2026-07-27T09:31:04Z"
    }
  ],
  "usage": {
    "backend_execution_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
    "billing_unit": "successful_delivery",
    "records_returned": 42,
    "credits_estimated": 1,
    "credits_consumed": 42,
    "billing_status": "billable",
    "limit_hit": false
  },
  "warnings": [],
  "request_id": "req_5a1c8e3f9b2d7046"
}
```

## Response: crawl mode

In `crawl` mode, `pages[]` replaces `discovered_urls[]`. Each page carries the same fields as a `titan_fetch` page, plus crawl lineage:

| Extra field      | Meaning                          |
| ---------------- | -------------------------------- |
| `source_url`     | The page that linked to this one |
| `depth`          | Link distance from the seed      |
| `crawl_seed_url` | The seed the crawl started from  |

```json theme={null}
{
  "run_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
  "status": "completed",
  "mode": "crawl",
  "pages": [
    {
      "url": "https://example.com/docs/api/authentication",
      "title": "Authentication",
      "status_code": 200,
      "content_format": "markdown",
      "content": "# Authentication\n\nAll requests require a bearer token...",
      "source_url": "https://example.com/docs/",
      "depth": 1,
      "crawl_seed_url": "https://example.com/docs/",
      "retrieved_at": "2026-07-27T09:31:47Z"
    }
  ],
  "usage": { "credits_consumed": 18, "limit_hit": false },
  "warnings": [],
  "request_id": "req_5a1c8e3f9b2d7046"
}
```

## Polling an async crawl

The default call returns immediately:

```json theme={null}
{
  "run_id": "7b2e4f18-9c53-4a06-b1d7-8e2f5a9c3b41",
  "status": "queued",
  "mode": "map",
  "warnings": [
    { "code": "next_step", "message": "call titan_get_run to retrieve discovered_urls or pages" }
  ]
}
```

Poll that `run_id` with [`titan_get_run`](/docs/mcp/tools/titan-get-run) until it reaches a terminal status.

## Credits

| Mode    | Estimate          | Charged                     |
| ------- | ----------------- | --------------------------- |
| `map`   | 1 credit          | Per URL actually delivered  |
| `crawl` | Up to `max_pages` | Per page actually extracted |

Cross-origin records dropped at the edge are **not** billed, so a crawl whose results were all filtered out costs nothing.

`limit_hit: true` in `usage` means the crawl stopped because it reached `max_pages`, not because it exhausted the site—raise the limit or narrow `include_patterns` if you need fuller coverage.

## Warnings you may see

| Code                   | Meaning                                      |
| ---------------------- | -------------------------------------------- |
| `cross_origin_dropped` | Records outside the seed host were removed   |
| `next_step`            | The run is still going; call `titan_get_run` |

## Keeping crawls tight

| Situation                                 | Approach                                                |
| ----------------------------------------- | ------------------------------------------------------- |
| Only one section matters                  | `include_patterns` scoped to that path prefix           |
| Site has archives or PDFs you do not want | `exclude_patterns` for those paths and extensions       |
| You only need structure                   | `mode: "map"` — no content extraction                   |
| The site is large                         | Start at `max_depth: 1`, widen only if coverage is thin |

## Next steps

* [titan\_get\_run](/docs/mcp/tools/titan-get-run) — poll and paginate crawl results
* [titan\_fetch](/docs/mcp/tools/titan-fetch) — read the URLs a map discovered
* [Site to knowledge base example](/docs/mcp/examples/site-to-knowledge-base)