> ## Documentation Index
> Fetch the complete documentation index at: https://webscraping.titannet.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Example: site to knowledge base

> Map a documentation site, select the pages that matter, and turn them into a searchable knowledge base your agent can answer from.

## Scenario

You want your agent to answer questions about a specific site—a vendor's documentation, your own help center, a competitor's product pages—accurately and without re-fetching on every question.

The pattern is **map, select, fetch, index**. Discover the inventory cheaply, decide what belongs in the knowledge base, extract only that, and store it.

## What you need

| Requirement  | Value                                         |
| ------------ | --------------------------------------------- |
| Scopes       | `mcp:crawl`, `mcp:fetch`, `mcp:runs:read`     |
| Tools        | `titan_crawl`, `titan_fetch`, `titan_get_run` |
| Typical cost | 1 credit per mapped URL, 1 per page extracted |

## The flow

```mermaid theme={null}
flowchart LR
    A[titan_crawl mode=map] --> B[URL inventory]
    B --> C[Filter to pages worth indexing]
    C --> D[titan_fetch in batches]
    D --> E[Chunk and embed]
    E --> F[Vector store]
```

Mapping first is what makes this affordable. A map reports what exists without extracting content, so you pay for reading only the pages you decided are worth reading.

## Step 1: map the site

```json theme={null}
{
  "name": "titan_crawl",
  "arguments": {
    "url": "https://docs.example.com/",
    "mode": "map",
    "max_pages": 100,
    "max_depth": 3,
    "include_patterns": ["/docs/"],
    "exclude_patterns": ["/docs/changelog/", "/docs/v1/", "\\.pdf$"]
  }
}
```

Crawl is asynchronous by default, so this returns a `run_id` immediately.

<Note>
  Crawls stay on the seed host. To cover `docs.example.com` and `blog.example.com`, run two maps.
</Note>

## Step 2: poll until the map finishes

```typescript theme={null}
let run = await call("titan_crawl", crawlArgs);

while (run.status === "queued" || run.status === "running") {
  await sleep(5000);
  run = await call("titan_get_run", {
    run_id: run.run_id,
    include_results: false,
  });
}

const inventory = await call("titan_get_run", {
  run_id: run.run_id,
  limit: 1000,
});
```

Polling with `include_results: false` keeps each check small, then one final call collects everything.

## Step 3: select what to index

The map gives you URLs, titles, and depth. Filter before spending fetch credits:

```typescript theme={null}
const toIndex = inventory.results
  .filter((r) => !r.url.includes("/api/generated/"))  // machine-generated reference
  .filter((r) => r.depth <= 2)                        // skip deep leaf pages
  .map((r) => r.url);
```

Judgment that belongs here:

* Drop generated API reference if your agent should answer conceptually
* Drop archived or versioned duplicates of current pages
* Drop index pages that only link elsewhere and carry no content

## Step 4: fetch in batches

`titan_fetch` takes up to 100 URLs per call. Batch below that so responses stay manageable:

```typescript theme={null}
const BATCH_SIZE = 25;
const pages = [];

for (let i = 0; i < toIndex.length; i += BATCH_SIZE) {
  const batch = toIndex.slice(i, i + BATCH_SIZE);

  let result = await call("titan_fetch", {
    urls: batch,
    format: "markdown",
    only_main_content: true,
    max_chars_per_url: 12000,
    idempotency_key: `kb-${siteId}-batch-${i}`,
  });

  while (result.status === "running" || result.status === "queued") {
    await sleep(3000);
    result = await call("titan_get_run", { run_id: result.run_id });
  }

  pages.push(...result.pages);

  if (result.failed?.length) {
    console.warn(`Batch ${i}: ${result.failed.length} URLs failed`);
  }
}
```

The batch-scoped `idempotency_key` means an interrupted build can resume without re-paying for batches that already completed.

## Step 5: chunk and store

Markdown output chunks cleanly on headings, which keeps sections intact instead of splitting mid-argument:

```typescript theme={null}
for (const page of pages) {
  const chunks = splitOnHeadings(page.content, { maxChars: 2000 });

  for (const [i, chunk] of chunks.entries()) {
    await vectorStore.upsert({
      id: `${page.url}#${i}`,
      text: chunk,
      metadata: {
        url: page.url,
        title: page.title,
        retrieved_at: page.retrieved_at,
      },
    });
  }
}
```

Keep `url` and `retrieved_at` in metadata. The first gives your agent citations; the second tells you what has gone stale.

## Keeping it fresh

Re-map on a schedule and diff against what you have indexed:

```typescript theme={null}
const currentUrls = new Set(newInventory.results.map((r) => r.url));
const indexedUrls = new Set(await vectorStore.listUrls());

const added = [...currentUrls].filter((u) => !indexedUrls.has(u));
const removed = [...indexedUrls].filter((u) => !currentUrls.has(u));

await fetchAndIndex(added);
await vectorStore.deleteByUrls(removed);
```

Because a map costs one credit per URL and no extraction, refreshing the inventory is far cheaper than re-fetching the site.

For pages that change without their URL changing, re-fetch on an age policy using `retrieved_at`, and set `freshness: "live_only"` on those calls to bypass cache.

## When to crawl instead of map

`mode: "crawl"` extracts content during discovery, in one run instead of two:

```json theme={null}
{
  "url": "https://docs.example.com/",
  "mode": "crawl",
  "max_pages": 50,
  "max_depth": 2,
  "content_max_chars": 8000
}
```

| Use                | When                                                        |
| ------------------ | ----------------------------------------------------------- |
| `map` then `fetch` | You want to filter before paying for content—the usual case |
| `crawl`            | The site is small and you want everything anyway            |

Map-then-fetch also lets you use a larger `max_chars_per_url` on the pages you keep, rather than the smaller crawl-mode default across every page.

## Cost in practice

Indexing a 100-page documentation site:

| Step                    | Credits |
| ----------------------- | ------: |
| Map 100 URLs            |     100 |
| Fetch 60 selected pages |      60 |
| **Total**               | **160** |

Crawling all 100 pages with content would cost 100 and give you 40 pages you did not want—cheaper in credits, worse in index quality. Choose based on whether filtering matters for your corpus.

## Next steps

* [titan\_crawl](/docs/mcp/tools/titan-crawl) — patterns, depth, and limits
* [Research agent](/docs/mcp/examples/research-agent) — answering from live search instead of an index
* [Credits and usage](/docs/mcp/credits-and-usage) — cost control