Example: site to knowledge base

Scenario

You want your agent to answer questions about a specific site—a vendor’s documentation, your own help center, a competitor’s product pages—accurately and without re-fetching on every question. The pattern is map, select, fetch, index. Discover the inventory cheaply, decide what belongs in the knowledge base, extract only that, and store it.

What you need

Requirement	Value
Scopes	`mcp:crawl`, `mcp:fetch`, `mcp:runs:read`
Tools	`titan_crawl`, `titan_fetch`, `titan_get_run`
Typical cost	1 credit per mapped URL, 1 per page extracted

The flow

Mapping first is what makes this affordable. A map reports what exists without extracting content, so you pay for reading only the pages you decided are worth reading.

Step 1: map the site

{
  "name": "titan_crawl",
  "arguments": {
    "url": "https://docs.example.com/",
    "mode": "map",
    "max_pages": 100,
    "max_depth": 3,
    "include_patterns": ["/docs/"],
    "exclude_patterns": ["/docs/changelog/", "/docs/v1/", "\\.pdf$"]
  }
}

Crawl is asynchronous by default, so this returns a run_id immediately.

Crawls stay on the seed host. To cover docs.example.com and blog.example.com, run two maps.

Step 2: poll until the map finishes

let run = await call("titan_crawl", crawlArgs);

while (run.status === "queued" || run.status === "running") {
  await sleep(5000);
  run = await call("titan_get_run", {
    run_id: run.run_id,
    include_results: false,
  });
}

const inventory = await call("titan_get_run", {
  run_id: run.run_id,
  limit: 1000,
});

Polling with include_results: false keeps each check small, then one final call collects everything.

Step 3: select what to index

The map gives you URLs, titles, and depth. Filter before spending fetch credits:

const toIndex = inventory.results
  .filter((r) => !r.url.includes("/api/generated/"))  // machine-generated reference
  .filter((r) => r.depth <= 2)                        // skip deep leaf pages
  .map((r) => r.url);

Judgment that belongs here:

Drop generated API reference if your agent should answer conceptually
Drop archived or versioned duplicates of current pages
Drop index pages that only link elsewhere and carry no content

Step 4: fetch in batches

titan_fetch takes up to 100 URLs per call. Batch below that so responses stay manageable:

const BATCH_SIZE = 25;
const pages = [];

for (let i = 0; i < toIndex.length; i += BATCH_SIZE) {
  const batch = toIndex.slice(i, i + BATCH_SIZE);

  let result = await call("titan_fetch", {
    urls: batch,
    format: "markdown",
    only_main_content: true,
    max_chars_per_url: 12000,
    idempotency_key: `kb-${siteId}-batch-${i}`,
  });

  while (result.status === "running" || result.status === "queued") {
    await sleep(3000);
    result = await call("titan_get_run", { run_id: result.run_id });
  }

  pages.push(...result.pages);

  if (result.failed?.length) {
    console.warn(`Batch ${i}: ${result.failed.length} URLs failed`);
  }
}

The batch-scoped idempotency_key means an interrupted build can resume without re-paying for batches that already completed.

Step 5: chunk and store

Markdown output chunks cleanly on headings, which keeps sections intact instead of splitting mid-argument:

for (const page of pages) {
  const chunks = splitOnHeadings(page.content, { maxChars: 2000 });

  for (const [i, chunk] of chunks.entries()) {
    await vectorStore.upsert({
      id: `${page.url}#${i}`,
      text: chunk,
      metadata: {
        url: page.url,
        title: page.title,
        retrieved_at: page.retrieved_at,
      },
    });
  }
}

Keep url and retrieved_at in metadata. The first gives your agent citations; the second tells you what has gone stale.

Keeping it fresh

Re-map on a schedule and diff against what you have indexed:

const currentUrls = new Set(newInventory.results.map((r) => r.url));
const indexedUrls = new Set(await vectorStore.listUrls());

const added = [...currentUrls].filter((u) => !indexedUrls.has(u));
const removed = [...indexedUrls].filter((u) => !currentUrls.has(u));

await fetchAndIndex(added);
await vectorStore.deleteByUrls(removed);

Because a map costs one credit per URL and no extraction, refreshing the inventory is far cheaper than re-fetching the site. For pages that change without their URL changing, re-fetch on an age policy using retrieved_at, and set freshness: "live_only" on those calls to bypass cache.

When to crawl instead of map

mode: "crawl" extracts content during discovery, in one run instead of two:

{
  "url": "https://docs.example.com/",
  "mode": "crawl",
  "max_pages": 50,
  "max_depth": 2,
  "content_max_chars": 8000
}

Use	When
`map` then `fetch`	You want to filter before paying for content—the usual case
`crawl`	The site is small and you want everything anyway

Map-then-fetch also lets you use a larger max_chars_per_url on the pages you keep, rather than the smaller crawl-mode default across every page.

Cost in practice

Indexing a 100-page documentation site:

Step	Credits
Map 100 URLs	100
Fetch 60 selected pages	60
Total	160

Crawling all 100 pages with content would cost 100 and give you 40 pages you did not want—cheaper in credits, worse in index quality. Choose based on whether filtering matters for your corpus.

Next steps

titan_crawl — patterns, depth, and limits
Research agent — answering from live search instead of an index
Credits and usage — cost control

Get started

Tool reference

How it works

Integration examples

Scenario

What you need

The flow

Step 1: map the site

Step 2: poll until the map finishes

Step 3: select what to index

Step 4: fetch in batches

Step 5: chunk and store

Keeping it fresh

When to crawl instead of map

Cost in practice

Next steps

​Scenario

​What you need

​The flow

​Step 1: map the site

​Step 2: poll until the map finishes

​Step 3: select what to index

​Step 4: fetch in batches

​Step 5: chunk and store

​Keeping it fresh

​When to crawl instead of map

​Cost in practice

​Next steps

Scenario

What you need

The flow

Step 1: map the site

Step 2: poll until the map finishes

Step 3: select what to index

Step 4: fetch in batches

Step 5: chunk and store

Keeping it fresh

When to crawl instead of map

Cost in practice

Next steps