> ## Documentation Index
> Fetch the complete documentation index at: https://webscraping.titannet.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Example: research agent

> Build an agent that answers questions from the live web by searching for sources, reading the promising ones, and citing what it used.

## Scenario

You want an agent that answers questions about things that happened after its training cutoff—product launches, pricing changes, release notes—and cites its sources instead of guessing.

That is the **search then fetch** pattern: discover candidates cheaply, read selectively, synthesize from real content.

## What you need

| Requirement  | Value                                          |
| ------------ | ---------------------------------------------- |
| Scopes       | `mcp:search`, `mcp:fetch`, `mcp:runs:read`     |
| Tools        | `titan_search`, `titan_fetch`, `titan_get_run` |
| Typical cost | 1 credit for the search, 1 per page read       |

## The flow

```mermaid theme={null}
sequenceDiagram
    participant User
    participant Agent
    participant Titan
    User->>Agent: Question about recent events
    Agent->>Titan: titan_search(query)
    Titan-->>Agent: Ranked URLs + snippets
    Agent->>Agent: Pick the 3-5 most relevant
    Agent->>Titan: titan_fetch(selected URLs)
    Titan-->>Agent: Clean markdown
    Agent-->>User: Answer with citations
```

The selection step in the middle is what keeps this cheap. Fetching everything a search returns wastes credits and context on results the snippets already showed were irrelevant.

## Step 1: search for sources

```json theme={null}
{
  "name": "titan_search",
  "arguments": {
    "query": "Postgres 18 release notes performance",
    "search_provider": "brave",
    "max_results": 10,
    "freshness": "month"
  }
}
```

`freshness: "month"` matters for questions about recent events—without it you get well-ranked pages from years ago.

Search costs one credit whether you ask for 10 results or 100, so ask for enough to choose from.

## Step 2: choose what to read

Use `title` and `snippet` to filter before spending credits. A useful heuristic:

* Prefer primary sources—official docs, release notes, vendor blogs—over aggregators
* Drop results whose snippet clearly answers a different question
* Keep 3 to 5 URLs; more rarely improves the answer and always costs more

## Step 3: read the selected pages

```json theme={null}
{
  "name": "titan_fetch",
  "arguments": {
    "urls": [
      "https://www.postgresql.org/docs/18/release-18.html",
      "https://www.postgresql.org/about/news/postgresql-18-released/"
    ],
    "format": "markdown",
    "only_main_content": true,
    "max_chars_per_url": 8000
  }
}
```

`only_main_content: true` strips navigation and footers, so what reaches the model is the article rather than the site chrome.

## Step 4: handle partial results

With several URLs, expect some to fail. Read both arrays:

```json theme={null}
{
  "status": "partial",
  "pages": [
    { "url": "https://www.postgresql.org/docs/18/release-18.html", "content": "# Release 18\n..." }
  ],
  "failed": [
    { "url": "https://www.postgresql.org/about/news/postgresql-18-released/", "code": "fetch_failed" }
  ]
}
```

Synthesize from `pages`, and mention the gap rather than hiding it. If a source you needed failed, fall back to the next-best URL from the original search rather than searching again.

## A prompt that produces this behavior

```text System prompt theme={null}
You answer questions using live web sources.

Process:
1. Call titan_search to find candidate sources. Use freshness when the
   question is about recent events.
2. Read the titles and snippets. Pick the 3-5 most relevant URLs,
   preferring primary sources over aggregators.
3. Call titan_fetch on only those URLs.
4. Answer from the fetched content. Cite each claim with its URL.
5. If a fetch fails, say which source was unavailable rather than
   filling the gap from memory.

Never answer from memory when the question concerns current facts.
Never fetch every search result — select first.
```

The instruction to select before fetching is what keeps cost proportional to the question.

## Complete implementation

```typescript theme={null}
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";

const transport = new StreamableHTTPClientTransport(
  new URL("https://mcp.webscraping.titannet.io/mcp"),
  {
    requestInit: {
      headers: { Authorization: `Bearer ${process.env.TITAN_API_KEY}` },
    },
  },
);

const client = new Client({ name: "research-agent", version: "1.0.0" });
await client.connect(transport);

async function call(name: string, args: Record<string, unknown>) {
  const result = await client.callTool({ name, arguments: args });
  const body = JSON.parse(result.content[0].text);
  if (result.isError) throw new Error(`${body.code}: ${body.message}`);
  return body;
}

async function research(question: string) {
  const search = await call("titan_search", {
    query: question,
    search_provider: "brave",
    max_results: 10,
    freshness: "month",
  });

  // Selection: primary sources first, capped at 4.
  const selected = search.results
    .filter((r) => !AGGREGATORS.has(new URL(r.url).hostname))
    .slice(0, 4)
    .map((r) => r.url);

  if (selected.length === 0) {
    return { answer: "No usable sources found.", sources: [] };
  }

  const fetched = await call("titan_fetch", {
    urls: selected,
    format: "markdown",
    only_main_content: true,
    max_chars_per_url: 8000,
    idempotency_key: `research-${hash(question)}`,
  });

  return {
    sources: fetched.pages.map((p) => ({ url: p.url, title: p.title, content: p.content })),
    unavailable: fetched.failed.map((f) => f.url),
    creditsUsed: fetched.usage.credits_consumed,
  };
}
```

The `idempotency_key` derived from the question means a retried research call replays the original fetch instead of paying twice. See [Idempotency and retries](/docs/mcp/idempotency-and-retries).

## Handling slow fetches

A large batch can exceed the 30-second wait window. When it does, poll rather than re-fetch:

```typescript theme={null}
async function fetchWithPolling(urls: string[]) {
  let result = await call("titan_fetch", { urls, format: "markdown" });

  while (result.status === "running" || result.status === "queued") {
    await sleep(3000);
    result = await call("titan_get_run", { run_id: result.run_id });
  }

  return result;
}
```

## Cost in practice

A typical question:

| Step                            | Credits |
| ------------------------------- | ------: |
| One search returning 10 results |       1 |
| Fetching 4 selected pages       |       4 |
| **Total**                       |   **5** |

Fetching all 10 results instead would cost 11 credits for an answer that is rarely better. Selection is the whole game.

## Next steps

* [Site to knowledge base](/docs/mcp/examples/site-to-knowledge-base) — indexing a whole site instead
* [Programmatic integration](/docs/mcp/examples/programmatic-integration) — the full client setup
* [titan\_search](/docs/mcp/tools/titan-search) — operators for sharper queries