Skip to main content

Documentation Index

Fetch the complete documentation index at: https://webscraping.titannet.io/docs/llms.txt

Use this file to discover all available pages before exploring further.

Titan is built around actions, not only “given these URLs, return fields.” A task can describe discovery (search), link expansion (crawl), structured extraction (scrape), and HTTP tool calls (api_call). You combine them into single-action tasks (one action_type) or multi-action tasks (execution_plan with multiple steps and input_source wiring). You can express discovery, link expansion, structured extraction, and outbound HTTP in one task model on your infrastructure, with templates, scripts, billing, and execution history aligned to the same task and execution IDs.
Tab titles (cURL, Go, TypeScript, Python, Rust) match Authentication and the Quickstart. Rust examples use ureq and serde_json with .expect() for short, copy-friendly snippets.

The four action types

action_typePurpose
searchDiscover URLs or documents from the open web using queries and ranking before you commit to specific pages.
crawlFollow links from seeds with depth and limits—navigation-first expansion of what to read.
scrapeGiven concrete URLs, extract structured data (and media where applicable) against an output schema.
api_callCall external HTTP APIs as a first-class step (payloads, validation, and keys are part of your configuration).
Serious workflows often chain two or more actions (for example search → scrape, crawl → scrape, or api_call → scrape).

How tasks express actions

Classic (non-modular) task

You supply urls, objective, output_schema, and optional template or script selection. The platform treats this as a single extraction-style run unless you add modular fields.

Single-action modular task

Set action_type to one of search, crawl, scrape, or api_call, plus modular fields, and omit execution_plan:
  • input_source — where inputs come from for that step (see table below).
  • template_id / template_slug and optional script_id / script_version — bind validation, defaults, and runtime script.
  • limits, payload, retry_policy — step-level behavior.
The service validates configuration, resolves the script, and saves a single-step plan so each execution loads the same definition. Responses expose is_modular when an action is in play; is_multi_action stays false for a single effective step.

Multi-action modular task (execution_plan)

Send an execution_plan on create or update with one or more steps. Each step includes step_id, action_type, input_source, optional template or script per step, limits, payload, retry policy, and optional next_step_id / metadata. Rules that matter for the API:
  • execution_plan cannot be combined with top-level modular fields (action_type, template_id, output_schema, limits, payload, and so on). The plan must be self-contained; the same mutual exclusion applies on PUT /api/v1/tasks/:id when updating with a plan.
  • Saving multi-step tasks requires your Titan environment to support persisted multi-step workflows. If that is not enabled, creating a task with an execution_plan may fail with a clear error—check with your operator or support for your environment.
  • secret_payload is rejected on create; pass secrets at run time via POST /api/v1/tasks/:id/run or POST /api/v1/executions payloads.
For modular runs, the stored workflow is the source of truth: execution_plan overrides on trigger are not supported the way they are for some classic tasks.

Input sources (input_source)

ValueMeaning
static_urlsUse the task’s configured urls (after validation for the action). Typical for the first step or whenever inputs should only be user-supplied seeds.
previous_stepConsume outputs from the prior step (for example URLs from search feeding scrape).
task_url_inventoryDrive work from the task URL inventory for large or evolving URL sets. Not every environment exposes this flow.
Validation rejects inconsistent combinations; POST /api/v1/tasks errors reflect the same rules applied when a run is started.

Templates and scripts

Templates bundle metadata and versioned scripts; script rows can declare an action_type so you know which behavior a version implements. List script versions with GET /api/v1/templates/:id/scripts (include_inactive=true when you need inactive rows). Multi-action plans can point different steps at different templates when your product allows it.

Preview and iteration

  • POST /api/v1/executions/preview — dry-run an execution_plan (objective + plan) without persisting a permanent task.
  • POST /api/v1/templates/:id/preview — template-scoped preview (requires executions:write; subject to preview quotas).
Set TITAN_API_URL to your Task Service origin (no /api/v1 suffix) and TITAN_TOKEN to a bearer token with executions:write. Replace the empty steps array with a real plan when you call this for real.
curl -sS -X POST "$TITAN_API_URL/api/v1/executions/preview" \
  -H "Authorization: Bearer $TITAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "objective": "Dry-run a modular plan before persisting a task",
    "execution_plan": { "steps": [] }
  }'