Skip to main content

Documentation Index

Fetch the complete documentation index at: https://webscraping.titannet.io/docs/llms.txt

Use this file to discover all available pages before exploring further.

If you are building a backend integration, internal tool, agent runtime, or custom frontend, this is the service you will use most often. It is the user-facing control plane for task definitions (including modular search, crawl, scrape, and API-call steps), execution lifecycle, templates, results, datasets, and stable media access.
Rust snippets use ureq and serde_json (ureq = { version = "2", features = ["json"] }, serde_json = "1"). Go snippets use the standard library only. Tab titles match Authentication and the Quickstart so your language choice stays in sync.

What you will build

In this guide, you will build a minimal programmatic integration that can:
  1. Authenticate with a bearer token
  2. Create a task
  3. Trigger execution
  4. Poll execution status
  5. Export results

Before you begin

RequirementExample
Task Service base URLhttps://api.webscraping.titannet.io (no /api/v1 suffix in env vars below)
Bearer tokenJWT or API key
Required scopestasks:read, tasks:write, executions:read, executions:write, data:read
export TITAN_API_URL="https://api.webscraping.titannet.io"
export TITAN_TOKEN="your-bearer-token"

How Task Service fits into the platform

Response model

Most JSON endpoints return a success envelope with the typed payload under data:
{
  "success": true,
  "data": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "active",
    "created_at": "2025-01-01T00:00:00Z",
    "updated_at": "2025-01-01T00:00:00Z"
  }
}
Errors use success: false and an error object with code and message (see API reference overview and HTTP errors and exceptions). When building a client, parse and check the envelope first, then read fields from data. A few routes (for example GET …/results/export) stream files instead of JSON envelopes—treat those as raw HTTP bodies.

Main endpoint groups

CapabilityCommon endpoint group
Task management/api/v1/tasks
Execution control/api/v1/executions
Templates/api/v1/templates
Result access/api/v1/executions/:id/results
Dataset access/api/v1/tasks/:id/datasets and /api/v1/executions/:id/datasets
Media access/api/v1/executions/:id/media

Build a minimal client

These helpers assume TITAN_API_URL is the origin (for example https://api.webscraping.titannet.io) and append /api/v1.
# JSON body POST
titan_post() {
  local path="$1"
  shift
  curl -sS -X POST "${TITAN_API_URL}/api/v1${path}" \
    -H "Authorization: Bearer ${TITAN_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$@"
}

# GET with bearer
titan_get() {
  curl -sS "${TITAN_API_URL}/api/v1$1" \
    -H "Authorization: Bearer ${TITAN_TOKEN}"
}

Create a task programmatically

The table below describes the classic extraction shape most teams start with: known URLs plus an objective and output schema. Titan also supports modular and multi-step tasks where urls, output_schema, and other fields are expressed per action or inside an execution_plan instead of only at the top level. For those patterns, see Action types overview and the Task Service operations in the API Reference tab.
FieldRequired (classic)Meaning
nameYesHuman-readable task label
urlsYesTarget URLs
objectiveYesWhat the run should accomplish
output_schemaYesJSON schema describing the structured result
execution_typeYessingle or scheduled
scheduleFor scheduled tasksCron string such as 0 * * * *

Example: create a single-run task

curl -sS -X POST "$TITAN_API_URL/api/v1/tasks" \
  -H "Authorization: Bearer $TITAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Product extraction",
    "urls": ["https://example.com/product/1"],
    "objective": "Extract product details",
    "output_schema": {
      "type": "object",
      "properties": {
        "title": { "type": "string" },
        "price": { "type": "string" }
      }
    },
    "execution_type": "single"
  }'

Example: create a scheduled task

curl -sS -X POST "$TITAN_API_URL/api/v1/tasks" \
  -H "Authorization: Bearer $TITAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Hourly price monitor",
    "urls": ["https://example.com/product/1"],
    "objective": "Track the current price",
    "output_schema": {
      "type": "object",
      "properties": {
        "price": { "type": "string" },
        "timestamp": { "type": "string" }
      }
    },
    "execution_type": "scheduled",
    "schedule": "0 * * * *"
  }'

Trigger and monitor execution

PatternEndpointBest for
Run saved taskPOST /api/v1/tasks/:id/runStandard user workflows
Trigger execution directlyPOST /api/v1/executionsExecution-oriented integrations

Run a saved task

curl -sS -X POST "$TITAN_API_URL/api/v1/tasks/$TASK_ID/run" \
  -H "Authorization: Bearer $TITAN_TOKEN"
The JSON response wraps an ExecutionResponse in data; use data.id as the execution UUID for later GET /api/v1/executions/:id calls.

Poll execution state

curl -sS "$TITAN_API_URL/api/v1/executions/$EXECUTION_ID" \
  -H "Authorization: Bearer $TITAN_TOKEN"
Read data.status from the envelope on each poll.

Export results

When the execution reaches a terminal status such as completed, switch from execution monitoring to result access.

Result metadata (JSON envelope)

curl -sS "$TITAN_API_URL/api/v1/executions/$EXECUTION_ID/results" \
  -H "Authorization: Bearer $TITAN_TOKEN"

Export as JSON file (raw body, no envelope)

curl -sS "$TITAN_API_URL/api/v1/executions/$EXECUTION_ID/results/export" \
  -H "Authorization: Bearer $TITAN_TOKEN" \
  -o results.json

Download as CSV

curl -sS "$TITAN_API_URL/api/v1/executions/$EXECUTION_ID/results/download" \
  -H "Authorization: Bearer $TITAN_TOKEN" -o results.csv

A full client flow

Copy the Build a minimal client snippet for your language, then run the flow below (same titanRequest / titan_json names where applicable).
# Requires titan_post / titan_get from "Build a minimal client" (cURL tab).
TASK_ID=$(titan_post /tasks "$(cat <<'JSON'
{
  "name": "My task",
  "urls": ["https://example.com"],
  "objective": "Extract data",
  "output_schema": { "type": "object", "properties": { "title": { "type": "string" } } },
  "execution_type": "single"
}
JSON
)" | jq -r '.data.id')

RUN_JSON=$(curl -sS -X POST "${TITAN_API_URL}/api/v1/tasks/${TASK_ID}/run" \
  -H "Authorization: Bearer ${TITAN_TOKEN}")
EXEC_ID=$(echo "$RUN_JSON" | jq -r '.data.id')

while true; do
  ST=$(titan_get "/executions/${EXEC_ID}" | jq -r '.data.status')
  echo "status=$ST"
  echo "$ST" | grep -qE 'completed|failed|cancelled|completed_with_errors' && break
  sleep 3
done

curl -sS "${TITAN_API_URL}/api/v1/executions/${EXEC_ID}/results/export" \
  -H "Authorization: Bearer ${TITAN_TOKEN}" -o results.json

Best practices

PracticeWhy it matters
Keep one reusable task per workflowProduces cleaner execution history
Parse the response envelope consistentlyAvoids ad hoc response handling
Poll executions, then switch to result endpointsKeeps runtime and result logic separate
Use API keys for automationAvoids browser-session coupling
Keep output schemas stableMakes downstream consumption simpler

Troubleshooting

ProblemWhat to check
Task creation returns 400Invalid body or output schema
Execution never moves past queuedWorker availability or scheduler health
Export endpoints failExecution may not be completed yet
Media does not render in frontendUse the correct export mode or media download path

Next steps