Skip to main content

Documentation Index

Fetch the complete documentation index at: https://webscraping.titannet.io/docs/llms.txt

Use this file to discover all available pages before exploring further.

Titan is a web intelligence and automation platform. The same system that runs classic “given these URLs, return these fields” jobs also supports finding pages on the open web, walking sites by following links, extracting structured data into schemas you define, and calling HTTP APIs as first-class steps in a workflow. You express that work as tasks; the platform runs them on managed workers, tracks executions, stores datasets and media, and ties usage to billing and analytics with stable identifiers end to end.

​
Tasks, actions, and orchestration

A task is your durable definition: what you want to achieve, how often it should run (if at all), and how results should be shaped. Under the hood, Titan can run modular actions—discrete kinds of work you compose when a problem is bigger than a single page fetch:
IdeaWhat it does
SearchTurn questions or keywords into candidate URLs or documents before you commit to specific pages.
CrawlExpand coverage from starting URLs by following links within the depth and limits you set.
ScrapeRead known URLs and return schema-shaped records (and media when your flow supports it).
API callCall partner or first-party HTTP APIs with payloads you control, then chain onward into extraction or downstream steps if needed.
You can use one action per task or chain several into a multi-step plan—for example search then scrape, or API call then crawl then scrape—so retrieval logic lives in one place instead of spread across ad hoc scripts and browsers. For terminology and API patterns, see Action types overview.

​
What you get as a product

  • Control plane — Create and update tasks, trigger runs, poll status, and fetch results through a consistent Task Service API, with JWT and API key authentication.
  • Execution model — Each run is an execution with history, retries, and exports you can reason about in dashboards or automation.
  • Workers — Browser nodes and extension-based workers perform the heavy lifting; you integrate at the task and API layer, not by hand-managing headless fleets.
  • Templates — Reuse validated patterns per site or vertical so teams share the same extraction and navigation contracts.
  • Operations — Dashboards for tasks, executions, templates, usage, and billing-related views where your deployment exposes them.

​
Who this documentation is for

  • Integrators calling the Task Service (and related APIs) from backends, data pipelines, or internal tools.
  • Product and data teams who need reliable schedules, schemas, and auditability—not one-off scripts on a laptop.
  • Agent and AI builders who want grounded, fresh web context without owning proxies, rate limits, and browser pools in the application tier.
  • Operators who need a clear map of control plane versus worker responsibilities.

​
How the pieces fit together

At a high level, Titan separates concerns the same way most managed platforms do:
  1. Experience — Dashboard and auth for people; API keys and tokens for automation.
  2. Control plane — Tasks, scheduling, executions, ingestion, templates, billing, and analytics services working together.
  3. Workers — Execution nodes that run browser automation and return structured outputs to the platform.
Most API users spend their time on the Task Service and execution lifecycle. Understanding workers matters when you tune performance, trust boundaries, or deployment topology.

​
Why teams standardize on Titan

Whether the workload is a fixed list of product pages, a discovery-heavy monitor, or an agent-driven “retrieve then reason” loop, the payoff is the same: one platform for discovery, navigation, extraction, and API glue—with execution history, datasets, and usage aligned to the same task and execution IDs. Your application stays focused on decisions and business logic; Titan carries scale, retries, and operational consistency.

​
Next steps