The web was built for humans. AI agents and software need a compiler.
Today, we’re opening Atlas to developers.
Atlas is a deterministic semantic compiler and web scraper built for AI and autonomous agents. It transforms bloated, unstructured HTML, single-page applications (SPAs), and complex documentation into clean, machine-readable structured intelligence that LLMs, AI agents, and data pipelines can reliably execute on.
Rather than treating web extraction as a prompt-engineering problem or relying on an LLM to probabilistically guess what a page means, Atlas takes a compiler-first approach: a deterministic pipeline designed to translate unstructured web documents into clean, structured representations.
Why Web Extraction Is Broken
Modern applications are increasingly dependent on live web data, yet the tooling to extract it has remained remarkably fragile:
- The LLM Token Tax: Raw web pages are filled with megabytes of layout boilerplate, tracking telemetry scripts, presentational CSS, and navigation menus. Feeding raw HTML directly into LLM context windows wastes context budgets, drives up inference costs, and dilutes model attention, leading to reasoning hallucinations.
- Fragile DOM Selectors: Traditional scrapers rely on hardcoded CSS selectors or XPath queries that break the moment an upstream site modifies its layout, class names, or component tree.
- Sluggish Headless Browsers: Running heavy containerized headless browsers consumes hundreds of megabytes of RAM per page, introduces excessive latency, and frequently crashes under production workloads.
- No Intermediate Representation (IR): Downstream systems need predictable schemas, tabular precision, and clean semantics—not unvalidated string blobs mixed with layout trivia and navigation chrome.
We believe web intelligence infrastructure should be more than a collection of extraction prompts. It should be predictable, programmable, and built to integrate directly into the software developers are already building.
A Compiler-First Architecture
Atlas is built around a simple premise: web extraction should be treated as a compilation problem, not an ad-hoc scraping or prompting exercise.
- Superior Output with Fewer Tokens: Unlike conventional web scrapers that produce noisy text dumps or fragile selector slices, Atlas extracts high-fidelity semantic Markdown that captures complete page information while drastically reducing token overhead.
- Eliminate Over 90% of Page Noise: By structurally identifying substantive article and documentation content, Atlas can reduce more than 90% of extra junk—including navigation bars, cookie banners, tracking scripts, and presentational boilerplate—without losing multi-column table structures, syntax-highlighted code fences, or outbound links.
- Deterministic by Design: The core extraction process relies on deterministic compiler passes rather than probabilistic LLM inference. Outputs are consistent, reproducible, and immune to prompt injection attacks embedded in untrusted web pages.
- Built for Downstream Software: Instead of raw HTML mixed with layout trivia, Atlas delivers clean, strictly typed outputs ready for immediate ingestion into AI context windows, vector databases, and automated workflows.
5 Core Operations
Atlas provides a comprehensive API surface engineered for both interactive developer tools and high-throughput automated agent loops:
scrape(POST /v1/scrape): Real-time Markdown and metadata extraction for low-latency agent loops.compile(POST /v1/compile): Deep AST compilation with structural content graph extraction and visual formatting.map(POST /v1/map): Rapid topological discovery engine that crawls and heuristically ranks (0–100) all reachable URLs across a domain without downloading full page bodies.crawl(POST /v1/crawls): Distributed, asynchronous multi-page spidering with glob path filtering (includePaths,excludePaths) and depth limits.batch(POST /v1/batches): Resilient parallel processing for up to 1,000 explicit URLs per job with partial failure isolation and cryptographic webhook completion events.
Developer Platform & Agent Ecosystem
Atlas is built to integrate seamlessly with the developer tools and AI assistants you use every day:
- Official Multi-Language SDKs:
- TypeScript / JavaScript:
@atlascompiler/sdk(npm install @atlascompiler/sdk) with full type safety, automatic idempotency keys, and webhook verification helpers. - Python:
atlascompiler(pip install atlascompiler) supporting both synchronous (AtlasClient) and asynchronous (AsyncAtlasClient) workflows. - Go:
go.atlas-compiler.com/sdk(go get go.atlas-compiler.com/sdk) with standard library context cancellation and zero external dependencies.
- TypeScript / JavaScript:
- Model Context Protocol (MCP 2026-07-28):
- Official
@atlascompiler/mcpserver with 1-click interactive setup (npx -y @atlascompiler/mcp init) for Cursor, Claude Desktop, Windsurf, and VS Code. - Dual-transport architecture: local stdio process and cloud-native Streamable HTTP at
https://api.atlas-compiler.com/mcp. - 5 native agent tools (
atlas_scrape,atlas_compile,atlas_map,atlas_crawl,atlas_batch) with token budget safeguards and virtualatlas://resource URIs.
- Official
- Developer Dashboard: Live interactive Playground, instant API key generation, quota tracking, and real-time execution analytics.
Start Building in 30 Seconds
You can make your first request to the Atlas API right now with cURL:
curl -X POST https://api.atlas-compiler.com/v1/scrape \
-H "Authorization: Bearer atlas_your_api_key_here" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com"
}'
The response delivers clean, structured Markdown, rich metadata, and outbound document links without navigation chrome, footers, or tracking beacons:
{
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)",
"links": [
"https://www.iana.org/domains/example"
]
},
"metadata": {
"title": "Example Domain",
"canonicalUrl": "https://example.com/",
"author": null,
"description": "Illustrative example domain managed by IANA.",
"publishedDate": null,
"extractionConfidence": 0.98,
"language": "en"
},
"usage": {
"credits": 1
}
}
Or using the TypeScript SDK:
import { AtlasClient } from '@atlascompiler/sdk';
const client = new AtlasClient({
apiKey: process.env.ATLAS_API_KEY!,
});
const response = await client.scrape({
url: 'https://example.com',
});
if (response.status === 200) {
console.log('Title:', response.value.metadata.title);
console.log('Markdown:\n', response.value.data.markdown);
console.log('Credits Used:', response.value.usage.credits);
}
What’s Next
Today’s v1.0.0 public launch establishes the foundation for deterministic web compilation. In upcoming releases, we’re expanding the developer platform and agent ecosystem with:
- AI Agent Framework Connectors: Dedicated integrations and starter templates for LangGraph, CrewAI, LlamaIndex, and the Vercel AI SDK.
- Streaming Token Responses: Chunked
ReadableStreamdelivery pipelines for feeding real-time document output directly into model inference streams. - Custom Extraction Overlays: Programmable CSS/selector hint configurations to fine-tune compilation output on highly proprietary web architectures.
- Granular Webhook Delivery Policies: Extended event routing controls, custom retry intervals, and deep delivery telemetry in the developer dashboard.
- Expanded Public Benchmarks: Community-driven evaluation benchmarks across the top 10,000 developer documentation domains.
The web is full of information. We’re building better infrastructure to make it usable.
Welcome to Atlas.