Back to all articles
Tutorials 5 min read

Getting Started with Atlas: Your First Web Compilation in 5 Minutes

A step-by-step developer tutorial: from obtaining an API key to compiling your first web page into structured Markdown.

Getting clean web data into your applications and AI agents shouldn’t require complex browser automation scripts or brittle CSS selector maintenance.

Atlas transforms the web into a programmable interface. In this quickstart tutorial, we’ll walk through obtaining your API key, compiling your first public web page, interpreting the structured response, and integrating Atlas into your preferred language or AI assistant—all in under five minutes.


Prerequisites: Get Your API Key

Before making your first API call, you’ll need an active Atlas API key:

  1. Head to the Atlas Dashboard and sign in.
  2. Navigate to API Keys in the sidebar.
  3. Click Create API Key, name your key (e.g., development-local), and copy the generated secret.

Atlas API keys follow a unified format beginning with the atlas_ prefix:

export ATLAS_API_KEY="atlas_your_api_key_here"

Security Tip: Always store your key in environment variables or a secret manager. Never commit keys to public version control or expose them in client-side browser code.


Step 1: Your First Request with cURL

No SDK installation is required to test the API. Open your terminal and issue a standard POST request to the synchronous /v1/scrape endpoint:

curl -X POST https://api.atlas-compiler.com/v1/scrape \
  -H "Authorization: Bearer $ATLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

Within moments, Atlas fetches the target web page, strips navigation chrome and telemetry scripts, isolates the substantive content, and returns clean, structured data.


Step 2: Interpreting the Response

Atlas responses are strictly typed and deterministic. Here is the canonical JSON payload returned by the request above:

{
  "data": {
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)",
    "links": [
      "https://www.iana.org/domains/example"
    ]
  },
  "metadata": {
    "title": "Example Domain",
    "canonicalUrl": "https://example.com/",
    "author": null,
    "description": "Illustrative example domain managed by IANA.",
    "publishedDate": null,
    "extractionConfidence": 0.98,
    "language": "en"
  },
  "usage": {
    "credits": 1
  }
}

Breakdown of Response Fields

  • data.markdown: Standard GitHub Flavored Markdown (GFM) containing the core article or document body. Headings, code blocks, lists, and multi-column tables are preserved, while navigation bars, cookie consent banners, and tracking scripts are eliminated.
  • data.links: An array of outbound URLs discovered within the substantive document body, with relative links automatically resolved into fully qualified absolute URLs.
  • metadata: Extracted document context:
    • title: Page title resolved from OpenGraph metadata and primary heading tags.
    • canonicalUrl: The authoritative URL following any HTTP redirects (301, 302, 308).
    • extractionConfidence: A score between 0.0 and 1.0 reflecting extraction clarity.
    • language: BCP 47 language code (e.g., en, de, es).
  • usage.credits: Number of billing credits deducted for this operation (1 credit per standard synchronous compile).

Step 3: Choosing Your Integration Method

While cURL is great for testing, Atlas provides official client SDKs with built-in retries, type definitions, and automatic idempotency key generation.

Option A: TypeScript / JavaScript

Install the official client library:

npm install @atlascompiler/sdk
# or: pnpm add @atlascompiler/sdk

Compile a URL with full TypeScript autocomplete:

import { AtlasClient } from '@atlascompiler/sdk';

const client = new AtlasClient({
  apiKey: process.env.ATLAS_API_KEY!,
});

const response = await client.scrape({
  url: 'https://atlas-compiler.com',
  formats: ['markdown', 'links'],
});

if (response.status === 200) {
  console.log('Title:', response.value.metadata.title);
  console.log('Markdown:\n', response.value.data.markdown);
  console.log('Credits Used:', response.value.usage.credits);
}

Option B: Python

Install the official Python package:

pip install atlascompiler

Execute a compile using the synchronous or asynchronous client:

import os
from atlascompiler import AtlasClient

with AtlasClient(os.getenv("ATLAS_API_KEY")) as client:
    response = client.scrape({"url": "https://docs.python.org/3/"})
    
    if response.status == 200:
        print("Title:", response.value["metadata"]["title"])
        print("Markdown:\n", response.value["data"]["markdown"])

Option C: Go

Install the Go SDK:

go get go.atlas-compiler.com/sdk

Call the client using standard Go context patterns:

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"go.atlas-compiler.com/sdk"
)

func main() {
	client := atlas.New(os.Getenv("ATLAS_API_KEY"))

	res, _, err := client.Scrape(context.Background(), atlas.ScrapeUrlRequest{
		Url: "https://go.dev/blog/",
	})
	if err != nil {
		log.Fatalf("Compilation failed: %v", err)
	}

	if res.Response200 != nil {
		fmt.Println("Title:", res.Response200.Metadata.Title)
		fmt.Println(res.Response200.Data.Markdown)
	}
}

Step 4: Connecting AI Agents via Model Context Protocol (MCP)

If you’re using AI coding environments like Cursor, Windsurf, Claude Desktop, or VS Code, you don’t even need to write application code. Atlas provides an official Model Context Protocol (MCP) server.

Run the 1-click interactive wizard in your terminal:

npx -y @atlascompiler/mcp init

The setup wizard will:

  1. Detect your installed AI editors.
  2. Prompt for your Atlas API key.
  3. Automatically configure .cursor/mcp.json or your Claude Desktop config without exposing secrets in process arguments.

Once installed, your AI agent can invoke atlas_scrape or atlas_compile whenever it needs to research documentation or inspect live web APIs.


Error Handling & Problem Details (RFC 9457)

All non-2xx responses from the Atlas API return standardized RFC 9457 Problem Details objects, making error handling predictable across all languages:

{
  "type": "https://docs.atlas-compiler.com/errors#rate_limit_exceeded",
  "title": "Too Many Requests",
  "status": 429,
  "code": "rate_limit_exceeded",
  "detail": "Rate limit of 60 requests per minute exceeded for this workspace.",
  "requestId": "req_01J9X1234567890ABCDEF",
  "retryable": true
}

In the TypeScript SDK, errors are thrown as AtlasApiError:

import { AtlasApiError } from '@atlascompiler/sdk';

try {
  await client.scrape({ url: 'https://example.com' });
} catch (error) {
  if (error instanceof AtlasApiError) {
    console.error(`Atlas Error [${error.problem.status}] ${error.problem.code}: ${error.problem.detail}`);
    console.error('Request ID:', error.problem.requestId);
  }
}

Next Steps

Now that you’ve completed your first compilation, explore what else Atlas can do:

  • Domain Mapping (POST /v1/map): Map an entire website’s topology and rank all reachable URLs before crawling.
  • Distributed Crawling (POST /v1/crawls): Launch multi-page recursive crawls with glob pattern inclusion/exclusion rules.
  • Batch Compilation (POST /v1/batches): Ingest up to 1,000 URLs in parallel with partial failure isolation and webhook notifications.

Resources to continue building:

Back to Blog See What's New in Atlas ↗