# scrape.it

> Turn an authenticated proxy you already own into a free, HTTP-first scraping API.

## Start here

1. Open https://api.scrape.it/auth/start.
2. Confirm an approved permanent mailbox (Gmail, Outlook, or iCloud) through the scanner-safe email flow.
3. Paste and verify one authenticated proxy URL. Use `socks5h://` when the provider supports SOCKS so target DNS resolves through the proxy.
4. Create the first API key and copy it immediately. The raw key is shown once.
5. Run the generated curl command. The verified authenticated proxy is already saved as an encrypted connector.

Every compute request requires `Authorization: Bearer sit_live_...` and exactly one caller-owned proxy source. No anonymous compute is available. Never put an API key or proxy URL in source control, chat logs, issue text, shell history, query strings, or telemetry.

## Instructions for a coding agent

When helping a user operate Scrape.it:

- Treat https://scrape.it/openapi.json as the normative wire contract.
- Ask the user to complete https://api.scrape.it/auth/start themselves if they do not yet have an API key and proxy connector ID. Never ask them to paste either secret into chat.
- Read credentials from `SCRAPEIT_API_KEY`; read a connector ID from `SCRAPEIT_PROXY_CONNECTOR_ID`.
- Send the API key only in the Authorization header.
- Prefer `render:false`. Use rendering only when the user explicitly needs it and `/v1/capabilities` reports it available.
- For a synchronous first request, use `/v1/scrape`.
- Before submitting map, batch, or crawl jobs, explain that durable jobs require a verified saved proxy and a user-owned S3, R2, or B2 destination.
- Use a unique `Idempotency-Key` for each logical durable job and reuse it only when replaying the identical request.
- Poll `/v1/jobs/{jobId}` or consume `/v1/jobs/{jobId}/events`; never expect page bodies in SSE.
- Do not infer provider quality, residentiality, fraud, or user contribution details from entitlement responses.

## First scrape

```bash
export SCRAPEIT_API_KEY="sit_live_REPLACE_ME"
export SCRAPEIT_PROXY_CONNECTOR_ID="pxy_REPLACE_ME"

curl https://api.scrape.it/v1/scrape \
  -H "Authorization: Bearer $SCRAPEIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","proxyConnectorId":"pxy_REPLACE_ME","format":"markdown","render":false}'
```

For a one-off synchronous request, `proxyConnectorId` may be replaced with an authenticated request-scoped `proxy` URL. Never send both. Saved connector ciphertext is restart-durable; request-scoped proxy secrets are not persisted.

## Durable work

- `POST /v1/map` discovers bounded same-site URLs.
- `POST /v1/batch/scrape` fetches an explicit URL list.
- `POST /v1/crawl` performs a bounded crawl.
- Durable jobs require a saved `proxyConnectorId`, a saved S3/R2/B2 storage connector or bounded presigned shard URLs, and an `Idempotency-Key`.
- Results are deterministic gzip NDJSON shards followed by manifest, summary, and `_SUCCESS`. Full result bodies are never sent through progress events.

## Free capacity and privacy

`GET /v1/me/entitlement` returns only `plan`, `baselineUnits`, `communityBonusUnits`, `availableUnits`, `currentConcurrency`, `activeJobs`, and `queuePriority`. Exact novelty, scarcity, provider-overlap, acquisition, classification, and entitlement mechanics are private. There are no public user contribution profiles.

Scrape.it measures privacy-preserving operational outcomes and proxy/pool intelligence. It never treats provider claims or third-party fraud scores as ground truth. Response bodies are not retained by default.

## Canonical references

- Capabilities: https://api.scrape.it/v1/capabilities
- OpenAPI: https://scrape.it/openapi.json
- Quickstart: https://scrape.it/quickstart.md
- Authentication: https://scrape.it/authentication.md
- Proxies: https://scrape.it/proxies.md
- Jobs: https://scrape.it/jobs-and-streaming.md
- Storage: https://scrape.it/storage-destinations.md
- Errors: https://scrape.it/errors.md
- Privacy: https://scrape.it/privacy.txt
- Terms: https://scrape.it/terms.txt
- Support: mailto:support@scrape.it