Bring proxies.
Crawl for free.

Connect proxies you already pay for. Your agent scrapes, maps, or crawls through an authenticated proxy you own. No proxy, no service.

Get API Key llms.txt for agents

Use Scrape.it for my scraping requests in this task.

SCRAPEIT_API_KEY=<read from .env or environment variables>
SCRAPEIT_PROXY_CONNECTOR_ID=<read from .env or environment variables>
SCRAPEIT_CAPTCHA_CONNECTOR_ID=<optional cap_…>
SCRAPEIT_STORAGE_CONNECTOR_ID=<optional sto_…>

Read and follow https://scrape.it/llms.txt before making requests.
Read credentials from the environment. Do not ask me to authenticate again. Do not echo the API key. Use the saved proxy connector automatically.

When I provide a URL, scrape it and return the result. I may also ask you to batch a list of URLs, map a site, or crawl. Prefer a single scrape unless I ask for more. Leave render false. A map or crawl is at most 1000 URLs and depth 16. A batch is at most 250 URLs. Split larger work into more than one job.

Captcha is optional. If the target returns javascript-required and no captcha connector ID is set, return that error — there is no page body. If SCRAPEIT_CAPTCHA_CONNECTOR_ID is set, retry the same scrape with captcha.connectorId and captcha.mode on_challenge. Do not put solver keys in this prompt.

Storage is optional and only for durable jobs. POST /v1/scrape still returns the page in the JSON response without any bucket. Map, batch, and crawl need destination.storageConnectorId from SCRAPEIT_STORAGE_CONNECTOR_ID. If that ID is unset, scrape URLs one at a time instead of creating a job. Do not put bucket keys in this prompt.

Scrape.it does not keep page bodies after a sync scrape returns. Durable job files sit on our disk only until they upload to your bucket, then they are deleted. Job progress events never include page bodies. If a job cannot upload, it pauses rather than dropping results.

You must be able to call https://api.scrape.it with a Bearer token (Claude Code, Codex, Grok CLI, OpenCode, Cursor, Aider, or curl). Default web chats cannot.

export SCRAPEIT_API_KEY="sit_live_REPLACE_ME"
export SCRAPEIT_PROXY_CONNECTOR_ID="pxy_REPLACE_ME"
export SCRAPEIT_CAPTCHA_CONNECTOR_ID="cap_REPLACE_ME"
export SCRAPEIT_STORAGE_CONNECTOR_ID="sto_REPLACE_ME"

curl https://api.scrape.it/v1/scrape \
  -H "Authorization: Bearer $SCRAPEIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "proxyConnectorId": "'"$SCRAPEIT_PROXY_CONNECTOR_ID"'",
    "format": "markdown",
    "render": false
  }'

Bring proxies.Crawl for free.

Bring proxies.
Crawl for free.