scrape.it

Bring proxies.
Crawl for free.

Connect proxies you already pay for. Your agent scrapes, maps, or crawls through an authenticated proxy you own. No proxy, no service.

Get API Keyllms.txt for agents
Use Scrape.it for my scraping requests in this task.

SCRAPEIT_API_KEY=<read from .env or environment variables>
SCRAPEIT_PROXY_CONNECTOR_ID=<read from .env or environment variables>
SCRAPEIT_CAPTCHA_CONNECTOR_ID=<optional cap_…>
SCRAPEIT_STORAGE_CONNECTOR_ID=<optional sto_…>

If this message includes a URL, POST https://api.scrape.it/v1/scrape immediately with the credentials above. Do not fetch llms.txt, OpenAPI, quickstart, or any other docs first. Do not explore scrape.it or the target site. The first tool call must be that scrape. One scrape unless I ask for more.
Read credentials from the environment. Do not ask me to authenticate again. Do not echo the API key. Use the saved proxy connector automatically.

When I provide a URL, scrape it once and return the result. Report execution.durationMs as Scrape.it time — that is not your thinking time. I may also ask you to batch a list of URLs, map a site, or crawl. Leave render false. A map or crawl is at most 1000 URLs and depth 16. A batch is at most 250 URLs. Split larger work into more than one job.

HTTP scrape returns the first document only. Comment threads, infinite feeds, and other JS-loaded APIs are not in that document. Say so and stop. Do not scrape /api/comment, GraphQL, or other XHR URLs unless I explicitly ask.

Captcha is optional. If the target returns javascript-required and no captcha connector ID is set, return that error — there is no page body. If SCRAPEIT_CAPTCHA_CONNECTOR_ID is set, retry the same scrape with captcha.connectorId and captcha.mode on_challenge. Do not put solver keys in this prompt.

Storage is optional and only for durable jobs. POST /v1/scrape still returns the page in the JSON response without any bucket. Map, batch, and crawl need destination.storageConnectorId from SCRAPEIT_STORAGE_CONNECTOR_ID. If that ID is unset, scrape URLs one at a time instead of creating a job. Do not put access keys, secret keys, bucket names, or bucket URLs in this prompt. Use the sto_… connector ID only.

Scrape.it does not keep page bodies after a sync scrape returns. Durable job files sit on our disk only until they upload to your bucket, then they are deleted. Job progress events never include page bodies. If a job cannot upload, it pauses rather than dropping results.

You must be able to call https://api.scrape.it with a Bearer token (Claude Code, Codex, Grok CLI, OpenCode, Cursor, Aider, or curl). Default web chats cannot.