LLMs and AI

Find Oxylabs Web Scraper API scraping guides for most popular LLMs and AI platforms.

Oxylabs LLM scrapers let you submit prompts to leading AI platforms and receive structured and parsed responses. Supported targets:

• ChatGPT – source: chatgpt
• Gemini– source: gemini
• Perplexity – source: perplexity

All dedicated LLM & AI scrapers support custom browser instructions. Learn more.

Integration method

LLM & AI sources use the Push-Pull integration method. Instead of holding the connection open until the response is ready, you submit a job and collect the result once it's finished:

1. Submit the job: send your prompt to the Push-Pull endpoint. You get back a job ID immediately, not the result.
2. Wait for it to finish: either let us notify you with callback_url (recommended), or check the job status yourself.
3. Retrieve the result: once the status is done, fetch the parsed response.

Realtime and Proxy Endpoint are not available for chatgpt, gemini, and perplexity sources. These targets take longer to complete than a synchronous connection allows, so Push-Pull is the only supported method.

Request sample

All LLM targets use mostly the same request parameters, the main difference is the maximum prompt length.

Submit the payload to the Push-Pull endpoint - see the individual target pages for full code examples.

{
  "source": "chatgpt",
  "prompt": "best supplements for better sleep",
  "parse": true,
  "geo_location": "United States",
  "callback_url": "https://your-server.com/oxylabs-callback"
}

Parameter	Description	Type
source	The LLM target to scrape. Allowed values: chatgpt, perplexity, gemini.	string
prompt	Query or prompt to submit. Maximum length: 4,000 characters for chatgpt; 8,000 characters for perplexity and gemini.	string
callback_url	Recommended. URL where we deliver a notification once the job completes, so you don't have to check the status yourself. More info.	string
parse	Set to true to receive a structured JSON response.	boolean
geo_location	Geographic location to route the request from (e.g., "United States").	string
browser_instructions	Execute optional custom browser instructions when rendering JavaScript. More info.	array

- mandatory parameter

Note: JavaScript rendering is enabled by default for all LLM targets. Do not include render in your request payload.

Response

All LLM targets return the same top-level data structure (see the individual target pages for the full response field reference).

Submitting the job

Submitting returns the job details, including the ID you'll use to collect the result. It does not contain the scraped data.

Field	Description	Type
id	Unique identifier for the scraping job. Use this to retrieve the result.	string
status	Current status of the job: pending while it runs, done when the result is ready, faulted if it failed (retrieve the result only once the status is done).	string
source	Submitted source value.	string
parse	Submitted parse value.	boolean
prompt	Original prompt in the request.	string
geo_location	Geolocation used for the request. Check LLM availability before choosing a geo_location.	string
created_at	Job creation timestamp (UTC, YYYY-MM-DD HH:MM:SS).	string
updated_at	Job update timestamp.	string

Check LLM availability before choosing a geo_location

Each AI service is only available in certain countries, and we can only scrape it from countries where the service itself is available. For example, Gemini is not available in mainland China, so a Gemini request with "geo_location": "China" cannot succeed.

Before sending requests, check the provider's official list of supported countries:

• ChatGPT – ChatGPT Supported Countries
• Gemini – Where you can use the Gemini web app
• Perplexity – no official list is published.

Retrieving results

Once the job status is done, retrieve the result using the job.id you received. The parsed content specific to each target is found inside results[].content (see the individual target pages for the full field reference).

Field	Description	Type
results[].job_id	Job ID associated with the result.	string
results[].status_code	HTTP status code of page fetch.	integer
results[].url	Target LLM page.	string
results[].content	LLM response data. Fields vary by target.	object

Response data by target

The table below shows which data fields are available for each LLM target.

	ChatGPT	Gemini	Perplexity
Prompt input	✓	✓	✓
Parsed output (JSON)	✓	✓	✓
Plain text response	✓	✓	–
Markdown output	✓	✓	✓
Markdown JSON tree	✓	–	✓
Search / related queries	–	–	✓
Citations / sources	✓	✓	✓
Shopping results	✓	–	–
Ads	✓	–	–
Geolocation	✓	✓	✓
LLM model used	–	✓	✓

– conditional, returned only when content is in the LLM's response.