Documentation has been updated: see help center and changelog in one place.
⭐Explore
LogoLogo
Oxylabs dashboardContact usProduct
English
  • Documentation
  • Help center
  • Changelog
  • Overview
  • PROXIES
    • Integration Guides
      • Get IP Address for Integrations
      • Residential Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • FoxyProxy
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Proxifier
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
        • Hidemyacc
      • ISP Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • FoxyProxy
        • GoLogin
        • Incogniton
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • Proxifier
        • SEO Neo
        • Shadowrocket
        • Sphere
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
      • Mobile Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Proxifier
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
      • Dedicated Datacenter Proxies Guides
        • Enterprise
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • Lalicat Browser
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • Proxifier
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
        • Self-Service
          • Android
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • iOS
          • Lalicat Browser
          • MacOS
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • Proxifier
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
      • Datacenter Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • FoxyProxy
        • GoLogin
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • Proxifier
        • SEO Neo
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
    • Residential Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • ZIP/Postal code
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
        • Requests with Whitelisted IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • ISP Proxies
      • Making Requests
      • Proxy List
      • Proxy Rotation
      • Location Settings
      • Protocols
      • Whitelisting IPs
      • Response Codes
      • Restricted Targets
      • Fair usage policy
    • Mobile Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • Datacenter Proxies
      • Proxy List
      • IP Control
      • Select Country
      • Protocols
      • Whitelisting
      • Response Codes
      • Restricted Targets
      • Fair usage policy
      • Free Datacenter IPs
    • Dedicated Datacenter Proxies
      • Enterprise
        • Getting Started
        • Proxy List
        • Making Requests
        • Protocols
        • Whitelisting IPs
          • Dashboard
          • RESTful
            • Getting Whitelisted IPs List
            • Adding a Whitelisted IP
            • Removing a Whitelisted IP
            • Saving Changes (5min Cooldown)
        • Datacenter Proxy API
        • Proxy Rotator - Optional
        • Response Codes
      • Self-Service
        • Getting Started
        • Making Requests
        • Proxy List
        • Proxy Rotation
        • Location Settings
        • Protocols
        • Whitelisting IPs
        • Response Codes
        • Restricted Targets
        • Fair usage policy
    • Dedicated ISP Proxies
      • Getting Started
      • Proxy List
      • Making Requests
      • Protocols
      • Whitelisting IPs (RESTful)
        • Getting Whitelisted IPs List
        • Adding a Whitelisted IP
        • Removing a Whitelisted IP
        • Saving Changes (5min Cooldown)
      • Proxy API
      • Proxy Rotator - Optional
      • Response Codes
  • Advanced proxy solutions
    • Web Unblocker
      • Getting Started
      • Making Requests
        • Session
        • Geo-location
        • Headers & Cookies
        • Custom status code
        • POST requests
      • Headless Browser
        • JavaScript rendering
        • Browser instructions (Beta)
          • List of instructions
      • Sample Response
      • Response Codes
      • Rate Limits
      • Migration Guides
        • From Bright Data Web Unlocker
      • Usage Statistics
      • Billing Information
  • VIDEO DATA
    • High-Bandwidth Proxies
      • YouTube Downloader (yt_dlp) integration
  • Video Data API
  • Scraping Solutions
    • Web Scraper API
      • Integration Methods
        • Realtime
        • Push-Pull
        • Proxy Endpoint
      • Features
        • Localization
          • Proxy Location
          • SERP Localization
          • E-Commerce Localization
          • Domain, Locale, Results Language
        • JS Rendering & Browser Control
          • JavaScript Rendering
          • Browser Instructions
            • List of instructions
          • Capturing network requests (Fetch/XHR)
        • Result Processing & Storage
          • Dedicated Parsers
          • Custom Parser
            • Getting started
            • Parsing instruction examples
            • List of functions
              • Function examples
          • Download Images
          • Cloud Storage
        • HTTP Context & Job Management
          • Headers, Cookies, Method
          • User Agent Type
          • Client Notes
        • Scheduler
      • Solutions for AI Workflows
        • Model Context Protocol (MCP)
        • LangChain
        • LlamaIndex
      • Targets
        • Google
          • Search
            • Web Search
            • AI Overviews
            • Image Search
            • News Search
            • Local Search
            • Reverse Image Search
            • Google Suggest
          • Ads Max
          • Shopping
            • Shopping Product
            • Shopping Search
            • Shopping Pricing
          • Trends: Explore
          • Travel: Hotels
          • Lens
          • URL
        • Amazon
          • Product
          • Search
          • Pricing
          • Sellers
          • Best Sellers
          • Reviews
          • Questions & Answers
          • URL
        • YouTube
          • YouTube Scraping Guide for AI
          • YouTube Search
          • YouTube Video Trainability
          • YouTube Metadata
          • YouTube Downloader
          • YouTube Transcript
        • Generic Target
        • Walmart
          • Search
          • Product
        • Ebay
        • Etsy
          • Search
          • Product
        • Bing
          • Search
          • URL
        • ChatGPT
        • North American E-Commerce
          • Best Buy
            • Search
            • Product
          • Kroger
            • Product
            • Search
            • URL
          • Lowe's
            • Search
            • Product
            • URL
          • Target
            • Search
            • Product
            • Category
          • Bed Bath & Beyond
          • Costco
          • Menards
          • Petco
          • Staples
          • Grainger
          • Instacart
        • European E-Commerce
          • Allegro
            • Search
            • Product
          • Idealo
          • Mediamarkt
          • Cdiscount
        • Asian E-Commerce
          • Alibaba
          • Aliexpress
          • Lazada
          • Rakuten
          • Tokopedia
          • Flipkart
          • Avnet
          • Indiamart
        • Latin American E-Commerce
          • Mercado Livre
          • Magazine Luiza
          • Falabella
          • Dcard
      • Restricted Targets
      • Response Codes
      • Usage and Billing
        • Usage Statistics
        • Traffic and Billing
        • Rate Limits
    • OxyCopilot
    • Unblocking Browser
      • Chrome
      • Firefox
      • Restricted Targets
      • Integration with MCP
      • Troubleshooting Guide
  • Dashboard
    • Teams
    • Billing Information
      • Accessing Billing Information
      • Managing Payment Methods
      • Updating Billing Information
      • Canceling a Subscription
    • IP Replacement
  • Guides for Scraper APIs
    • Python SDK
    • Go SDK
    • Forming Requests
    • Forming URLs
    • Using Postman
  • Useful links
    • Oxylabs Dashboard
    • Release Notes
    • Network status
    • Open Source Tools
      • Oxy Parser
      • Oxy Mouse
      • Web Scraper API Scheduler
    • Discord Community
    • GitHub
    • Scraping Experts
  • SUPPORT
    • FAQ
    • Have a Question?
Powered by GitBook
On this page
  • Step 1: Search for videos
  • Basic search
  • Extended search
  • Search with filters
  • Step 2: Extract video IDs from search results
  • Step 3: Check AI training eligibility
  • Step 4: Get video metadata
  • Step 5: Retrieve content from selected videos
  • 5.1 Download video/audio content
  • 5.2 Retrieve video transcripts
  • Bulk processing
  • Best practices

Was this helpful?

  1. Scraping Solutions
  2. Web Scraper API
  3. Targets
  4. YouTube

YouTube Scraping Guide for AI

This guide will walk you through the workflow for collecting and filtering YouTube data for AI training purposes using Web Scraper API's specialized sources: youtube_search, youtube_video_trainability, youtube_metadata, youtube_download, and youtube_transcript.

Step 1: Search for videos

Start by searching for videos related to your topic of interest.

Basic search

For a quick search that returns up to 20 results:

{
  "source": "youtube_search",
  "query": "your search term"
}

Extended search

For more comprehensive results (up to 700 results):

{
  "source": "youtube_search_max",
  "query": "your search term"
}

Search with filters

Refine your search with filters:

{
  "source": "youtube_search",
  "query": "your search term",
  "type": "video",
  "duration": "4-20",
  "upload_date": "this_month",
  "sort_by": "view_count",
  "hd": true
}

Use the appropriate filters to narrow down results based on your specific needs. Options include content type (video, channel, playlist), duration, upload date, and quality settings.

Step 2: Extract video IDs from search results

After receiving search results, extract the video IDs for further processing. In the response from youtube_search or youtube_search_max, video IDs are directly available in the videoId field of each result item, as shown in this example response snippet:

{
    "results": [
        {
            "content": [
                {
                    "videoId": "LK9XuImr8Xg",  // This is the video ID you need
                    "thumbnail": {
                        "thumbnails": [
                            {
                                "url": "https://i.ytimg.com/vi/LK9XuImr8Xg/hq720_2.jpg?sqp=-oaymwE2COgCEMoBSFXyq4qpAygIARUAAIhCGABwAcABBvABAfgBtgiAAoAPigIMCAAQARhaIGUoLTAP&rs=AOn4CLDTvqEgoE2ZNfnn3EalF2ujcthVNw",
                                "width": 360,
                                "height": 202
                            }
                        ]
                    },
                    "title": {
                        // title details
                    }
                }
            ]
        }
    ]
}

Extract these video IDs into a list for use in subsequent API calls.

Step 3: Check AI training eligibility

Before downloading or using videos for AI training, check their eligibility:

{
  "source": "youtube_video_trainability",
  "video_id": "rFNDylrjn_w"
}

The response will indicate if the video can be used for AI training purposes:

  • ["all"] - Training permitted for all parties

  • ["none"] - No training permitted for any party

  • ["party1", "party2", ...] - Training permitted only for specific parties

Step 4: Get video metadata

Collect additional information about the videos to further evaluate their quality and relevance:

{
  "source": "youtube_metadata",
  "query": "VIDEO_ID",
  "parse": true
}

The response will contain metadata like view counts, comments, ratings, and other metrics that can help you assess content quality.

The parse parameter must be set to true for the metadata source.

Step 5: Retrieve content from selected videos

After identifying high-quality, trainable videos based on their eligibility and metadata, you can proceed with content retrieval. This can be done in two parallel steps:

5.1 Download video/audio content

{
  "source": "youtube_download",
  "query": "VIDEO_ID",
  "storage_type": "s3",
  "storage_url": "s3://your-bucket/your-folder/"
}

Additional options for download:

{
  "source": "youtube_download",
  "query": "VIDEO_ID",
  "storage_type": "s3",
  "storage_url": "s3://your-bucket/your-folder/",
  "context": [
    {
      "key": "download_type",
      "value": "video"
    },
    {
      "key": "video_quality",
      "value": "1080"
    }
  ]
}

This source is only available via the asynchronous Push-Pull integration and Cloud Storage feature.

Note:

  • Videos can be up to 3 hours in length

  • Default resolution is 720p (can be customized)

  • You can specify audio-only, video-only, or both

5.2 Retrieve video transcripts

Transcripts are not the same as closed captions (CC). Not all videos have transcripts available in all languages. If a transcript doesn't exist in your specified language, the API will return a 404 status code.

Checking if a video has transcripts:

The most efficient way to check transcript availability is by examining the video metadata (Step 4), which includes these fields:

{
    "is_transcript_available": true,
    "generated_subtitle_languages": [
        "en"
    ],
    "generated_transcript_languages": [
        "en"
    ]
}

This approach is more cost-effective than making requests that result in 404 errors, which are billable.

If the metadata shows transcripts are available, you can retrieve them with:

{
  "source": "youtube_transcript",
  "query": "VIDEO_ID",
  "context": [
    {
      "key": "language_code",
      "value": "en"
    }
  ]
}

For videos with manually created transcripts, specify:

{
  "source": "youtube_transcript",
  "query": "VIDEO_ID",
  "context": [
    {
      "key": "language_code",
      "value": "en"
    },
    {
      "key": "transcript_origin",
      "value": "uploader_provided"
    }
  ]
}

Checking if a video has transcripts (manually):

On YouTube, click the "..." menu below the video, then look for "Show transcript" in the menu options. If this option is missing, the video doesn't have transcripts available. When present, you can click it to view available transcript languages.

Bulk processing

For efficient processing of multiple videos, use batch endpoints:

{
  "source": "youtube_video_trainability",
  "query": ["VIDEO_ID_1", "VIDEO_ID_2", "VIDEO_ID_3"]
}

Best practices

  1. Follow the discovery workflow from search → trainability → metadata → content to maximize efficiency

  2. Narrow down search results before processing individual videos

  3. Always verify trainability before using content for AI

  4. Check response codes and implement retries for failed requests

PreviousYouTubeNextYouTube Search

Last updated 1 month ago

Was this helpful?