Skip to main content
ZICQ

Skills ZICQ category:Documents bright-data-best-practices

Bright Data Best Practices

Build production-ready Bright Data integrations with best practices baked in. Reference documentation for developers using coding assistants (Claude Code, Cursor, etc.) to implement web scraping, search, browser automation, and structured data extraction. Covers Web Unlocker API, SERP API, Web Scraper API, and Browser API (Scraping Browser).

1715 installs

Official URL:skills.sh

What this skill does

Intro in this page language first. The official description stays in its original wording; we do not rewrite SKILL.md.

What it does

Build production-ready Bright Data integrations with best practices baked in. Reference documentation for developers using coding assistants (Claude Code, Cursor, etc.) to implement web scraping, search, browser automation, and structured data extraction. Covers Web Unlocker API, SERP API, Web Scraper API, and Browser API (Scraping Browser).

When to use it

The official description does not include a separate “Use when”. Per the spec, agents activate this skill when the task matches keywords in that description.

How agents load it

Per Agent Skills progressive disclosure: name and description load at startup (~100 tokens); the full SKILL.md body loads when the skill activates; scripts/, references/, and assets/ load only as needed. This file's sections: CLI Setup Reference; Bright Data APIs; Choosing the Right API; Authentication Pattern (All APIs); Web Unlocker API; Key Parameters.

File analysis

File analysis: besides SKILL.md, the body references references/cli-setup.md, references/web-unlocker.md, references/serp-api.md, references/web-scraper-api.md, references/browser-api.md. Those resources load on demand.

CLI Setup ReferenceBright Data APIsChoosing the Right APIAuthentication Pattern (All APIs)Web Unlocker APIKey ParametersQuick PatternsGet markdown (best for LLM input)Geo-targeted requestScreenshot for debuggingAsync for bulk processingSERP API

Source category:skills.sh agent-skill

SKILL.md & Agent activation

Official spec ↗
name
bright-data-best-practices
description
Build production-ready Bright Data integrations with best practices baked in. Reference documentation for developers using coding assistants (Claude Code, Cursor, etc.) to implement web scraping, search, browser automation, and structured data extraction. Covers Web Unlocker API, SERP API, Web Scraper API, and Browser API (Scraping Browser).
  1. DiscoverThe client exposes names and descriptions to the agent.
  2. ActivateYour request or the task context selects the skill and loads its instructions.
  3. Load resourcesReferenced scripts, documentation and assets are used when needed.
Files referenced by the instructions · 5
  • references/cli-setup.md
  • references/web-unlocker.md
  • references/serp-api.md
  • references/web-scraper-api.md
  • references/browser-api.md

These paths are extracted from the text. Check the upstream package to verify the files exist.

Invocation syntax and available tools depend on your Agent client. Client integration guide ↗

Install this skill

Skills CLI ↗

Choose the target agent and installation scope, keep referenced package files, then verify the skill appears in the client's catalog.

This skill references supporting files. Retrieve the complete directory from the source; copying SKILL.md alone may leave missing dependencies.

Ask your Agent to install

Copy these instructions to a compatible agent and confirm the target directory matches your client.

Install the agent skill "bright-data-best-practices" into my project. The full SKILL.md and official description are at https://zicq.com/en/skills/skl-04b8e24ab69a5ae9-Bright-Data-Best-Practices.html
Save it as .cursor/skills/bright-data-best-practices/SKILL.md or .claude/skills/bright-data-best-practices/SKILL.md and keep the frontmatter name and description exactly as-is.
This skill also ships scripts/, references/, or assets/ — fetch the whole folder from https://github.com/brightdata/skills instead of creating only a SKILL.md.

Full package on GitHub ↗

Install from the terminal · Skills CLI

Requires Node.js and npx. First inspect the repository's skill list to confirm the name.

npx skills add 'https://github.com/brightdata/skills' --list

npx skills add 'https://github.com/brightdata/skills' --skill 'bright-data-best-practices'

The CLI lets you choose the agent interactively. The default scope is the project; use -g for user scope. Confirm package availability with the discovery command, then use npx skills list to inspect installed skills.

Readable layout

name: bright-data-best-practices description: "Build production-ready Bright Data integrations with best practices baked in. Reference documentation for developers using coding assistants (Claude Code, Cursor, etc.) to implement web scraping, search, browser automation, and structured data extraction. Covers Web Unlocker API, SERP API, Web Scraper API, and Browser API (Scraping Browser)." user-invocable: false

CLI Setup Reference

Install, authentication, and troubleshooting for the Bright Data CLI (bdata) are documented in a single canonical place:

→ references/cli-setup.md

Consult it before any task that shells out to bdata.

Bright Data APIs

Bright Data provides infrastructure for web data extraction at scale. Four primary APIs cover different use cases — always pick the most specific tool for the job.

Choosing the Right API

| Use Case | API | Why | |----------|-----|-----| | Scrape any webpage by URL (no interaction) | Web Unlocker | HTTP-based, auto-bypasses bot detection, cheapest | | Google / Bing / Yandex search results | SERP API | Specialized for SERP extraction, returns structured data | | Structured data from Amazon, LinkedIn, Instagram, TikTok, etc. | Web Scraper API | Pre-built scrapers, no parsing needed | | Click, scroll, fill forms, run JS, intercept XHR | Browser API | Full browser automation | | Puppeteer / Playwright / Selenium automation | Browser API | Connects via CDP/WebDriver | | Route your own HTTP client through a raw proxy (DC/ISP/Residential/Mobile) | Proxy networks | When you need direct proxy access with your own request logic instead of a managed API — see the proxy.md skill |

Authentication Pattern (All APIs)

All APIs share the same authentication model. The env vars below apply to direct REST API integrations — if you are using the bdata CLI, bdata login handles all of these automatically (see references/cli-setup.md).

export BRIGHTDATA_API_KEY="your-api-key"         # From Control Panel > Account Settings
export BRIGHTDATA_UNLOCKER_ZONE="zone-name"       # Web Unlocker zone name
export BRIGHTDATA_SERP_ZONE="serp-zone-name"      # SERP API zone name
export BROWSER_AUTH="brd-customer-ID-zone-NAME:PASSWORD"  # Browser API credentials

REST API authentication header for Web Unlocker and SERP API:

Authorization: Bearer YOUR_API_KEY

Web Unlocker API

HTTP-based scraping proxy. Best for simple page fetches without browser interaction.

Endpoint: POST https://api.brightdata.com/request

import requests

response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "zone": "YOUR_ZONE_NAME",
        "url": "https://example.com/product/123",
        "format": "raw"
    }
)
html = response.text

Key Parameters

| Parameter | Type | Description | |-----------|------|-------------| | zone | string | Zone name (required) | | url | string | Target URL with http:// or https:// (required) | | format | string | "raw" (HTML) or "json" (structured wrapper) (required) | | method | string | HTTP verb, default "GET" | | country | string | 2-letter ISO for geo-targeting (e.g., "us", "de") | | data_format | string | Transform: "markdown" or "screenshot" | | async | boolean | true for async mode |

Quick Patterns

# Get markdown (best for LLM input)
response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"zone": ZONE, "url": url, "format": "raw", "data_format": "markdown"}
)

# Geo-targeted request
json={"zone": ZONE, "url": url, "format": "raw", "country": "de"}

# Screenshot for debugging
json={"zone": ZONE, "url": url, "format": "raw", "data_format": "screenshot"}

# Async for bulk processing
json={"zone": ZONE, "url": url, "format": "raw", "async": True}

Critical rule: Never use Web Unlocker with Puppeteer, Playwright, Selenium, or anti-detect browsers. Use Browser API instead.

See references/web-unlocker.md for complete reference including proxy interface, special headers, async flow, features, and billing.


SERP API

Structured search engine result extraction for Google, Bing, Yandex, DuckDuckGo.

Endpoint: POST https://api.brightdata.com/request (same as Web Unlocker)

response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "zone": "YOUR_SERP_ZONE",
        "url": "https://www.google.com/search?q=python+web+scraping&brd_json=1&gl=us&hl=en",
        "format": "raw"
    }
)
data = response.json()
for result in data.get("organic", []):
    print(result["rank"], result["title"], result["link"])

Essential Google URL Parameters

| Parameter | Description | Example | |-----------|-------------|---------| | q | Search query | q=python+web+scraping | | brd_json | Parsed JSON output | brd_json=1 (always use for data pipelines) | | gl | Country for search | gl=us | | hl | Language | hl=en | | start | Pagination offset | start=10 (page 2), start=20 (page 3) | | tbm | Search type | tbm=nws (news), tbm=isch (images), tbm=vid (videos) | | brd_mobile | Device | brd_mobile=1 (mobile), brd_mobile=ios | | brd_browser | Browser | brd_browser=chrome | | brd_ai_overview | Trigger AI Overview | brd_ai_overview=2 | | uule | Encoded geo location | for precise location targeting |

Note: num parameter is deprecated as of September 2025. Use start for pagination.

Parsed JSON Response Structure

{
  "organic": [{"rank": 1, "global_rank": 1, "title": "...", "link": "...", "description": "..."}],
  "paid": [],
  "people_also_ask": [],
  "knowledge_graph": {},
  "related_searches": [],
  "general": {"results_cnt": 1240000000, "query": "..."}
}

Bing Key Parameters

| Parameter | Description | |-----------|-------------| | q | Search query | | setLang | Language (prefer 4-letter: en-US) | | cc | Country code | | first | Pagination (increment by 10: 1, 11, 21...) | | safesearch | off, moderate, strict | | brd_mobile | Device type |

Async for Bulk SERP

# Submit
response = requests.post(
    "https://api.brightdata.com/request",
    params={"async": "1"},
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"zone": SERP_ZONE, "url": "https://www.google.com/search?q=test&brd_json=1", "format": "raw"}
)
response_id = response.headers.get("x-response-id")

# Retrieve (retrieve calls are NOT billed)
result = requests.get(
    "https://api.brightdata.com/serp/get_result",
    params={"response_id": response_id},
    headers={"Authorization": f"Bearer {API_KEY}"}
)

Billing: Pay per 1,000 successful requests only. Async retrieve calls are not billed.

See references/serp-api.md for complete reference including Maps, Trends, Reviews, Lens, Hotels, Flights parameters.


Web Scraper API

Pre-built scrapers for structured data extraction from 100+ platforms. No parsing logic needed.

Sync Endpoint: POST https://api.brightdata.com/datasets/v3/scrape Async Endpoint: POST https://api.brightdata.com/datasets/v3/trigger

# Sync (up to 20 URLs, returns immediately)
response = requests.post(
    "https://api.brightdata.com/datasets/v3/scrape",
    params={"dataset_id": "YOUR_DATASET_ID", "format": "json"},
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"input": [{"url": "https://www.amazon.com/dp/B09X7M8TBQ"}]}
)

if response.status_code == 200:
    data = response.json()  # Results ready
elif response.status_code == 202:
    snapshot_id = response.json()["snapshot_id"]  # Poll for completion

Parameters

| Parameter | Type | Description | |-----------|------|-------------| | dataset_id | string | Scraper identifier from the Scraper Library (required) | | format | string | json (default), ndjson, jsonl, csv | | custom_output_fields | string | Pipe-separated fields: url\|title\|price | | include_errors | boolean | Include error info in results |

Request Body

{
  "input": [
    { "url": "https://www.amazon.com/dp/B09X7M8TBQ" },
    { "url": "https://www.amazon.com/dp/B0B7CTCPKN" }
  ]
}

Poll for Async Results

import time

# Trigger
snapshot_id = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={"dataset_id": DATASET_ID, "format": "json"},
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"input": [{"url": u} for u in urls]}
).json()["snapshot_id"]

# Poll
while True:
    status = requests.get(
        f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
        headers={"Authorization": f"Bearer {API_KEY}"}
    ).json()["status"]

    if status == "ready": break
    if status == "failed": raise Exception("Job failed")
    time.sleep(10)

# Download
data = requests.get(
    f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
    params={"format": "json"},
    headers={"Authorization": f"Bearer {API_KEY}"}
).json()

Progress status values: starting → running → ready | failed Data retention: 30 days. Billing: Per delivered record. Invalid input URLs that fail are still billable.

See references/web-scraper-api.md for complete reference including scraper types, output formats, delivery options, and billing details.


Browser API (Scraping Browser)

Full browser automation via CDP/WebDriver. Handles CAPTCHA, fingerprinting, and anti-bot detection automatically.

Connection:

  • Playwright/Puppeteer: wss://${AUTH}@brd.superproxy.io:9222
  • Selenium: https://${AUTH}@brd.superproxy.io:9515
const { chromium } = require("playwright-core");

const AUTH = process.env.BROWSER_AUTH;
const browser = await chromium.connectOverCDP(`wss://${AUTH}@brd.superproxy.io:9222`);
const page = await browser.newPage();
page.setDefaultNavigationTimeout(120000); // Always set to 2 minutes

await page.goto("https://example.com", { waitUntil: "domcontentloaded" });
const html = await page.content();
await browser.close();
from playwright.async_api import async_playwright

async with async_playwright() as p:
    browser = await p.chromium.connect_over_cdp(f"wss://{AUTH}@brd.superproxy.io:9222")
    page = await browser.new_page()
    page.set_default_navigation_timeout(120000)
    await page.goto("https://example.com", wait_until="domcontentloaded")
    html = await page.content()
    await browser.close()

Custom CDP Functions

| Function | Purpose | |----------|---------| | Captcha.solve | Manually trigger CAPTCHA solving | | Captcha.setAutoSolve | Enable/disable auto CAPTCHA solving | | Proxy.setLocation | Set precise geo location (call BEFORE goto) | | Proxy.useSession | Maintain same IP across sessions | | Emulation.setDevice | Apply device profile (iPhone 14, etc.) | | Emulation.getSupportedDevices | List available device profiles | | Unblocker.enableAdBlock | Block ads to save bandwidth | | Unblocker.disableAdBlock | Re-enable ads | | Input.type | Fast text input for bulk form filling | | Browser.addCertificate | Install client SSL cert for session | | Page.inspect | Get DevTools debug URL for live session |

// CDP session pattern for custom functions
const client = await page.target().createCDPSession();

// CAPTCHA solve with timeout
const result = await client.send("Captcha.solve", { timeout: 30000 });

// Precise geo location (must be before goto)
await client.send("Proxy.setLocation", {
  latitude: 37.7749,
  longitude: -122.4194,
  distance: 10,
  strict: true
});

// Block unnecessary resources
await client.send("Network.setBlockedURLs", { urls: ["*google-analytics*", "*.ads.*"] });

// Device emulation
await client.send("Emulation.setDevice", { deviceName: "iPhone 14" });

Session Rules

  • One initial navigation per session — new URL = new session
  • Idle timeout: 5 minutes
  • Max duration: 30 minutes

Geolocation

  • Country-level: append -country-us to credentials username
  • EU-wide: append -country-eu (routes through 29+ European countries)
  • Precise: use Proxy.setLocation CDP command (before navigation)

Error Codes

| Code | Issue | Fix | |------|-------|-----| | 407 | Wrong port | Playwright/Puppeteer → 9222, Selenium → 9515 | | 403 | Bad auth | Check credentials format and zone type | | 503 | Service scaling | Wait 1 minute, reconnect |

Billing: Traffic-based only. Block images/CSS/fonts to reduce costs.

See references/browser-api.md for complete reference including all CDP functions, bandwidth optimization, CAPTCHA patterns, and debugging.


Detailed References

  • references/web-unlocker.md — Web Unlocker: full parameter list, proxy interface, special headers, async flow, features, billing, anti-patterns
  • references/serp-api.md — SERP API: all Google params (Maps, Trends, Reviews, Lens, Hotels, Flights), Bing params, parsed JSON structure, async, billing
  • references/web-scraper-api.md — Web Scraper API: sync vs async, all parameters, polling, scraper types, output formats, billing
  • references/browser-api.md — Browser API: connection strings, session rules, all CDP functions, geo-targeting, bandwidth optimization, CAPTCHA, debugging, error codes

Related Skills

  • brightdata-proxy — For routing requests through Bright Data's raw proxy networks (Datacenter, ISP, Residential, Mobile) with your own HTTP client instead of a managed API. Covers network/IP-pool selection, the brd-customer-... username format, targeting & sticky-session params, SSL CA setup for Residential/Mobile, and integrations for cURL, Python (requests/httpx/aiohttp/Scrapy), Node (fetch/axios), Playwright, Puppeteer, and Selenium. Hand off to it whenever the task is raw proxy access rather than Web Unlocker / SERP / Web Scraper / Browser API. Escalation order when proxies hit consistent blocks: raw proxy → Web Unlocker → Browser API → Web Scraper API.

Related skills

Documents

Ontology

Typed knowledge graph for structured agent memory and composable skills. Use when creating/querying entities (Person, Project, Task, Event, …

Documents

Nano Pdf

Edit PDFs with natural-language instructions using the nano-pdf CLI.

Documents

Word / DOCX

Create, inspect, and edit Microsoft Word documents and DOCX files with reliable styles, numbering, tracked changes, tables, sections, and co…

Documents

Api Gateway

Call third-party APIs through the Maton gateway, which injects the credential for an app the user has already connected. Use this skill when…