跳到主内容
智客 ZICQ

技能库 智客分类:媒体内容 scrape

Scrape

通过Bright Data CLI( " bdata scrape " )将网页内容作为清洁标记下/HTML/JSON。 当用户想要获取一个页面,从URL列表中提取内容,或者爬行相加的列表时使用. 在必须首先发现URL时,不要给所支持的平台(Amazon、LinkedIn、TikTok、Instagram、YouTube、Reddit等)“数据反馈”和“搜索”。 需要亮相数据 CLI; 如果缺少, 主动引导安装 + 登录 .

10740 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

通过Bright Data CLI(`bdata scrape')将网络内容作为清洁标记下/HTML/JSON

何时用

用户想要获取一个页面,从 URL 列表中提取内容,或者爬行 pagized 列表。 在必须首先发现URL时,不要给所支持的平台(Amazon、LinkedIn、TikTok、Instagram、YouTube、Reddit等)“数据反馈”和“搜索”。 需要亮点

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Bright Data — Scrape、Setup gate (run first)、Pick your path、Action、Clean markdown (default)、Raw HTML (when you need the DOM)。

文件分析

文件分析:除 SKILL.md 外,正文引用了 references/cli-setup.md、references/patterns.md、references/examples.md、references/flags.md,属于带资源的技能包,这些文件按需再读。

官方 description(原文)

Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing.

Bright Data — ScrapeSetup gate (run first)Pick your pathActionClean markdown (default)Raw HTML (when you need the DOM)Structured JSON (when the Unlocker returns parsed fields)Visual snapshot (saves PNG)Geo-targeted (override the exit country)Verification gate (run before claiming success)Red flagsReferences

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
scrape
description
Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing.
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。
指令中引用的文件 · 4
  • references/cli-setup.md
  • references/patterns.md
  • references/examples.md
  • references/flags.md

以下路径提取自原文;文件是否齐全请以来源仓库中的完整目录为准。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

该技能引用了附属文件,请从来源获取完整目录;仅复制 SKILL.md 可能缺少依赖。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「scrape」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-a9910fc0e93e1e64-%E6%8B%BC%E5%9B%BE.html
请存为 .cursor/skills/scrape/SKILL.md 或 .claude/skills/scrape/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
该技能还带 scripts/、references/、assets/ 等文件,请从 https://github.com/brightdata/skills 取完整目录,不要只建一个 SKILL.md。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/brightdata/skills' --list

npx skills add 'https://github.com/brightdata/skills' --skill 'scrape'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: scrape description: Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing. --- # Bright Data — Scrape Get clean content (markdown, HTML, JSON, screenshot) from one or more URLs via the Bright Data CLI. This skill owns the "fetch raw or lightly-structured content" job. For platform-specific structured data (Amazon, LinkedIn, TikTok, etc.), **stop and use `data-feeds` instead** — you'll get clean JSON without selector logic. ## Setup gate (run first) Before any scrape, verify the CLI is installed and authenticated: ```bash if ! command -v bdata >/dev/null 2>&1; then echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md" elif ! bdata zones >/dev/null 2>&1; then echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)" fi ``` If either check fails, halt and route the user to `skills/bright-data-best-practices/references/cli-setup.md`. Do not attempt the legacy `curl` fallback silently — ask the user first. ## Pick your path | Situation | Action | |---|---| | Single URL | `bdata scrape -f markdown` | | Small list (≤ ~20 URLs) | shell loop, 1 at a time (see `references/patterns.md`) | | Larger list (dozens+) | `xargs -P 4` with parallelism cap (see `references/patterns.md`) | | Paginated listing | scrape page 1 → extract next-page URL → append → repeat (see `references/examples.md`) | | JS-heavy / login-gated / interaction-required | escalate to `bdata browser` (see `brightdata-cli` skill) | | Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, … | **stop — hand off to `data-feeds`** | | No URL yet, just a topic | **hand off to `search`** | ## Action Core commands: ```bash # Clean markdown (default) bdata scrape "https://example.com/article" -f markdown -o article.md # Raw HTML (when you need the DOM) bdata scrape "https://example.com" -f html -o page.html # Structured JSON (when the Unlocker returns parsed fields) bdata scrape "https://example.com" -f json --pretty -o page.json # Visual snapshot (saves PNG) bdata scrape "https://example.com" -f screenshot -o page.png # Geo-targeted (override the exit country) bdata scrape "https://example.com" --country de -f markdown ``` Full flag reference: [`references/flags.md`](references/flags.md). ## Verification gate (run before claiming success) 1. **Non-empty output:** `test -s "$out_path"` — or, for stdout, at least 200 bytes of content. 2. **Not a block page** — grep the output for any of these signatures (case-insensitive): - `Access Denied` - `Just a moment` - `Attention Required` - `Checking your browser` - `captcha` - `cf-browser-verification` - `cloudflare` *(with < 2KB total body)* 3. **Expected markers present** for the task: e.g., a product page should contain a price pattern (`\$\d`); an article should contain at least one `

` or `# ` heading. 4. **On failure, escalation ladder:** - Retry with a different `--country` (e.g., `--country de` if the origin site is US) - Escalate to `bdata browser` for full JS rendering (hand off to `brightdata-cli` skill) Do not report success until all checks above pass. ## Red flags - Claiming success without inspecting the output. - Silencing errors with `2>/dev/null` — you'll miss auth failures and rate-limit errors. - Running `bdata scrape` on Amazon/LinkedIn/TikTok/Instagram/YouTube/Reddit URLs — these are supported by `data-feeds` and return structured data directly. Scraping loses the structure. - Scraping the same URL repeatedly in the same task — cache the first result. - Looping `bdata scrape` sequentially for large lists instead of using `xargs -P 4` (or similar) with a parallelism cap. - Using `curl` against `api.brightdata.com` directly — legacy path; only when the CLI isn't available. ## References - [`references/flags.md`](references/flags.md) — every flag with when-to-use notes. - [`references/patterns.md`](references/patterns.md) — shell-loop batching, `xargs` parallelism, pagination recipe, retry/backoff, block-page recovery chain, legacy `curl` fallback. - [`references/examples.md`](references/examples.md) — (1) single page → markdown, (2) batch a list of URLs with parallelism cap, (3) paginated listing, (4) block-page recovery.

相关技能

媒体内容

Nano Banana Pro

以纳米·香蕉Pro(Gemini 3 Pro Image)来生成/编辑图像. 用于创建/修改请求,包括编辑。 支持文本到图像+图像到图像; 1K/2K/4K; 使用 -- input-image.

媒体内容

Youtube Watcher

从YouTube视频中获取并读取文字记录. 需要汇总视频时使用,回答有关视频内容的问题,或者从中提取信息.

媒体内容

Atxp

Access ATXP支付API工具用于网页搜索,AI图像生成,音乐创建,视频生成,X/Twitter搜索,电子邮件,代理账户管理. 当用户需要实时网络搜索时使用,AI生成的媒体(图像,音乐,视频),X/Twitter搜索,发送/接收电子邮件,或创建和资助代理账户. 需要通过“ …

媒体内容

Youtube

YouTube Data API与管理的OAuth的集成. 搜索视频,管理播放列表,访问频道数据,并与评论互动. 用户想与YouTube互动时使用此技能. 对于其他第三方应用,使用api-gateway技能(https://clawhub.ai/byungkyu/api-gate…