做什么
数据管道和ETL自动化 -- -- 数据集成和分析的提取、转换、加载工作流程
技能库 智客分类:数据与分析 data-pipeline
数据管道和ETL自动化 -- -- 数据集成和分析的提取、转换、加载工作流程
官方网址:skills.sh
先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。
数据管道和ETL自动化 -- -- 数据集成和分析的提取、转换、加载工作流程
官方 description 未单独写出 Use when。按规范,代理会在用户任务与这段 description 的关键词匹配时激活本技能。
按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Data Pipeline、Overview、ETL Patterns、Basic ETL Flow、n8n ETL Workflow、Data Sources。 其中含规范建议的小节:分步指令、输入输出示例。
文件分析:这是一份仅含 SKILL.md 的指令型技能,代理激活后整份正文进入上下文。
Data pipeline and ETL automation - extract, transform, load workflows for data integration and analytics
Data PipelineOverviewETL PatternsBasic ETL Flown8n ETL WorkflowData SourcesCommon ExtractorsTransformationsCommon TransformationsCode Transform ExamplesData DestinationsCommon Loaders
· 许可:MIT
来源分类:skills.sh agent-skill
namedata-pipelinedescription具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗
先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。
复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。
把 Agent Skill「data-pipeline」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-414c2bf7d156427a-%E6%95%B0%E6%8D%AE%E7%AE%A1%E9%81%93.html 请存为 .cursor/skills/data-pipeline/SKILL.md 或 .claude/skills/data-pipeline/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。
npx skills add 'https://github.com/claude-office-skills/skills' --list
npx skills add 'https://github.com/claude-office-skills/skills' --skill 'data-pipeline'
CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。
name: data-pipeline description: "Data pipeline and ETL automation - extract, transform, load workflows for data integration and analytics" version: "1.0.0" author: claude-office-skills license: MIT
category: data tags:
models: recommended: - claude-sonnet-4
mcp: server: data-mcp tools: - database_query - api_fetch - spreadsheet_write - file_transform
capabilities:
languages:
related_skills:
Build data pipelines and ETL workflows for data integration, transformation, and analytics automation. Based on n8n's data workflow templates.
This skill covers:
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ EXTRACT │───▶│ TRANSFORM │───▶│ LOAD │
│ │ │ │ │ │
│ • APIs │ │ • Clean │ │ • Database │
│ • Databases │ │ • Map │ │ • Warehouse │
│ • Files │ │ • Aggregate │ │ • Files │
│ • Webhooks │ │ • Enrich │ │ • APIs │
└─────────────┘ └─────────────┘ └─────────────┘
workflow: "Daily Sales ETL"
schedule: "2am daily"
nodes:
# EXTRACT
- name: "Extract from Shopify"
type: shopify
action: get_orders
filter: created_at >= yesterday
- name: "Extract from Stripe"
type: stripe
action: get_payments
filter: created >= yesterday
# TRANSFORM
- name: "Merge Data"
type: merge
mode: combine_by_key
key: order_id
- name: "Transform"
type: code
code: |
return items.map(item => ({
date: item.created_at.split('T')[0],
order_id: item.id,
customer_email: item.email,
total: parseFloat(item.total_price),
currency: item.currency,
items: item.line_items.length,
source: item.source_name,
payment_status: item.payment.status
}));
# LOAD
- name: "Load to BigQuery"
type: google_bigquery
action: insert_rows
table: sales_daily
- name: "Update Google Sheets"
type: google_sheets
action: append_rows
spreadsheet: "Daily Sales Report"
extractors:
databases:
- postgresql:
connection: connection_string
query: "SELECT * FROM orders WHERE date >= $1"
- mysql:
connection: connection_string
query: custom_sql
- mongodb:
connection: connection_string
collection: orders
filter: {date: {$gte: yesterday}}
apis:
- rest_api:
url: "https://api.example.com/data"
method: GET
headers: {Authorization: "Bearer {token}"}
pagination: handle_automatically
- graphql:
url: "https://api.example.com/graphql"
query: graphql_query
files:
- csv:
source: sftp/s3/google_drive
delimiter: ","
encoding: utf-8
- excel:
source: file_path
sheet: "Sheet1"
- json:
source: api/file
path: "data.items"
saas:
- salesforce: get_objects
- hubspot: get_contacts/deals
- stripe: get_charges
- shopify: get_orders
transformations:
cleaning:
- remove_nulls: drop_or_fill
- trim_whitespace: all_string_fields
- deduplicate: by_key
- validate: against_schema
mapping:
- rename_fields: {old_name: new_name}
- convert_types: {date_string: date}
- map_values: {status_code: status_name}
aggregation:
- group_by: [date, category]
- sum: [revenue, quantity]
- count: orders
- average: order_value
enrichment:
- lookup: from_reference_table
- geocode: from_address
- calculate: derived_fields
filtering:
- where: condition
- limit: n_rows
- sample: percentage
// Clean and normalize data
function transform(items) {
return items.map(item => ({
// Clean strings
name: item.name?.trim().toLowerCase(),
// Parse dates
date: new Date(item.created_at).toISOString().split('T')[0],
// Convert types
amount: parseFloat(item.amount) || 0,
// Map values
status: statusMap[item.status_code] || 'unknown',
// Calculate fields
total: item.quantity * item.unit_price,
// Filter nested
tags: item.tags?.filter(t => t.active).map(t => t.name),
// Default values
source: item.source || 'direct'
}));
}
// Aggregate data
function aggregate(items) {
const grouped = {};
items.forEach(item => {
const key = `${item.date}_${item.category}`;
if (!grouped[key]) {
grouped[key] = {
date: item.date,
category: item.category,
total_revenue: 0,
order_count: 0
};
}
grouped[key].total_revenue += item.amount;
grouped[key].order_count += 1;
});
return Object.values(grouped);
}
loaders:
data_warehouses:
- bigquery:
project: project_id
dataset: analytics
table: sales
write_mode: append/truncate
- snowflake:
account: account_id
warehouse: compute_wh
database: analytics
schema: public
- redshift:
cluster: cluster_id
database: analytics
databases:
- postgresql:
upsert: on_conflict_update
- mysql:
batch_insert: 1000_rows
files:
- s3:
bucket: data-lake
path: /processed/{date}/
format: parquet
- google_cloud_storage:
bucket: data-bucket
spreadsheets:
- google_sheets:
mode: append/overwrite
- airtable:
base: base_id
table: table_name
apis:
- webhook:
url: destination_url
batch_size: 100
scheduling:
patterns:
hourly:
cron: "0 * * * *"
use_for: real_time_dashboards
daily:
cron: "0 2 * * *"
use_for: daily_reports
weekly:
cron: "0 3 * * 1"
use_for: weekly_summaries
on_demand:
trigger: webhook/manual
use_for: ad_hoc_analysis
dependencies:
- pipeline_a: must_complete_before pipeline_b
- wait_for: all_extracts_complete
retries:
max_attempts: 3
delay: exponential_backoff
alert_on: final_failure
monitoring:
metrics:
- rows_processed
- execution_time
- error_count
- data_freshness
alerts:
pipeline_failed:
channels: [slack, pagerduty]
template: |
🚨 *Pipeline Failed*
Pipeline: {pipeline_name}
Stage: {failed_stage}
Error: {error_message}
[View Logs]({logs_url})
data_quality:
trigger: anomaly_detected
conditions:
- row_count: differs_by > 50%
- null_rate: exceeds_threshold
- schema: changed_unexpectedly
stale_data:
trigger: last_update > threshold
threshold: 2_hours
data_quality:
schema_validation:
- required_fields: [id, date, amount]
- field_types:
id: integer
date: date
amount: number
- allowed_values:
status: [active, pending, closed]
statistical_checks:
- null_rate: < 5%
- duplicate_rate: < 1%
- value_range:
amount: [0, 1000000]
business_rules:
- total_equals_sum_of_line_items
- dates_are_not_in_future
- email_format_valid
trend_analysis:
- row_count: within_2_std_of_mean
- total_value: within_expected_range
Request: "Create a daily sales data pipeline"
Output:
# Daily Sales Data Pipeline
## Pipeline Overview
Shopify + Stripe → Transform → BigQuery + Sheets
## Schedule
- Runs: 2am daily
- Timezone: UTC
- Retry: 3 attempts
## Extract
### Shopify Orders
```yaml
source: shopify
filter: created_at >= yesterday
fields: [id, email, total_price, line_items, created_at]
source: stripe
filter: created >= yesterday
fields: [id, amount, status, metadata.order_id]
// Join and clean data
{
date: order.created_at.split('T')[0],
order_id: order.id,
customer: order.email,
revenue: parseFloat(order.total_price),
items: order.line_items.length,
payment_status: payment.status
}
analytics.sales_daily
---
*Data Pipeline Skill - Part of Claude Office Skills*
数据与分析
TL; DR:23个营销剧本(CRO,SEO,拷贝,分析,实验,定价,发布,广告,社交). 用于快速获取清单+副本/粘贴可交付品.
数据与分析
免费等级不需要 API 关键值 。 专业级别的密码货币和股票市场数据集成,用于实时价格、公司概况和全球分析。 由Node.js提供动力,外部依赖性为零.
数据与分析
Azure存储服务包括Blob存储,文件共享,等式存储,表存储,和数据湖. 解答关于存储访问级别(热,凉,冷,存档),何时使用每个级别,以及级别比较的问题. 提供对象存储,SMB文件共享,async消息,NoSQL密钥-值,和大数据分析. 包括生命周期管理。 USE FOR: b…
数据与分析
Azure Data Explorer(Kusto/ADX)中使用KQL进行日志分析,遥测和时间序列分析的查询并分析数据. When: KQL 查询, Kusto 数据库查询, Azure Data Explorer, ADX 集群, 日志分析, 时间序列数据, IoT 遥测, …