跳到主内容
智客 ZICQ

技能库 智客分类:Agent 工作流 bigquery-pipeline-audit

Bigquery Pipeline Audit

对Python + BigQuery 管道进行成本安全,一能,生产准备的审计. 返回带有精确补丁位置的结构化报告 .

8832 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

对Python + BigQuery 管道进行成本安全,一能,生产准备的审计. 返回带有精确补丁位置的结构化报告 .

何时用

官方 description 未单独写出 Use when。按规范,代理会在用户任务与这段 description 的关键词匹配时激活本技能。

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:BigQuery Pipeline Audit: Cost, Safety and Production Readiness、A) COST EXPOSURE: What will actually get billed?、B) DRY RUN AND EXECUTION MODES、C) BACKFILL AND LOOP DESIGN、D) QUERY SAFETY AND SCAN SIZE、E) SAFE WRITES AND IDEMPOTENCY。

文件分析

文件分析:这是一份仅含 SKILL.md 的指令型技能,代理激活后整份正文进入上下文。

官方 description(原文)

Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.

BigQuery Pipeline Audit: Cost, Safety and Production ReadinessA) COST EXPOSURE: What will actually get billed?B) DRY RUN AND EXECUTION MODESC) BACKFILL AND LOOP DESIGND) QUERY SAFETY AND SCAN SIZEE) SAFE WRITES AND IDEMPOTENCYF) OBSERVABILITY: Can you debug a failure?Final

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
bigquery-pipeline-audit
description
Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「bigquery-pipeline-audit」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-67134143602c0fa9-Bigquery-Pipeline-Audit.html
请存为 .cursor/skills/bigquery-pipeline-audit/SKILL.md 或 .claude/skills/bigquery-pipeline-audit/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/github/awesome-copilot' --list

npx skills add 'https://github.com/github/awesome-copilot' --skill 'bigquery-pipeline-audit'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: bigquery-pipeline-audit description: 'Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.' --- # BigQuery Pipeline Audit: Cost, Safety and Production Readiness You are a senior data engineer reviewing a Python + BigQuery pipeline script. Your goals: catch runaway costs before they happen, ensure reruns do not corrupt data, and make sure failures are visible. Analyze the codebase and respond in the structure below (A to F + Final). Reference exact function names and line locations. Suggest minimal fixes, not rewrites. --- ## A) COST EXPOSURE: What will actually get billed? Locate every BigQuery job trigger (`client.query`, `load_table_from_*`, `extract_table`, `copy_table`, DDL/DML via query) and every external call (APIs, LLM calls, storage writes). For each, answer: - Is this inside a loop, retry block, or async gather? - What is the realistic worst-case call count? - For each `client.query`, is `QueryJobConfig.maximum_bytes_billed` set? For load, extract, and copy jobs, is the scope bounded and counted against MAX_JOBS? - Is the same SQL and params being executed more than once in a single run? Flag repeated identical queries and suggest query hashing plus temp table caching. **Flag immediately if:** - Any BQ query runs once per date or once per entity in a loop - Worst-case BQ job count exceeds 20 - `maximum_bytes_billed` is missing on any `client.query` call --- ## B) DRY RUN AND EXECUTION MODES Verify a `--mode` flag exists with at least `dry_run` and `execute` options. - `dry_run` must print the plan and estimated scope with zero billed BQ execution (BigQuery dry-run estimation via job config is allowed) and zero external API or LLM calls - `execute` requires explicit confirmation for prod (`--env=prod --confirm`) - Prod must not be the default environment If missing, propose a minimal `argparse` patch with safe defaults. --- ## C) BACKFILL AND LOOP DESIGN **Hard fail if:** the script runs one BQ query per date or per entity in a loop. Check that date-range backfills use one of: 1. A single set-based query with `GENERATE_DATE_ARRAY` 2. A staging table loaded with all dates then one join query 3. Explicit chunks with a hard `MAX_CHUNKS` cap Also check: - Is the date range bounded by default (suggest 14 days max without `--override`)? - If the script crashes mid-run, is it safe to re-run without double-writing? - For backdated simulations, verify data is read from time-consistent snapshots (`FOR SYSTEM_TIME AS OF`, partitioned as-of tables, or dated snapshot tables). Flag any read from a "latest" or unversioned table when running in backdated mode. Suggest a concrete rewrite if the current approach is row-by-row. --- ## D) QUERY SAFETY AND SCAN SIZE For each query, check: - **Partition filter** is on the raw column, not `DATE(ts)`, `CAST(...)`, or any function that prevents pruning - **No `SELECT *`**: only columns actually used downstream - **Joins will not explode**: verify join keys are unique or appropriately scoped and flag any potential many-to-many - **Expensive operations** (`REGEXP`, `JSON_EXTRACT`, UDFs) only run after partition filtering, not on full table scans Provide a specific SQL fix for any query that fails these checks. --- ## E) SAFE WRITES AND IDEMPOTENCY Identify every write operation. Flag plain `INSERT`/append with no dedup logic. Each write should use one of: 1. `MERGE` on a deterministic key (e.g., `entity_id + date + model_version`) 2. Write to a staging table scoped to the run, then swap or merge into final 3. Append-only with a dedupe view: `QUALIFY ROW_NUMBER() OVER (PARTITION BY ) = 1` Also check: - Will a re-run create duplicate rows? - Is the write disposition (`WRITE_TRUNCATE` vs `WRITE_APPEND`) intentional and documented? - Is `run_id` being used as part of the merge or dedupe key? If so, flag it. `run_id` should be stored as a metadata column, not as part of the uniqueness key, unless you explicitly want multi-run history. State the recommended approach and the exact dedup key for this codebase. --- ## F) OBSERVABILITY: Can you debug a failure? Verify: - Failures raise exceptions and abort with no silent `except: pass` or warn-only - Each BQ job logs: job ID, bytes processed or billed when available, slot milliseconds, and duration - A run summary is logged or written at the end containing: `run_id, env, mode, date_range, tables written, total BQ jobs, total bytes` - `run_id` is present and consistent across all log lines If `run_id` is missing, propose a one-line fix: `run_id = run_id or datetime.utcnow().strftime('%Y%m%dT%H%M%S')` --- ## Final **1. PASS / FAIL** with specific reasons per section (A to F). **2. Patch list** ordered by risk, referencing exact functions to change. **3. If FAIL: Top 3 cost risks** with a rough worst-case estimate (e.g., "loop over 90 dates x 3 retries = 270 BQ jobs").

相关技能

Agent 工作流

Skill Creator

创造有效技能指南。 当用户想创造出新的技能(或更新现有的技能),以专业知识,工作流程,或工具集成来扩展克洛德的能力时,应该使用这种技能.

Agent 工作流

Clawdhub

使用ClawdHub CLI搜索,安装,更新并发布从taladhub.com的代理技能. 需要获取苍蝇上的新技能时使用,将安装的技能同步到最新版本或特定版本,或者发布 npm-instainddhub CLI 的新/更新的技能文件夹.

Agent 工作流

Agent Team Orchestration

管弦乐团多代理团队,任务设定周期,交接协议,审查工作流程. 使用时间: (1)建立2+特派员队伍,具有不同专业,(2)确定任务路线和生命周期(收录框_ spec_建设_审查_完成),(3)在特派员之间制定交接协议,(4)建立审查和质量关口,(5)管理特派员之间的交流和文物共享.

Agent 工作流

Superpowers

Spec-first,TDD,子代理驱动的软件开发工作流程. 当:(1)构建任何新功能或应用——触发脑暴_计划_子代理执行回路,(2)调试出一个bug或测试失败——触发系统性的根起过程,(3)用户说"让我们构建","帮助我计划","我想添加X",或"这个被打破",(4)完成一个功…