做什么
使用 TRIBE 软件 v2, Meta 预测 fMRI 大脑对视频,音频和文本刺激的反应的多模式基础模型
技能库 智客分类:媒体内容 tribev2-brain-encoding
使用 TRIBE 软件 v2, Meta 预测 fMRI 大脑对视频,音频和文本刺激的反应的多模式基础模型
官方网址:skills.sh
先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。
使用 TRIBE 软件 v2, Meta 预测 fMRI 大脑对视频,音频和文本刺激的反应的多模式基础模型
官方 description 未单独写出 Use when。按规范,代理会在用户任务与这段 description 的关键词匹配时激活本技能。
按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:TRIBE v2 Brain Encoding Model、Installation、Inference only、With brain visualization (PyVista & Nilearn)、Full training dependencies (PyTorch Lightning, W&B, etc.)、Quick Start — Inference。
文件分析:这是一份仅含 SKILL.md 的指令型技能,代理激活后整份正文进入上下文。
Use TRIBE v2, Meta's multimodal foundation model for predicting fMRI brain responses to video, audio, and text stimuli
TRIBE v2 Brain Encoding ModelInstallationInference onlyWith brain visualization (PyVista & Nilearn)Full training dependencies (PyTorch Lightning, W&B, etc.)Quick Start — InferenceLoad pretrained model and predict from videoLoad from HuggingFace (downloads weights to cache)Build events dataframe from a video filePredict brain responsesMultimodal input — video + audio + textAll modalities together (text is auto-converted to speech and transcribed)
来源分类:skills.sh agent-skill
nametribev2-brain-encodingdescription具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗
先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。
复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。
把 Agent Skill「tribev2-brain-encoding」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-497c6e2a1cd587d8-Tribev2-%E8%84%91%E7%BC%96%E7%A0%81.html 请存为 .cursor/skills/tribev2-brain-encoding/SKILL.md 或 .claude/skills/tribev2-brain-encoding/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。
npx skills add 'https://github.com/reason-machines/trending-skills' --list
npx skills add 'https://github.com/reason-machines/trending-skills' --skill 'tribev2-brain-encoding'
CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。
name: tribev2-brain-encoding description: Use TRIBE v2, Meta's multimodal foundation model for predicting fMRI brain responses to video, audio, and text stimuli triggers:
Skill by ara.so — Daily 2026 Skills collection
TRIBE v2 is Meta's multimodal foundation model that predicts fMRI brain responses to naturalistic stimuli (video, audio, text). It combines LLaMA 3.2 (text), V-JEPA2 (video), and Wav2Vec-BERT (audio) encoders into a unified Transformer architecture that maps multimodal representations onto the cortical surface (fsaverage5, ~20k vertices).
# Inference only
pip install -e .
# With brain visualization (PyVista & Nilearn)
pip install -e ".[plotting]"
# Full training dependencies (PyTorch Lightning, W&B, etc.)
pip install -e ".[training]"
from tribev2 import TribeModel
# Load from HuggingFace (downloads weights to cache)
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
# Build events dataframe from a video file
df = model.get_events_dataframe(video_path="path/to/video.mp4")
# Predict brain responses
preds, segments = model.predict(events=df)
print(preds.shape) # (n_timesteps, n_vertices) on fsaverage5
from tribev2 import TribeModel
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
# All modalities together (text is auto-converted to speech and transcribed)
df = model.get_events_dataframe(
video_path="path/to/video.mp4",
audio_path="path/to/audio.wav", # optional, overrides video audio
text_path="path/to/script.txt", # optional, auto-timed
)
preds, segments = model.predict(events=df)
print(preds.shape) # (n_timesteps, n_vertices)
from tribev2 import TribeModel
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = model.get_events_dataframe(text_path="path/to/narration.txt")
preds, segments = model.predict(events=df)
from tribev2 import TribeModel
from tribev2.plotting import plot_brain_surface
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = model.get_events_dataframe(video_path="path/to/video.mp4")
preds, segments = model.predict(events=df)
# Plot a single timepoint on the cortical surface
plot_brain_surface(preds[0], backend="nilearn") # or backend="pyvista"
export DATAPATH="/path/to/studies"
export SAVEPATH="/path/to/output"
export SLURM_PARTITION="your_slurm_partition"
huggingface-cli login
# Paste a HuggingFace read token when prompted
# Request access at: https://huggingface.co/meta-llama/Llama-3.2-3B
python -m tribev2.grids.test_run
# Cortical surface model
python -m tribev2.grids.run_cortical
# Subcortical regions
python -m tribev2.grids.run_subcortical
from tribev2 import TribeModel
# Load pretrained weights
model = TribeModel.from_pretrained(
"facebook/tribev2",
cache_folder="./cache" # local cache for HuggingFace weights
)
# Build events dataframe (word-level timings, chunking, etc.)
df = model.get_events_dataframe(
video_path=None, # str path to .mp4
audio_path=None, # str path to .wav
text_path=None, # str path to .txt
)
# Run prediction
preds, segments = model.predict(events=df)
# preds: np.ndarray of shape (n_timesteps, n_vertices)
# segments: list of segment metadata dicts
tribev2/
├── main.py # Experiment pipeline: Data, TribeExperiment
├── model.py # FmriEncoder: Transformer multimodal→fMRI model
├── pl_module.py # PyTorch Lightning training module
├── demo_utils.py # TribeModel and inference helpers
├── eventstransforms.py # Event transforms (word extraction, chunking)
├── utils.py # Multi-study loading, splitting, subject weighting
├── utils_fmri.py # Surface projection (MNI / fsaverage) and ROI analysis
├── grids/
│ ├── defaults.py # Full default experiment configuration
│ └── test_run.py # Quick local test entry point
├── plotting/ # Brain visualization backends
└── studies/ # Dataset definitions (Algonauts2025, Lahner2024, …)
Edit tribev2/grids/defaults.py or set environment variables:
# tribev2/grids/defaults.py (key fields)
{
"datapath": "/path/to/studies", # override with DATAPATH env var
"savepath": "/path/to/output", # override with SAVEPATH env var
"slurm_partition": "learnfair", # override with SLURM_PARTITION env var
"model": "FmriEncoder",
"modalities": ["video", "audio", "text"],
"surface": "fsaverage5", # ~20k vertices
}
from tribev2.main import Data, TribeExperiment
from tribev2.pl_module import TribePLModule
import pytorch_lightning as pl
# Configure experiment
experiment = TribeExperiment(
datapath="/path/to/studies",
savepath="/path/to/output",
modalities=["video", "audio", "text"],
)
data = Data(experiment)
module = TribePLModule(experiment)
trainer = pl.Trainer(
max_epochs=50,
accelerator="gpu",
devices=4,
)
trainer.fit(module, data)
from tribev2.utils_fmri import project_to_fsaverage, get_roi_mask
# Project MNI coordinates to fsaverage5 surface
surface_data = project_to_fsaverage(mni_data, target="fsaverage5")
# Get a specific ROI mask (e.g., early visual cortex)
roi_mask = get_roi_mask(roi_name="V1", surface="fsaverage5")
v1_responses = preds[:, roi_mask]
print(v1_responses.shape) # (n_timesteps, n_v1_vertices)
from tribev2 import TribeModel
import numpy as np
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
video_paths = ["video1.mp4", "video2.mp4", "video3.mp4"]
all_predictions = []
for vp in video_paths:
df = model.get_events_dataframe(video_path=vp)
preds, segments = model.predict(events=df)
all_predictions.append(preds)
# all_predictions: list of (n_timesteps_i, n_vertices) arrays
from tribev2 import TribeModel
from tribev2.utils_fmri import get_roi_mask
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = model.get_events_dataframe(video_path="video.mp4")
preds, segments = model.predict(events=df)
# Focus on auditory cortex
ac_mask = get_roi_mask("auditory_cortex", surface="fsaverage5")
auditory_responses = preds[:, ac_mask] # (n_timesteps, n_ac_vertices)
preds, segments = model.predict(events=df)
for i, seg in enumerate(segments):
print(f"Segment {i}: onset={seg['onset']:.2f}s, duration={seg['duration']:.2f}s")
print(f" Brain response shape: {preds[i].shape}")
LLaMA 3.2 access denied
# Must request access at https://huggingface.co/meta-llama/Llama-3.2-3B
# Then authenticate:
huggingface-cli login
# Use a HuggingFace token with read permissions
CUDA out of memory during inference
# Use CPU for inference on smaller machines
import torch
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
model.to("cpu")
Missing visualization dependencies
pip install -e ".[plotting]"
# Installs pyvista and nilearn backends
Slurm training not submitting
# Check env vars are set
echo $DATAPATH $SAVEPATH $SLURM_PARTITION
# Or edit tribev2/grids/defaults.py directly
Video without audio track causes error
# Provide audio separately or use text-only mode
df = model.get_events_dataframe(
video_path="silent_video.mp4",
audio_path="separate_audio.wav",
)
@article{dAscoli2026TribeV2,
title={A foundation model of vision, audition, and language for in-silico neuroscience},
author={d'Ascoli, St{\'e}phane and Rapin, J{\'e}r{\'e}my and Benchetrit, Yohann and Brookes, Teon
and Begany, Katelyn and Raugel, Jos{\'e}phine and Banville, Hubert and King, Jean-R{\'e}mi},
year={2026}
}
媒体内容
以纳米·香蕉Pro(Gemini 3 Pro Image)来生成/编辑图像. 用于创建/修改请求,包括编辑。 支持文本到图像+图像到图像; 1K/2K/4K; 使用 -- input-image.
媒体内容
从YouTube视频中获取并读取文字记录. 需要汇总视频时使用,回答有关视频内容的问题,或者从中提取信息.
媒体内容
Access ATXP支付API工具用于网页搜索,AI图像生成,音乐创建,视频生成,X/Twitter搜索,电子邮件,代理账户管理. 当用户需要实时网络搜索时使用,AI生成的媒体(图像,音乐,视频),X/Twitter搜索,发送/接收电子邮件,或创建和资助代理账户. 需要通过“ …
媒体内容
YouTube Data API与管理的OAuth的集成. 搜索视频,管理播放列表,访问频道数据,并与评论互动. 用户想与YouTube互动时使用此技能. 对于其他第三方应用,使用api-gateway技能(https://clawhub.ai/byungkyu/api-gate…