Social Department Builds QuickSearch for Culture-Aware Video Discovery
Summary:Social Department, working with TrackIt, AWS, and TwelveLabs, has built QuickSearch, a production retrieval system that turns editorial judgments into four single-action searches: Shareable Scenes, Quotable Quotes, Memeable Moments, and Trend Align. Its architecture separates asynchronous video understanding from continuously changing trend acquisition, then joins both at request time. The retrieval stack combines six k-NN channels with seven BM25 channels in Amazon OpenSearch Serverless, appli
What Actually Shipped
QuickSearch is not a new foundation model. It is a production retrieval system that compiles editorial judgments into repeatable search strategies. Social Department owns the pipeline architecture, retrieval logic, and editorial criteria; TrackIt implemented the solution, TwelveLabs supplied the video-understanding models, and AWS provided Bedrock, OpenSearch, and the surrounding cloud infrastructure.
The system exposes four single-action searches. Shareable Scenes retrieves substantial, self-contained dramatic sequences. Quotable Quotes identifies concise lines suitable for reuse. Memeable Moments evaluates and extracts a high-resolution frame from a winning clip rather than accepting a generic thumbnail. Trend Align translates live cultural topics into queries that can be matched against an established media library.
Architecture
1. Asynchronous content understanding
When a video reaches Amazon S3, an S3 Object Created event is routed through EventBridge to a Step Functions state machine. Roughly two dozen Python Lambda functions then perform metadata resolution, transcoding, shot segmentation, embedding generation, scene description, enrichment, and indexing.
Elemental MediaConvert first produces a 480p H.264 asset with adaptive deinterlacing. Amazon Rekognition detects raw shots asynchronously, after which a Lambda function merges and splits them into content-aware ranges of approximately 10 to 20 seconds. These ranges preserve scene boundaries, while embeddings inside each range are sampled at a fixed interval.
TwelveLabs Marengo 3.0 produces multimodal embeddings spanning visual, audio, and transcript features through asynchronous Amazon Bedrock invocation. Pegasus 1.2 runs in a parallel Distributed Map and generates scene narratives, emotions, themes, dialogue types, and narrative position. Because Pegasus accepts pre-cut media rather than time ranges, another Lambda function extracts each segment with several seconds of padding on either side to retain scene transitions.
Transcription, on-screen text recognition, visual labels, and face collections run alongside the embedding stage. Claude models extract character maps, show-level concepts, and character roles. The resulting embeddings, transcripts, OCR fields, scene descriptions, and labels are indexed in Amazon OpenSearch Serverless.
The workflow is designed for long-running, recoverable jobs. Lambda tasks retry with exponential backoff, while the Pegasus stage adds throttling-aware retries with full jitter. Non-critical enrichment degrades gracefully, and hard failures are routed to an error handler that marks the asset as failed.
2. Trend acquisition is a separate lifecycle
A media library changes when new content arrives, whereas cultural trends can move within hours. QuickSearch therefore does not reprocess the library whenever a trend changes. A scheduled EventBridge cron collects routine trends once per region, while the Heat Seeker fetches early-stage, momentum-ordered trends only when explicitly requested and reuses stored results when their TTL has not expired.
Worldwide, US, UK, and Australian feeds are discovered as separate populations rather than derived by filtering one global list. Claude Haiku maps raw trends to an internal category schema, normalizes descriptions, deduplicates near-matches, and limits the number retained per category. An editorial gate then removes trends centered on death, violence, self-harm, natural disasters, politics, protests, crime, or public-health crises. Rejected trends retain fingerprints so subsequent refreshes do not repeatedly screen them.
Content understanding and trend acquisition run on separate schedules and are joined only when a user selects Trend Align. This avoids recomputing media embeddings while allowing rapidly changing cultural signals to remain current.
3. One user action expands into staged retrieval
The React application is served from S3 through CloudFront. Its Fastify API runs on Lambda behind API Gateway. A QuickSearch request is recorded in DynamoDB, queued in SQS, and processed asynchronously by a worker Lambda. Transient failures use SQS retries, while non-retriable errors go directly to a dead-letter queue.
Each action expands into a set of queries shaped by its editorial criteria. Trend Align is the most involved case because a cultural topic must be translated into concepts that actually appear in video imagery and audio; Claude performs that translation. Queries are then routed across six k-NN channels—visual, audio, transcript, on-screen text, scene description, and labels—and seven BM25 text channels. Modality weights vary by query type, allowing dialogue searches to emphasize transcripts and emotional features while visual queries suppress them.
k-NN similarity scores and BM25 scores are not directly comparable, so the system combines them with Reciprocal Rank Fusion:
RRF(d) = Σc 1 / (k + rc(d))
Here, rc(d) is document d's rank in channel c, and k is a smoothing constant. RRF depends on rank position rather than raw score magnitude, reducing scale mismatch between dense and lexical retrieval. The fused candidates pass through the Amazon Rerank cross-encoder and a Claude validation stage before an action-specific Claude model ranks them against the editorial criteria. Sonnet is used where cultural recall can change the answer, while Haiku handles bounded, cost-sensitive tasks.
Adjacent clips with highly similar visual embeddings are merged into coherent moments of roughly 20 to 60 seconds. For Memeable Moments, a dedicated Lambda function extracts several candidate frames, evaluates face quality with Rekognition, checks image sharpness, and re-extracts the winner at full resolution before removing letterboxing and returning a signed CloudFront URL.
4. Versioned caching and safety controls
Completed results are cached in DynamoDB for 30 days. The cache key includes the action's pipeline version and, for Trend Align, a hash of the active trend set. Model upgrades, editorial changes, or refreshed trends can therefore invalidate only the affected results.
The production boundary is explicitly human-in-the-loop. Code checks generated material against its source; a quote absent from the episode transcript is discarded. Bedrock Guardrails screens customer search text for prompt attacks, and customer personnel approve every post before publication. AWS states that customer data remains inside its data boundary, is not shared with model providers, and is not used to train the underlying models.
Engineering Trade-offs
Temporal granularity versus compute. Ranges of 10 to 20 seconds follow scene structure, but fixed-interval embeddings can miss meme-worthy frames that appear only briefly. A 480p transcode reduces processing and storage costs while potentially losing fine text, subtle expressions, or details in low-quality source material. Pegasus also requires pre-cut files, padding, and additional temporary I/O.
Recall versus model cost. Six dense-retrieval channels and seven BM25 channels broaden candidate coverage, but they increase index size, query fan-out, and latency. The system avoids placing a large model in front of the entire library: inexpensive vector and lexical retrieval first builds a candidate pool, then expensive reranking and Claude calls operate on a bounded set. Per-query Bedrock cost can be approximated as one query embedding, one rerank, and a small number of Claude invocations, but the source provides no token counts, OpenSearch consumption, or unit prices.
Freshness versus regional complexity. Scheduled collection offers predictable cost, while demand-driven Heat Seeker avoids paying for unused fetches. Discovering four regional feeds separately avoids treating a global trend as a regional label, but multiplies the number of schedules, sources, and normalization rules that must be maintained.
Managed services versus platform coupling. Bedrock removes model-serving and scaling work and exposes several providers through a common API. The tradeoff is tighter regional coupling. Marengo's asynchronous path requires S3 input and output to reside in the same Region as the endpoint, so the project stages videos in US East N. Virginia before invocation and reads the embeddings back afterward. This preserves the required data locality but adds cross-Region transfer, compliance review, and recovery complexity.
Cache savings versus consistency. A 30-day cache can sharply reduce repeated query cost, but pipeline versions must change whenever editorial rules change. Trend Align must also include the trend-set hash in its key. A missing dependency can silently serve stale cultural judgments long after the underlying model or criteria has changed.
Production Evidence and Its Limits
Social Department reports that discovery work previously taking three to four days now takes about ten minutes. The project reportedly moved from specification to a live NAB demo and production delivery in eight weeks. These are useful workflow signals, but they are not controlled benchmark results.
The material does not disclose Precision, Recall@K, or nDCG for the four actions; candidate-pool sizes; false-selection rates across genres or cultures; human acceptance coverage; p50 or p95 latency; ingest cost per video-hour; Bedrock spend per query; cache hit rate; or false-rejection rates for sensitive trends. The ten-minute figure should therefore be treated as a reported operational outcome, not proof of superior retrieval quality.
Deployment Guidance for Developers
- Define executable editorial criteria first. Establish positive and negative examples, clip-duration constraints, and human adjudication rules for every action. Evaluate them against per-title ground truth instead of relying on intuition or prompt wording.
- Separate stable assets from decaying signals. Update media vectors, transcripts, and character relationships incrementally. Give trends and popularity feeds separate schedules, versions, and tables, and join them only at request time.
- Separate cheap recall from expensive judgment. Use dense and lexical retrieval to build a high-recall candidate pool, combine them with RRF, and restrict rerankers and LLMs to a bounded candidate set. Size that pool against the ranker's output-token limit to prevent silent truncation.
- Retain deterministic checks and human approval. Validate quotations against transcripts, preserve source timecodes for frames, and require approval before publication. Guardrails are an input-side control, not a substitute for provenance and access control.
- Evaluate total cost, not model price alone. Include transcoding, media movement, OpenSearch, Lambda execution, queue time, regional traffic, and cache storage. Track quality, human coverage, tail latency, trend false rejections, and cache-invalidation correctness alongside latency and spend.
— END —Source: AWS Machine Learning Blog (2026-11-18)
Tags: #QuickSearch #Multimodal Video Retrieval #Amazon Bedrock #Hybrid Retrieval #Cultural Context
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments