Open Vision Banana Fully Open-Sources a Generator-Based Perception Recipe
Summary:Open Vision Banana (OVB) turns pretrained image generators into unified visual perception systems by recasting semantic segmentation, depth estimation, referring segmentation, and instance segmentation as conditional image editing. The project releases its dataset construction and processing pipeline, training and evaluation code, and final, intermediate, and ablation checkpoints. Built on 4B- and 9B-parameter open-weight base models with 2.1 million to 3.8 million public training pairs, OVB re
Core Idea: Perception as Conditional Image Editing
Open Vision Banana (OVB) does not introduce a separate architecture for every perception task. Instead, it reuses the real-world knowledge already encoded in pretrained image generators. Given an input image, a natural-language instruction, and task conditions, the model edits or repaints the image to produce semantic maps, depth maps, or instance-level outputs.
This is not simply a diffusion model attached to a segmentation head. The generator must convert pixel manipulation into controllable structured output while retaining much of its pretrained generative capability. OVB's main contribution is therefore a complete engineering recipe spanning data construction, parameter adaptation, evaluation, and test-time scaling.
Training Recipe and Task Unification
OVB uses 4B- and 9B-parameter open-weight base models and constructs roughly 2.1 million to 3.8 million training pairs from public datasets. Semantic segmentation, depth estimation, referring segmentation, and instance segmentation are trained through a unified conditional-generation interface.
The paper stresses that color encoding, prompt design, parameter adaptation, and task balancing interact. A color mapping that helps in isolation may become harmful under a different prompt, while stronger adaptation does not necessarily improve cross-task generalization. Selecting a configuration by combining isolated ablation results can therefore produce a system that is worse than its individual components suggest.
Parameter-efficient adaptation such as LoRA is a natural choice when the base model's editing ability must be preserved. Updating only injected adapter parameters reduces optimizer-state memory and limits catastrophic drift in pretrained generative priors. However, the abstract does not disclose the exact adapter design, rank, training schedule, or GPU cost, so full reproduction still depends on the released code and checkpoints.
Zero-Shot Segmentation and Depth Estimation
OVB reaches 73.9 mIoU on zero-shot Cityscapes semantic segmentation, exceeding Vision Banana's reported result of 69.9 by four points. This suggests that public training data, a complete implementation, and systematic ablations can materially affect the effectiveness of generator-based perception rather than merely reproduce an undisclosed recipe.
For depth estimation, OVB rivals leading affine-invariant depth estimators. Affine invariance is useful when exact camera calibration or metric depth scale is unavailable, but matching a leading category does not mean outperforming it across every metric and operating condition. The abstract does not provide complete error tables, inference resolutions, or speed comparisons.
OVB's non-zero-shot variants remain within five points of Vision Banana on referring and reasoning segmentation, but fall further behind on instance segmentation. The unified conditional-generation interface appears effective for semantic structure and text grounding, while instance-level object separation exposes a deeper limitation that cannot be solved by class colors alone.
Legend Ensembling: Correcting Color Bindings at Test Time
OVB maps classes to colors and reconstructs class masks from generated outputs. The generator does not always bind colors to classes consistently, however, and a single incorrect color assignment can affect the entire prediction.
Legend ensembling generates multiple predictions for the same image under different class-to-color mappings. It converts each output into a class mask and performs majority voting. Variation across legends acts as a perturbation to unstable bindings, while predictions that survive across mappings are more likely to be semantically valid.
The method reportedly improves mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining. The tradeoff is additional inference: each image must be generated and post-processed multiple times. Latency, peak memory, and throughput will therefore tend to scale with the number of legends. Real-time or high-volume deployments should evaluate smaller ensembles, batching, and opportunities to share base-model computation.
How to Interpret the Results
OVB's strongest features are its open data pipeline, code, final and intermediate checkpoints, reported zero-shot Cityscapes improvement, and retraining-free legend ensembling. It also largely preserves the base model's image-editing ability.
The claims still require caution. Cityscapes is a widely used benchmark, and performance can benefit from dataset co-occurrence and prompt adaptation. Saying the system rivals state-of-the-art affine-invariant depth estimators does not establish leadership on every depth metric. The abstract also omits independent reproduction, training compute, inference latency, peak memory, and hardware-normalized throughput. The instance-segmentation gap further shows that a unified generative framework cannot yet replace task-specific optimization for instance-level reasoning.
Practical Deployment Guidance
- Evaluate each task independently: Measure semantic segmentation, depth, referring segmentation, and instance segmentation separately rather than relying on one averaged perception score.
- Establish a single-legend baseline first: Record throughput, latency, and memory before adding ensemble members, then build an accuracy-versus-cost Pareto curve.
- Check generative capability retention: Test editing instruction following, image consistency, and general generation quality alongside perception metrics.
- Analyze binding errors by class: Break failures down by category, frequency, and visual similarity to determine whether voting removes random binding noise or conceals systematic semantic errors.
- Audit licenses and dataset overlap: Verify permissions for every public-data source, apply deduplication, and check for near-duplicates between training and evaluation sets.
OVB does not prove that image generators should replace every vision model. Its value is that it provides a reproducible and inspectable path from generative image models to unified perception, with useful evidence for developers studying open-weight adaptation, generative vision, and test-time computation.
— END —Source: Hugging Face Daily Papers (2026-11-11)
Tags: #Open Vision Banana #Generative Vision #Image Generators #Visual Perception #Test-Time Scaling
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments