Lumen Cutout Reworks BiRefNet for Fully On-Device 1024px Phone NPU Segmentation
By Mr.Xu Community Post
Published:
Summary:An independent researcher has released Lumen Cutout under the MIT License, adapting BiRefNet-lite for fully on-device high-resolution subject segmentation. The project replaces or rewrites deformable convolutions, shifted-window attention, channel-split convolution, and the high-resolution decoder so that the complete graph can compile for Qualcomm NPUs. Quantization-aware training and distillation are performed through the exported INT8 NPU graph. The author reports a 23.7 ms latency for a 102
Project Overview
Lumen Cutout does not simply compress the original BiRefNet-lite. It first removes mobile compiler and memory bottlenecks, then uses fully INT8 quantization-aware training and distillation to account for quantization error in the deployed graph.
The original network contains 20 deformable convolutions, 5D/6D tensors in its Swin backbone, shifted windows implemented through cyclic rolls, and large 1024-pixel activations. These components either exceed the supported operator set in Qualcomm QNN/LiteRT or exhaust on-chip memory. The project performs a series of graph rewrites and architectural changes:
- Replaces window partitioning with a 4D implementation;
- Replaces cyclic rolls with fixed masks;
- Splits convolution channels to reduce intermediate tensor sizes;
- Executes attention one head at a time;
- Runs the main backbone at 640 pixels and reconstructs a 1024-pixel mask from two input paths;
- Uses a slim full-resolution tail and retains one global window in the fourth Swin stage.
These changes solve compiler compatibility and residency problems, but they do not automatically preserve segmentation quality. The author therefore performs QAT through the exported NPU graph so that training sees the rounding and operator behavior of the actual INT8 deployment path.
Performance and Deployment
On a Galaxy S26 powered by Snapdragon 8 Elite Gen 5, the project reports a 23.7 ms latency for a 1024×1024 mask. A Galaxy S25 with the same SoC takes 31.1 ms, while a mid-range device with Snapdragon 7 Gen 1 takes 182 ms. All 2,060 layers execute on the NPU without a visible CPU fallback.
The comparison with the original model highlights the engineering benefit. Stock BiRefNet-lite runs out of memory at 1024 pixels on the same phones. At 512 pixels, it can run but falls back to the CPU, taking 677 ms with a 2.1 GB peak memory footprint. Lumen Cutout's primary contribution is therefore not simply fewer FLOPs; it turns an otherwise impractical high-resolution pipeline into a single NPU partition.
The release includes AOT binaries for 42 Snapdragon chips, with a separate GitHub Actions build job for each target. Chip-specific differences remain in the build matrix, while application code can use a shared LiteRT/QNN inference interface.
Accuracy
On held-out ECSSD and DUT-OMRON datasets, the INT8 student obtains S-measure scores of 0.939 versus 0.938 for the FP32 teacher and 0.830 versus 0.826, respectively. Aggregate metrics remain close. The author also notes that quantized edges are somewhat softer, showing that similar global scores do not imply identical pixel-level boundaries.
On DIS-VD, which emphasizes fine structures such as wires and fences, the model loses 0.017 S-measure. Six public models also outperform it on DUT-OMRON. The system is thus an engineering compromise for mobile interaction rather than a new state-of-the-art segmentation model.
Silent Distillation Failure
One of the project's most reusable findings concerns a gate branch that failed without producing an obvious error. The sigmoid gate in the deepest decoder branch saturated at approximately 1e-11 in the student, effectively disabling that path. The remaining decoder compensated well enough that the final mask loss remained stable, even though the teacher relied on the branch with a gate value of about 0.46.
The author restored the branch from the teacher and used a feature loss to keep it open. This illustrates why final-task loss alone is insufficient for monitoring distilled models. Gate saturation rates, branch activation magnitudes, or feature-space distances should be included in training gates to detect internal collapse that remains hidden behind a strong aggregate metric.
Engineering Trade-offs and Limitations
High-resolution output is essential for interactive subject extraction, but it also increases activation-memory pressure. Lumen Cutout avoids running the full-resolution backbone directly by using dual-input reconstruction and a slim tail. This reduces memory use, but fine-detail recovery now depends on the fusion path.
The reported 1024-pixel speed advantage depends on Snapdragon NPUs, precompiled binaries, and a single graph partition. It should not be extrapolated to MediaTek or Tensor backends.
Power consumption has not yet been measured, so the energy cost behind the 23.7 ms result is unknown. Sustaining 42 NPU inferences per second also does not guarantee that the entire camera or gallery pipeline reaches 42 frames per second: preprocessing, UI-thread contention, and memory movement remain outside the reported model latency.
Deployment Guidance
Mobile teams should evaluate the project in a staged manner:
- Test the precompiled binary for the target SoC first to establish real-device latency and peak memory before taking on the full QNN/LiteRT conversion flow;
- Evaluate edge quality, thin-line recall, and human preference on application-specific gallery data rather than relying only on S-measure;
- Inspect activation magnitudes and gate values after quantization to detect silently disabled decoder branches;
- Include cold start, sustained load, camera preview, and background switching in end-to-end tests;
- Maintain a CPU or GPU fallback path because the current release covers Snapdragon devices only.
Lumen Cutout's core contribution is moving a BiRefNet-class segmentation model from merely offline execution toward fully resident inference on a flagship phone. It is not yet a general-purpose edge segmentation platform, and its cross-NPU portability remains unproven. For Android gallery cutouts, real-time effects, and offline image editing, however, it provides a remarkably complete and reproducible engineering reference.
— END —Source: Reddit r/MachineLearning (2026-11-15)
Tags: #Mobile NPU #Image Segmentation #INT8 Quantization #BiRefNet #QNN
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments