ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #On-Device AI #NER #LLMs & Foundation Models #Model Evaluation #Latency Optimization

arXiv Publishes Study on On-Device NER: Evaluating Accuracy, Cost, Reliability, and Confidence

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 13 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:arXiv has released a study on on-device Named-Entity Recognition (NER), evaluating nine systems across three paradigms for accuracy, latency, and output validity. The research highlights the deployability of encoder-based models, which offer lower latency and more reliable outputs compared to generative LLMs, despite the latter's competitive accuracy on clean datasets. The study also examines confidence calibration techniques and their impact on task performance, providing valuable insights for


Background and Objectives

With the increasing demand for on-device AI applications, Named-Entity Recognition (NER) is transitioning from cloud-based services to local devices. This shift introduces new challenges: achieving low latency, high accuracy, and reliable NER without relying on cloud APIs, while ensuring the validity of the output.

Methodology

The research team evaluated nine different NER systems across three paradigms:

  1. Classical taggers: such as spaCy.
  2. Bidirectional encoder specialists: like GLiNER (ranging from 166M to 460M parameters).
  3. Locally-run generative LLMs: including Qwen3 (0.6B/1.7B/4B-Instruct) and DeepSeek-R1 (1.5B/8B).

The evaluation was conducted on three datasets with varying characteristics, and two new evaluation axes were introduced: latency and output validity.

Key Findings

  1. Accuracy vs. Deployability:

    • While a 4B instruction LLM leads on clean newswire data, encoder-based models offer better deployability with lower latency and more reliable outputs, despite slightly trailing in accuracy.
    • The smallest generative models exhibit up to 27% invalid output on long inputs, a problem that can only be fixed by scaling, not by adjusting the output budget.
  2. Confidence Calibration:

    • GLiNER's confidence scores rank correctness well (AUROC 0.76 to 0.86) but are overconfident (ECE 0.24 to 0.47), which can be halved by temperature scaling.
    • Thresholding provides a small honest out-of-sample F1 gain, and an all-local small-to-large cascade strategy offers a modest, corpus-dependent gain over cost-matched random routing.
  3. Impact of Gold Standard:

    • The provenance of the gold standard significantly affects model rankings: moving from LLM-authored silver to human gold raises the rankings of all encoder models and lowers those of all generative models.

Technical Highlights

  • Multi-Dimensional Evaluation Framework: Introduces latency and output validity into the NER evaluation framework, providing a more comprehensive performance assessment for on-device applications.
  • Confidence Calibration Techniques: Proposes methods like temperature scaling to mitigate model overconfidence and improve output reliability.
  • Gold Standard Analysis: Reveals the impact of different gold standards on model rankings, offering important insights for future NER research.

Industry Implications and Developer Recommendations

  • On-Device NER Prospects: As on-device AI demands grow, encoder-based models will likely be the preferred choice due to their low latency and high reliability.
  • Model Selection Advice: Developers should choose models based on specific application scenarios and employ confidence calibration techniques to enhance output quality.
  • Future Research Directions: Further exploration of more efficient confidence calibration methods and optimization of model performance on long inputs is recommended.

Source: ArXiv NLP/LLM (cs.CL) (2026-10-02)

— END —

Tags: #On-Device AI #NER #LLMs & Foundation Models #Model Evaluation #Latency Optimization

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…