SSLA-Det researchers present low-latency event-based object detection at ECCV 2026; reduces per-event computation by over 20x

SSLA-Det researchers present low-latency event-based object detection at ECCV 2026; reduces per-event computation by over 20x

This paper has been accepted for publication at the European Conference on Computer Vision (ECCV), 2026 Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention Haiqing Hao1 , Zhipeng Sui1 , Rong Zou2 , Zijia Dai3 , Nikola Zubić2 , Davide Scaramuzza2 , and Wenhui Wang1⋆ 1 State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China 2 Robotics and Perception Group, University of Zurich, Zurich, Switzerland 3 ShanghaiTech University, Shanghai, China Abstract. Event cameras provide sequential visual data with spatial sparsity and high temporal resolution, making them attractive for low- latency object detection. Existing asynchronous event-based neural net- works exploit this low-latency advantage by updating predictions event by event, but still suffer from two bottlenecks: recurrent architectures are difficult to train efficiently on long sequences, and improving accu- racy often increases per-event computation and latency. Linear atten- tion is appealing because it enables parallel training and recurrent infer- ence. However, its dense state updates make per-event computation scale with the state size, yielding a poor accuracy-efficiency trade-off for ob- ject detection, where accurate localization requires fine-grained spatial states. The key challenge is therefore to introduce sparse state activa- tion that exploits the spatial sparsity of events while preserving efficient parallel training. We propose Spatially-Sparse Linear Attention (SSLA), which introduces a mixture-of-spaces state decomposition and a scatter- compute-gather training procedure, enabling state-level sparsity as well as training parallelism. Building on SSLA, we develop an end-to-end asynchronous linear attention model, SSLA-Det, for low-latency event- based object detection. On Gen1 and N-Caltech101, SSLA-Det achieves state-of-the-art accuracy among asynchronous methods, reaching 0.375 mAP and 0.515 mAP, respectively, while reducing per-event computation by over 20\times compared with the strongest prior asynchronous baseline, demonstrating the potential of linear attention for low-latency event- based vision. Code is available at: https://github.com/haohq19/ssla. Keywords: Event camera · Linear attention · Object detection 1 Introduction Event cameras provide sequential visual data with spatial sparsity and high temporal resolution, making them highly promising for low-latency perception [9,11,29]. Asynchronous event-based neural networks realize this potential by up- dating their predictions every time a new event arrives [36,37]. This event-driven ⋆Corresponding author: wwh@tsinghua.edu.cn

2 H. Hao et al. (a) Events (b) SSLA (c) Detection 10 2 10 0 10 2 10 4 Computation per new event (MFLOPS/ev) 0.1 0.2 0.3 0.4 0.5 mAP Proposed Asynchronous Synchronous SSLA-S SSLA-B SSLA-M SSLA-L DAGr-N DAGr-S DAGr-M DAGr-L FARSE-CNN* AEGNN* NVS-S* AsyNet* EVA+RVT-B ERGO-12 EventPillars SMamba SAST-CB GET Goal (d) mAP vs. FLOPS Fig. 1: Our method processes (a) asynchronous event sequence with (b) a sparsely acti- vated linear attention neural network for (c) low-latency event-based object detection. On the Gen1 dataset, our SSLA-Det models achieve SOTA asynchronous mAP and lower FLOPS compared with previous asynchronous baselines (d). ^{*} refers to AP_{50}. processing paradigm is particularly appealing for object detection in latency- critical scenarios, such as autonomous driving [11], drone obstacle avoidance [7], and vision-based control [15]. Despite their much lower latency, existing asynchronous event-based neural networks still lag behind their synchronous counterparts in accuracy [33,49,50]. The gap stems from two coupled architectural bottlenecks. The first is the parallel-recurrent bottleneck: the event-by-event inference paradigm naturally relies on recurrent architectures, whereas efficient training on long event se- quences requires parallelization along the sequence dimension [13,39]. The sec- ond is the accuracy-efficiency trade-off: improving accuracy typically requires larger and deeper models, while scaling the model increases per-event compu- tation and consequently latency. A natural way to mitigate this trade-off is to exploit the spatial sparsity of event camera data through sparse neural network activation [36,37]. Nevertheless, as receptive fields expand layer by layer in deep networks, sparse inputs can still induce dense activations. To preserve sparsity in deep layers and reduce computation, prior work has designed specialized ar- chitectures [11,36], but this comes with additional architectural constraints that further limit accuracy. Linear attention4 has emerged as a promising asynchronous event-based model architecture since it naturally addresses the parallel-recurrent bottle- neck [12,20,31,46,47]. However, existing methods are limited to relatively simple global-level classification tasks [38,41], while more challenging local-level tasks, like object detection, remain unexplored. The main obstacle is the poor accuracy- efficiency trade-off due to the lack of state-level sparsity, i.e., linear attention updates all elements of its state, making per-event computation scale with the state size. This is problematic for object detection, where accurate localization 4 For convenience, here we use linear attention as a shorthand for parallel-trainable linear recurrent models, including state space models (SSMs) and linear recurrent neural networks (linear RNNs).

Low-latency Event-based Object Detection with SSLA 3 requires fine-grained spatial representations and therefore a large state size [13]. Although sparsifying state activation is conceptually straightforward, the key challenge is to maintain parallel training while gaining sparsity. We address this challenge by introducing Spatially-Sparse Linear Attention (SSLA), a linear attention module with state-level sparsity while preserving its parallel training advantages. To enable state-level sparsity, we introduce a mixture-of-spaces (MOS) structure inspired by [6] that decomposes the global state into substates with spatially overlapping receptive fields, and each event activates only a few substates based on its location. To preserve the sparsity in deep networks, we aggregate the activations of each event from all its acti- vated substates, preventing the expansion of the activated region. We further propose a position-aware projection (PAP) that projects events based on their relative positions within every activated substate, injecting state-relative spatial priors. We derive a scatter-compute-gather training procedure that parallelizes this sparse activation structure by sequence-level reorganization. Specifically, events are scattered into state-specific subsequences, computed in parallel by linear attention, and then gathered back into the original event sequence. In this way, SSLA makes linear attention sparse in space, recurrent in time, and parallel in training. Building on the SSLA module, we present SSLA-Det, to the best of our knowledge, the first end-to-end asynchronous linear attention model for low- latency event-based object detection (Fig. 1 (a)-(c)). Experiments on Gen1 and N-Caltech101 show that SSLA-Det achieves a substantially improved accuracy- efficiency trade-off over prior asynchronous methods (Fig. 1 (d)), including state- of-the-art (SOTA) asynchronous mAP (0.375 on Gen1 and 0.515 on N-Caltech101) at much lower computational cost (over 20\times reduction compared with previous SOTA [11]). Our contributions are as follows: – We propose an SSLA module for sequential event modeling, including a MOS structure for state-level sparsity, PAP for spatial prior encoding, and a scatter-compute-gather procedure for efficient parallel training. – We present SSLA-Det, to the best of our knowledge, the first end-to-end asynchronous linear attention model for event-based object detection. – SSLA-Det sets a new accuracy-efficiency frontier, reaching 0.375 mAP on Gen1 and 0.515 mAP on N-Caltech101, while reducing computation by over 20\times compared with the prior asynchronous SOTA method. 2 Related Work 2.1 Asynchronous Event-based Neural Networks Asynchronous event-based neural networks treat event camera data as different geometric structures to leverage their spatial sparsity. This strategy includes graphs [3,4,11,22,37], submanifolds [25,36], point clouds [39,43], and sequences [13,19,38,41]. Graph-based methods [2–4,11,22,37] transform events into sparsely connected spatial-temporal graphs, and derive local update rules on the graph for

4 H. Hao et al. recurrent inference. However, they have limitations in temporal accumulation [4], failing to handle long event sequences. Submanifold methods [25, 36] assume that events lie on a spatial submanifold, and conduct convolution only on the submanifold to keep sparsity. However, this submanifold assumption does not strictly hold, and thus leads to suboptimal performance. Point cloud methods [39,43] represent events as spatial-temporal 3-D point clouds, and process with PointNet [35]. This approach is limited to shallow neural networks, and thus has difficulty in challenging tasks. Sequence-based methods use causal sequence- to-sequence models for event processing including softmax attention or linear recurrent models. For example, [19] uses attention to process batched events at each timestep. EventSSM [38] and S7 [41] use SSMs, which enable parallel training and recurrent inference, but are limited to global-level classification tasks. EVA [13] explores linear attention for event-based object detection but still demands a dense backbone, while our method is fully end-to-end asynchronous. 2.2 State-level Sparsity in Linear Attention A recent direction to improve linear attention is to introduce state-level spar- sity. Mixture-of-Memories [6] and Sparse State Expansion [28] maintain multiple independent states and use a learned router to send each token to only a few memories. Our SSLA follows the same sparse state activation idea but is funda- mentally different in that the construction of substates is spatially structured, which enables geometric routing of event embeddings and admits the position- aware projection that encodes spatial inductive bias. Concurrent work [40] also introduces local states in linear attention for local-level event-based vision, but requires spatial contraction at discrete timestamps, which is not asynchronously, event-by-event trainable. Our work, instead, uses a scatter-compute-gather al- gorithm to reorganize events into subsequences and does not rely on spatial contraction, which is fully event-by-event asynchronous. 3 Method 3.1 Problem Formulation: Asynchronous Event Processing Event camera data are represented as a sequence of events \ p rotec t \mathcal {E}={e_i}{i=1}^L, where each event e i = ( \ma thbf {x}i, t_i, p_i ) carries its spatial coordinates \p r otect \mathbf {x}i \in \mathbb {R}^ 2, a timestamp t i \in \mathbb {R} and a polarity p i \in {+1, -1\ }. The sequence is temporally ordered so that t_ i \leq t_{i+1}. Asynchronous event processing learns a causal stateful neural network \protect \mathcal {M} that takes \protect \mathcal {E} as input and makes predictions incrementally, which can be formulated as a causal sequence-to-sequence problem from \ifmm ode \lbrace \else \textbraceleft \fi e_i}{i=1}^L to \ifmmo de \lbrace \else \textbraceleft \fi \hat {\mathbf {y}}i}{i=1}^L. Specifically, for each new incoming event e i, the model updates its state \protect \mathbf {S}_ i and produces a new prediction \protect \hat {\mathbf {y}}_ i, as (\hat {\ m athbf {y}}_i, \mathbf {S}i) = \mathcal {M}(e_i, \mathbf {S}{i-1} ). 3.2 Preliminaries: Linear Attention Linear attention models (linear RNNs, SSMs) are linear-time alternatives to softmax attention [44] for sequence-to-sequence modeling. Causal linea