Source: https://www.astrmira.com/en/research/papers/low-bit-decisions/
Language: en

[Home](https://www.astrmira.com/en/)/[Research](https://www.astrmira.com/en/research/)/2609.09854

VECTOR SEARCH / ARXIV:2609.09854

# When Does Low-Bit Quantization Preserve the Decisions of Vector Search?

Places quantization error inside the ranking and graph-pruning decisions that vector search actually makes, relating reliability to comparison margins, correlated residuals, and execution traces.

[Read on arXiv](https://arxiv.org/abs/2609.09854)[Open versioned PDF](https://arxiv.org/pdf/2609.09854v1)

Paper details

**Authors**: Wenxuan Xiao, Xu Cao

**First submitted**: 2026-09-09

**Latest revision**: 2026-09-09

**Public version**: arXiv 2609.09854v1

Vector search · Decision stability · Quantization theory

A comparison changes only when quantization error crosses its decision boundary.

## From distance error to a concrete choice

Vector search repeatedly chooses which candidate is closer and which edge should remain. Average error and global rank correlation do not explain why these local decisions fail, so this work analyzes the comparisons executed by the algorithm itself.

The key quantities are the original margin between candidates and whether quantization error can cross it. The analysis separates risk near the decision boundary from the tail behavior of calibrated residuals, including correlations induced by a shared query or graph node.

## Connecting local decisions to a graph trace

For Vamana neighbor selection under a fixed candidate order, the paper connects agreement on individual pruning decisions with agreement of the final neighbor list on a frozen exact state.

When distributional assumptions are unreliable, independent held-out blocks provide risk estimates for a fixed quantization rule. Binary codes, RaBitQ, Lucene BBQ, and product quantization can be studied through the same decision interface.

## Choosing quantization for the search process

Across learned, classical, and synthetic representations, normalized comparison margins predict ranking and pruning flips better than global rank correlation. The scope covers fixed candidate sets and frozen traces; end-to-end recall also depends on candidate coverage.

Read and cite

## Continue with the original paper.

The complete derivations, experimental setup, and results are available in the public manuscript.

[Abstract and version history ↗](https://arxiv.org/abs/2609.09854)  
[Full paper PDF ↗](https://arxiv.org/pdf/2609.09854v1)

Related papers

REPRESENTATION GEOMETRY · 2026.05

### What survives at one or two bits?

Research at a glance

Covariance Structure and Coordinate Heterogeneity Govern Binary Quantization of Contrastive Embeddings

Explains low-bit ranking behavior through covariance structure and coordinate heterogeneity, including why an extra magnitude bit and random rotation help different representations in different ways.

Representation geometry · Binary quantization

[Explore this research](https://www.astrmira.com/en/research/papers/covariance-binary-quantization/)

[arXiv](https://arxiv.org/abs/2605.17524)

SEARCH SYSTEMS · 2026.05

### QuIVer: building the search graph directly in binary space

Research at a glance

QuIVer: Rethinking ANN Graph Topology via Training-Free Binary Quantization

Builds, prunes, and navigates an ANN graph in a training-free two-bit space, reading full vectors only for final reranking and exposing the data regimes where compact topology works.

Approximate nearest neighbors · Graph indexes

[Explore this research](https://www.astrmira.com/en/research/papers/quiver/)

[arXiv](https://arxiv.org/abs/2605.02171)