papers

Things I’m reading or keep coming back to.

multimodal representation

• The Platonic Representation Hypothesis: vision and language models converging on a shared model of reality.
• Perception Encoder: the best visual features aren’t at the output of the network.
• SigLIP 2: multilingual encoders with better localization and dense features.
• ImageBind: binding six modalities into one embedding space.
• Matryoshka Representation Learning: nested embeddings at multiple truncation sizes.

video understanding

• V-JEPA 2: self-supervised video models that understand, predict, and plan.
• VideoPrism: a general-purpose video encoder.
• InternVideo2: scaling video foundation models.
• VideoMAE: masked autoencoders for video, data-efficient.
• TimeSformer: space-time attention for video.

efficient inference

• Token Merging: combine similar tokens for 2x ViT throughput.
• Run-Length Tokenization: drop temporally repeated video patches before inference.
• Hiera: a simpler, faster hierarchical vision transformer.
• TSM: zero-parameter temporal modelling by shifting features.

retrieval & indexing

• ScaNN: anisotropic vector quantization for large-scale search.
• RaBitQ: binary quantization with error bounds.
• ColBERTv2: late interaction retrieval with residual compression.

calibration & risk control

• Conformal Risk Control: distribution-free control of any monotone loss.
• Learn then Test: calibrating hyperparameters with valid error control.
• RCPS: risk-controlling prediction sets.

evaluation

• From Frames to Events: rethinking metrics for human-centric anomaly detection.
• Rethinking Metrics and Benchmarks of Video Anomaly Detection: why frame-level AUC hides the failures that matter.
• Rethinking the Evaluation of Video Summaries: why some summarization numbers don’t count.