Vajra Search with a Rust backend (v0.2.1): Greater Build and Latency Performance
Vajra Search is the successor to Vajra-bm25, with a new backend implemented in Rust that covers the vector index core and which is published to PyPI as v0.2.1.
Welcome to my space for exploring ideas across AI, aviation, geopolitics, philosophy, and culture. Each blog represents a different facet of my interests and work.
Vajra Search is the successor to Vajra-bm25, with a new backend implemented in Rust that covers the vector index core and which is published to PyPI as v0.2.1.
Technical notes on the Vajra search stack transition from BM25 benchmarks and Python vector search (v0.4.1/v0.5.0) to the Rust HNSW backend (`vajra_search`), with reproducible Wikipedia results.
Vajra's HNSW vector search started 217× slower to build and 8× slower to query than ZVec. Six targeted engineering changes brought it to 36× and 1.6× respectively — without rewriting in C++ or adding Numba.
A neural system that learns to parse mathematical notation rather than compute it, achieving 90%+ accuracy by separating structure understanding from calculation.
When you train a neural network, you're running a dynamical system that carves out a representation space. ToDACoMM measures the topology of what gets carved, revealing a striking divide between encoders and decoders, and now extends to MLPs and large-scale transformer analysis.
A comprehensive look back at my year in AI - scaling up at work, building the Praval agentic AI framework, Vajra Search, Tlon mathematics, ToDACoMM, exploring mechanistic interpretability, honest research failures, and reflections on curiosity vs. accomplishment.