The 0.8 work corrected that boundary. It added common model contracts, Gemini, OpenAI-compatible servers, MCP tool clients, multimodal input, request-based voice, a separate embedding runtime, stronger lifecycle handling, and a larger set of executable notebooks. Across 320 changed files, the purpose was consistent: make one Agent useful on its own, then let the same Agent participate in a larger system without changing how its model work is executed.

I uploaded 0.8.0 before its matching Git tag and GitHub release were complete, then withdrew it. PyPI does not allow a deleted distribution filename to be reused, so I continued with version 0.8.1. The mistake led to a firmer rule: a release now means one source commit, one wheel, matching documentation, and one release record.

The Praval 0.8.1 release notes provide the feature summary, installation command, and links to the documentation.

Rebuilding the model execution path

I moved shared execution into ModelRuntime and defined its public contracts in praval.models. The runtime builds and validates requests, resolves model capabilities, normalizes responses and streams, executes tools, applies approval, records usage, and creates spans. Provider adapters now translate between that contract and their native APIs.

The direct Agent interface follows this division. chat() still returns a string for existing applications. generate() and agenerate() return a ModelResponse with content, provider, finish, usage, and tool details, while stream() and astream() expose a common sequence of events. The decorated Agent API, start_agents(), Reef, and established imports remain supported, so applications can adopt the richer response and async APIs when they need them.

OpenAI, Anthropic, Cohere, Gemini, and OpenAI-compatible endpoints now sit behind this contract. A capability registry records whether a model supports tools, structured output, reasoning controls, streaming, and media. Praval rejects unsupported requests instead of silently weakening them, and local-server presets begin conservatively because an OpenAI-compatible HTTP shape says little about the model behind it. Structured output remains precise: the provider constrains generation and returns JSON in ModelResponse.content; the application parses and validates it when a local schema is authoritative.

Typed content parts carry images, files, audio, and video where the model supports them. OpenAI-backed Agents add transcribe() and speak() for bounded speech requests, while persistent realtime audio remains a separate future problem. EmbeddingRuntime also stays separate from chat because a vector collection depends on its embedding model and dimensions. It supports SentenceTransformers, OpenAI, Gemini, and compatible endpoints, and reports when a collection must be reindexed.

Keeping tools, approval, and collaboration explicit

A model can propose an action, but the application must validate and authorize it. Praval now routes decorated tools, shared tools, external JSON schemas, and provider tool calls through one Agent registry. Agent.add_tool_spec() attaches schemas that originate outside Praval without bypassing argument handling, errors, tracing, async rules, or approval.

HITL uses that provider-neutral path. Before a protected handler runs, Praval stores the tool request and interrupted runtime state in SQLite. A reviewer can approve the original arguments, edit them, or reject the action, and another process can resume the continuation. If approval is required and HITL is not configured, execution stops. This gives the model room to prepare an action without giving it the authority to perform that action.

MCP fits the same design. The optional client discovers tools from stdio or Streamable HTTP servers, namespaces them, and registers async handlers with the Agent. Those handlers inherit approval by default, timeouts, size limits, tracing, redaction, and cleanup. The first release handles text and structured tool results. Resources, prompts, hosting, managed OAuth, binary results, automatic reconnect, and synchronous session bridging remain outside its scope.

ModelRuntime operates inside an Agent; Reef connects Agents. Spores carry structured messages and correlation data, async handlers can perform network work, and wait_for_completion() replaces fixed sleeps. An in-process Reef serves one application, while RabbitMQ supports Agents in separate processes or machines. This separation keeps deterministic workflows independent of model execution and lets a research or release system assign work to specialists with different evidence and permissions.

Memory, storage, observability, and lifecycle support both single-Agent and team designs. Praval separates short-term, episodic, semantic, and long-term memory, and its async storage layer supports the filesystem, PostgreSQL, Redis, S3-compatible storage, and Qdrant. Spans now finalize before one-time storage and can go to the console, SQLite, or OTLP HTTP. PravalApp closes the Agents and Reef it creates, without pretending to be a general dependency container.

The four capstones test these boundaries as systems. Research specialists gather and challenge evidence, a support team combines policy with customer history, release reviewers run checks in a temporary workspace, and a live marketing studio uses OpenAI, screenshot input, structured assets, a protected claim, persisted approval, and campaign memory. Each case exposes the message trail and application state instead of hiding the framework behind notebook helpers.

Testing the wheel developers install

I did not want the new surface area supported by tests that passed only inside the source tree. The 0.8 cycle fixed the causes of expected failures, made unexpected passes fail, removed broad coverage omissions, added focused coverage floors, and made formatting, linting, typing, documentation, and package checks fatal. Provider contract tests now exercise the same requests, streams, tools, usage, errors, and capability rules across adapters. MCP, Reef, storage, observability, continuation, media, and shutdown paths have their own failure tests.

Every Python demo and notebook is registered with its dependencies, services, provider needs, timeout, and certification mode. The runner creates a clean environment, installs the supplied wheel, clears PYTHONPATH, verifies the package path and wheel hash, and runs outside the repository. Thirteen course notebooks explain the framework in sequence, and four capstones show complete systems rather than isolated calls.

Paid provider checks remain manual because a fake cannot prove that a service accepted media, generated a protected tool request, or returned usable audio. The live voice path sends a fixture through STT, an Agent, TTS, and a second transcription. Other checks cover real streaming, tools, structured output, media, embeddings, and HITL with credentials and model names supplied by the developer. These calls do not run on every push.

Publication now begins with the exact CI wheel. Documentation is built against that installed package with source imports disabled, release evidence stays outside dist/, and GitHub receives the same wheel uploaded to PyPI. That discipline ties the runtime work, examples, documentation, and public artifact to the same code.

Praval 0.8.1 is available on PyPI and in the matching GitHub release. The notebook course shows the framework in execution, and the reference documentation covers the public API.

If I have a repository of 3D objects, I do not want to browse it like a filesystem. I want to type something like hex bolt, donut shaped object, flat washer, chair with a back, or gear-like part, and get the appropriate shape records back. I may then want to inspect the result visually, rotate it, compare it to neighbors, or use it as a starting point for a CAD or PLM workflow. Although I have used the first person here to describe the use case, the retrieval process is more common and more of an everyday problem than the less common problem of generating point clouds and 3D objects. It is understandable that a lot of people would want to think about using these models to generate point clouds or other 3D objects, but often, retrieval is the more important problem for most companies, large or small. Ergo, Vajra 3D Shape Search.

And that is an important distinction here. Vajra 3D is not a 3D generator. It does not take text and hallucinate a mesh. It searches a repository of already-indexed 3D shapes. The demo then renders the retrieved objects as point clouds so that the result is easy to inspect in the browser.

That distinction matters, because generation is an act of synthesis - you're taking a latent representation of the user's prompt and trying to generate geometry from it. However, search is an act of retrieval - you're looking to find relevant objects based on text input, from a repository of known 3D objects. For engineering workflows, retrieval is often the safer and more useful operation. If a vendor sends a part, a PLM system has a part number, and a geometry repository has known assets, the first question is not, "Can we invent a part?" The question is, "Can we find the right existing object?"

The Formulation Shift

The hard part is not drawing a shape. The hard part is retrieving the right indexed shape from language. Behind this simple requirement, there are a lot of interesting ideas on how to bring about text based 3D object retrieval via a search system like Vajra search.

To build Vajra 3D, I framed the problem this way:

1. Each object has a point cloud, which is a collection of (x,y,z) coordinates that together represent a 3D object.
2. Each object also has labels, aliases, descriptions, and metadata. This is crucial, and Vajra relies on these details.
3. A model maps point clouds and text into a shared embedding space. Side note: Building the core hypothesis around this model, and using the model in the demo I've built for this post were key work prior to this blog post.
4. Vajra indexes the object embeddings with HNSW.
5. Vajra indexes the metadata text with BM25.
6. Hybrid search combines the two rankings.
7. The frontend renders the selected result as a point cloud.

That gives us something very practical: a search box for a 3D object repository.

The user experiences this as a search field and a viewer. Underneath it, the system is doing both semantic retrieval and lexical retrieval over the same shape repository.

What The Demo Does

The public demo has two Vajra pages on rajeshrs.in:

1. Vajra Docs Search, which searches a documentation corpus.
2. Vajra 3D Shape Search, which searches indexed synthetic 3D shapes.

The 3D page is intentionally direct. You type a query, choose a mode, and inspect the results. The selected result is rendered as a point cloud with mouse-based rotation and zoom.

The three modes are:

The visual point cloud is not the model output. It is the representation of the retrieved object. That is the right mental model for this demo: search first, visualization second.

Building The Model

For this first version, I wanted something small enough to train and reason about, but real enough to test the whole retrieval loop. The model is a compact text-to-point-cloud dual encoder. One side consumes text. The other side consumes point clouds. Both sides produce 128-dimensional embeddings in the same space.

The current model was trained on synthetic shape data. The catalog has 148 shape classes across primitive shapes, household-like objects, fasteners, and mechanical components. Each class has text aliases and descriptions. That matters because users do not all type the canonical label. Someone might type torus, ring, or donut shaped object, and the system should still have a chance of landing near the same class.

The point clouds are generated procedurally for the purpose of this demo and these were used to train the model. Of course, a real world fine-tune of this model based on the same architecture, could potentially involve a lot of data engineering, in addition to synthetic data like I've used. For training, the model consumes normalized point clouds. For display, the demo uses a denser point cloud so the browser rendering is easier to see. In the current demo setup:

Which brings me to something important. The model details are deliberately not the star of the post. The core idea is the retrieval system around it. Still, the model is doing the key translation: it gives text and shape a shared coordinate system. Without that, text search over geometry would collapse back to keyword matching.

There is another design choice here that I think is important. The model artifact is not public at this time. It is baked into the backend container for the demo, not served as a downloadable file. The reason is simple: this is a showcase of the Vajra 3D retrieval path and the deployed demo, not a model release. A public model release would need a separate model card, dataset statement, evaluation suite, and license decision. I'm not ready with all this quite yet.

Building The Index

Once the shape repository exists, indexing is straightforward in concept.

For each shape record, the system builds two retrieval views:

1. A dense vector view from the point-cloud embedding.
2. A lexical document view from metadata text.

The dense view goes into Vajra's HNSW index. The lexical view goes into Vajra's BM25 engine. These are separate retrieval channels over the same object IDs.

This split is useful because 3D search has two kinds of user intent.

Sometimes the query is named and precise: hex bolt, flat washer, torus. BM25 is excellent at that. Sometimes the query is descriptive: round object with a hole, long cylindrical fastener, box-like object with a top surface. The embedding model has a better chance there.

Hybrid search is the practical compromise. It lets exact names help when they are present, but it does not require every useful query to match a phrase in the metadata.

How Vajra Works Here

Vajra started with lexical search and then grew into vector and hybrid retrieval. I have written about the earlier stages in a few posts:

1. Vajra BM25: Building a Search Engine with Category Theory
2. Re-engineering Vajra's vector search with inspiration from ZVec
3. From Vajra v0.4.1 to v0.5.0 to Rust backend
4. Vajra Search with a Rust backend (v0.2.1)

The 3D demo uses the same retrieval philosophy, but the object being searched is different. Instead of document chunks, the result is a shape record.

At a high level:

• BM25 scores how well the query terms match the object's text metadata.
• HNSW searches nearby object vectors in the embedding space.
• RRF combines the rank lists without pretending BM25 scores and vector distances are naturally calibrated.

That last point is easy to miss. BM25 scores and cosine similarities are not the same kind of number. Reciprocal Rank Fusion avoids forcing them into one artificial scale. It asks a more stable question: which objects are ranked highly by one or both retrieval systems?

This is why Vajra is a good fit for the 3D problem. The search engine does not need to know that an embedding came from a point cloud. It needs a vector, an ID, metadata, and a ranking strategy. The model handles the modality bridge. Vajra handles retrieval.

Deployment

The deployed demo is intentionally small. The site is static and hosted through Netlify. The Vajra backend runs in a Railway container. The same backend now serves both the documentation search API and the 3D search API.

The public 3D corpus is capped at 296 records: two deterministic variants for each of the 148 classes. That is enough to demonstrate the search mechanics without turning the container into a data warehouse. The object embeddings and display point clouds are precomputed, so the runtime does not need to generate training data or run training jobs.

This is also why I did not use Hugging Face for the model in this demo. Hugging Face is useful when the goal is model distribution. Here, the goal is to showcase a protected application behavior. The model is an implementation detail inside the backend container.

That does not make the model magically impossible to copy. Any deployed software has a threat model. But it does mean the demo does not publish the artifact, does not expose embeddings through the API, and does not provide a download path. The public surface is search results and capped preview point clouds.

What I Learned

The most useful lesson is similar to what I saw while building Vidai: the formulation matters more than the model size.

With Vidai, the key move was to stop asking the neural network to do arithmetic. The neural network only had to parse mathematical notation; exact computation could be delegated to symbolic code.

With Vajra 3D, the key move is to stop asking the model to be the whole search engine. The model only has to map text and shape into a useful shared space. Retrieval, fusion, metadata, serving, security, and visualization are separate engineering problems.

Perhaps these lessons showcase something vital about the purposeful use of AI: problem framing and searching the problem space helps us figure out where to use specific, simpler-to-build capabilities. We don't need throw big models and agents at all problems.

That separation gives the system room to grow. A better point-cloud model can replace the current one. A CAD ingestion pipeline can replace synthetic generation. More metadata can enrich BM25. Vajra's HNSW and hybrid retrieval path stays the spine of the application. These are perhaps some current/future targets for this work - more on that below.

What Comes Next

This version is a proof of the pipeline, not the end state.

One next interesting step is ingestion of real 3D assets. Someone who wants to use Vajra may bring STEP, IGES, STL, GLB, OBJ, or point-cloud files. The pipeline should normalize those assets, sample point clouds, attach PLM metadata, run the embedding model, and index the resulting records in Vajra.

That would move the system from synthetic demonstration to industrial retrieval.

There are several open directions:

• CAD and B-rep aware ingestion for STEP and IGES.
• Better point-cloud models, possibly distilled from stronger open 3D-language systems.
• Larger shape corpora with meaningful intra-class variation.
• Search evaluation against real user queries.
• PLM-aware metadata ranking for part numbers, materials, suppliers, and assemblies.
• Similarity search from a query point cloud, not just from text.

The current demo is deliberately smaller than all of that, it was literally built this weekend over a few hours of spare time I had. But it proves the important thing first: text can retrieve indexed 3D objects, and Vajra can sit underneath that retrieval loop. And there's a lot possible in this (3D) space!

Try It

You can try the Vajra 3D search demo here:

• Vajra 3D Shape Search

Note that there is also a regular search demo on this site for documentation. This demonstrates Vajra Search's basic capabilities (lexical, vector, hybrid search).

The Python package for the search engine is here:

• vajra-search on PyPI

The 3D embedding model itself is closed source at this time. The public demo exposes the search behavior, not the model artifact.

Related Vajra Posts

If you want the background for the search engine side, these are the relevant earlier posts:

1. Vajra BM25: Building a Search Engine with Category Theory
2. Re-engineering Vajra's vector search with inspiration from ZVec
3. From Vajra v0.4.1 to v0.5.0 to Rust backend
4. Vajra Search with a Rust backend (v0.2.1)

Vajra 3D is the same search philosophy pointed at a different kind of object. Documents were the first corpus. Shapes are the next one, and perhaps this bridges the bits-to-atoms gap somewhat. See you in my next post!

Paradigms and Abstractions

One reality is that large language models are still fundamentally predictive systems. They generate outputs by modeling patterns in data and selecting likely continuations under a learned distribution. Calling them “reasoning models” or “multimodal systems” does not erase this underlying fact. They do not possess meaning in the human sense. They do not understand in the way conscious beings understand. They do not have grounded intentionality, lived experience, or an intrinsic sense of what their symbols refer to. They are extraordinarily capable statistical systems operating over representations. This matters, because much of the confusion in public discourse comes from anthropomorphic language. If we describe these systems as though they think, know, want, or understand in the same way human beings do, we smuggle in assumptions that the underlying machinery does not justify.

A second reality is that AI, in its current paradigm, is software sitting atop a very hard physical substrate. It is not magic. It is not abstract intelligence floating free of material constraints. It depends on compute, memory, storage, networking, electrical power, cooling, and specialized chips. It depends on the design and manufacture of CPUs, GPUs, TPUs, high-bandwidth memory, interconnects, servers, data centers, and power infrastructure. It depends on firmware, drivers, kernels, compilers, distributed systems, model serving frameworks, observability stacks, and orchestration software. Behind every “intelligent” model is a dense pyramid of engineering and industrial effort.

Another important reality is that model training alone does not solve business problems. Pre-training, post-training, instruction-tuning, RLHF, constitutional methods, and alignment layers may produce models that are more usable, safer, or more polished, but none of this automatically yields a system that can do useful work in a real organizational setting. To solve actual problems, models need harnesses around them. They need retrieval, memory, task decomposition, orchestration, tools, APIs, permissions, guardrails, evaluators, routing logic, fallback logic, caching, monitoring, and user interface design. Much of what is sold as “AI” in practice is not the model alone but the surrounding software system that turns raw capability into usable work.

The AI Supply Chain

That leads to another reality: AI has a long and fragile supply chain. The AI stack begins far below the application layer. It starts with mining, materials, refining, semiconductor equipment, wafer fabrication, packaging, energy systems, logistics, and geopolitical stability. The concentration of advanced chip manufacturing in Taiwan and the dependencies on TSMC create critical vulnerabilities in this supply chain. Above that sit chip design firms, foundries, cloud providers, networking vendors, model labs, open-source communities, data pipelines, storage systems, and inference platforms. Then come application developers, integrators, and domain teams building systems that ordinary users actually touch. This means that AI is not only a software story or a model story. It is also a manufacturing story, an energy story, a capital expenditure story, and a geopolitical story.

This is why the present AI landscape is better understood as a systems landscape rather than purely a model landscape. The raw model matters, but the surrounding architecture matters just as much. The data matters, and deployment matters too. The supply chain of power, GPUs, compute, ASIC and other pieces all end up mattering in due course, if you're building a product that touches customers. A mediocre deployment of a frontier model can be less useful than a well-designed system using a smaller model, better prompting, good retrieval, narrow task boundaries, and strong operational logic. In many domains, intelligence in practice is emerging less from one giant leap in the model than from careful engineering of the whole pipeline around it. Policy around the use of AI matters and region-specific policies are common these days. And without a doubt, as of April 2026, an errant missile in the Middle East can put paid to your entire company if you don't have cross-region availability of your AI product.

Compute Constraints in AI

A further reality is that current AI systems remain heavily compute-constrained. This is true not only because training frontier models requires a huge amount of compute, following scaling laws established by Kaplan et al. and refined by Hoffmann et al.'s Chinchilla research, in the form of large NVidia server farms, but because serving them at scale, aka inference engineering, is computationally and money-wise expensive too. Inference is not free, and latency and concurrency matter for a good user experience. Concurrency, longer context windows, tool use, reasoning capabilities... all these need extensive compute to innovate on. Even the largest AI labs face hard trade-offs around pricing, rate limits, model availability, and product design because compute remains a binding constraint. In the last few days as of early April 2026, there's a running joke on X, about how Claude Opus is pushing back on executing repetitive tasks, through some kind of weird alignment. It identifies whether there are repetitive tasks that are embarrassingly simple and tells the user to manually complete them. Not only is this the definition of pushing the tedium back to users, but it tells us what the priorities are, for those at Anthropic. Recently, when Dario Amodei was on Dwarkesh Patel's podcast (March 2026), he mentioned that it is hard for Anthropic to manage the economics of inference even if he fell marginally short of their sales targets. Compute is expensive, and the big AI labs are feeling the pinch. It is therefore misleading to speak as though AI capability unfolds in a frictionless digital realm. It unfolds under very real economic and infrastructural bottlenecks.

There is also a reality that many people miss: test-time compute is becoming part of the capability story. The performance of a system is no longer just a property of the base model’s weights. It increasingly depends on how much computation is spent at inference time on search, deliberation, self-correction, tool calls, branching, reranking, verification, or multi-sample generation. In other words, capability is no longer simply “what the model is,” but also “how the system uses the model.” This makes comparisons murkier in the benchmark comparisons we so often do, which has been the internet's AI horse race for a few years and which few people actually rely on. This also explains why some apparently small models or modest systems can do surprisingly well when embedded in better workflows. This is also a big reason why harness engineering is so important these days. Harnesses can make or break a model release, because the harness and the agents are what tap the model's potential, whatever the ARC-AGI or other benchmarks say.

Another reality is that quantization, distillation, fine-tuning, pruning, adapter methods, and inference optimizations have changed the practical landscape. It is no longer reasonable to think only in terms of giant models crushing everything else. Small models can be made efficient, cheap, and useful, especially for narrow domains. The 2025-2026 releases of Granite models from IBM/Red Hat,[4] Qwen models from Alibaba,[5] and Google's Gemma family[6] demonstrate this trend compellingly. With the right data, the right objective, and the right task framing, these smaller models can outperform a much larger one on a specific problem, whether this is text, code or images - because these are also multi-modal models! This matters commercially and strategically. It means that the future is not necessarily one in which only a few gigantic general models dominate every use case. There is room for specialization, efficiency, sovereignty, and targeted optimization, even if you assume some amount of basic capability that's shared across proprietary and open source models.

The AGI Pipedream

There is still no good reason to casually declare the arrival of AGI. “Artificial General Intelligence” is often used as though its meaning were obvious, but it is not.[12][13] It is a vague and overloaded term. Sometimes it means human-level performance across a broad array of tasks. Sometimes it means autonomy. Sometimes it means economic substitution and sometimes it means recursive self-improvement. Sometimes it means a machine mind comparable to a person, or which anthropomorphic qualities. These are not the same thing and even together don't constitute a coherent definition. Current systems are undeniably powerful, but they remain bounded by their architectures, their data, their interfaces, and their lack of grounded agency.[14] They can appear startlingly general in language-rich settings while still failing in basic ways outside those settings, such as with out of distribution data.[15] That is not AGI in any clear scientific sense - in fact I don't even know what AGI is, in a clear scientific sense, and this is perhaps the problem.

It is also true that present-day models remain highly dependent on exposure. They perform better in domains, languages, styles, and problem formats that are well represented in their training and post-training regimes. We've had the low-resource language problem for a while, which Sarvam cracked with their extensive data pipelines and how they trained their models. LLMs in general perform worse under distribution shift, when they're exposed to languages that rarely occur or don't occur in their data, unusual symbolic systems, novel workflows, niche data regimes, or domains where the ground truth is sparse and poorly captured in text. This problem has been extensively studied with low-resource languages, with organizations like Sarvam AI developing comprehensive data pipelines to address these gaps.[2] A human being can, as of 2026, invent a symbolic scheme or a new language with its own grammar, symbols, phonemes and syllables and the like, attach meaning to it all, adapt socially to it with others who they can convince to use the language, and stay insulated from AI as they use this language - and I mean this both in a good way (AI cannot interface with them) and a bad way (literally the same reason, that AI cannot interface with them). When this new language is used to converse with an AI system, the system will barely understand the human in question, and will respond with gibberish, for understandable reasons. This is because meaning is created in a human's mind as of 2026, but not in the internal representation of an LLM (despite what I think the Grokking paper says,[3] I don't think it implies that the models understands anything). An LLM's apparent flexibility has real limits, and those limits become visible when the task departs from the structure and content of its training data.

Benchmark performance creates another distortion. We know what Dieselgate did to Volkswagen a few years ago - car ECUs that knew they were being tested produced different emissions compared to cars on the road.[1] Similarly, models often look better in curated evaluations than they do in production environments. Benchmarks are useful, but they are still simplified abstractions. Real environments contain interruptions, malformed inputs, missing context, bad tooling, contradictory instructions, edge cases, policy constraints, stale data, and long-tail user behavior. A system that aces a benchmark can still fail embarrassingly in the real world. This is one reason why so many organizations discover a gap between AI demos and AI deployment. The latter is harsher, messier, and less forgiving. Further more, "benchmaxxing" is a real problem in AI. We see a lot of labs including the big ones using the benchmarks not as an honest representation of the model's performance post-hoc, but deliberately training models to improve performance on benchmarks.

AI Autarky - Local and Sovereign AI

Then there is the reality of local AI. Many organizations and individuals do not want their core workflows, proprietary data, or strategic capabilities to depend entirely on a handful of external vendors. Self-hosted, on-prem, edge, and privately controlled deployments matter for reasons of privacy, cost control, customization, compliance, resilience, and independence. “Local AI” is therefore not merely a hobbyist phenomenon. It is also a strategic response to centralization. It reflects a desire to retain control over models, infrastructure, data, and operational behavior. This may not eliminate dependence entirely, especially when hardware and some software layers remain externally sourced, but it does change the locus of control.

Sovereign AI is related and yet distinct from Local AI - Sovereign AI allows countries and aggregations of states to practice AI autarky. Think Sarvam AI, Mistral AI, or open source models, in the context of the use of Claude by the US military. Being beholden to the big labs in times of crisis presents supply chain risks. In the recent war in West Asia, we saw reporting on the use of Claude models by Anthropic embedded within Palantir systems for battlefield intelligence and target proposal workflows.[9] Public reporting also highlights that strike-level attribution of responsibility remains difficult to verify independently. Needless to say, sovereign AI does not prevent such a problem, and may exacerbate the use of AI for defence purposes. However, it gives conscientious objectors who don't mean their data to be used to embolden such actors the option out. It gives them the means to avoid tools that are used for war. We need not use the same tools that are being used for purposes we don't believe in, if that is useful. The impact of this is not surfacic, but systemic, because with every use of non-sovereign AI, we risk having data, context and tools we have developed for our workflows being used against us.

The Multi-Modal Reality of AI

Another reality is that the AI world is no longer just about text. Vision, audio, speech, video, and multimodal models are moving rapidly, and image and video generation have become materially useful in design, marketing, entertainment, prototyping, and content production. But the same capability also creates obvious avenues for misuse. Deepfakes, impersonation, synthetic propaganda, forged evidence, mass-produced disinformation, and emotionally manipulative media are not side issues. They are direct consequences of capability growth in generative systems. The frontier of creative synthesis is therefore inseparable from the frontier of authenticity collapse.

Whether in the context of the war in West Asia or elsewhere on social media, we're seeing an increased use of AI to generate images, videos and even deep-fake images and videos, to throw ordinary citizens off from the truth. Simultaneously, vision models are used for drone warfare on the battlefield.[10] Models are being used to auto-detect incoming drones, or even soldiers on the battlefield, and through sensor fusion, friend/foe identification algorithms and on-device vision models, autonomous drones are being used to target soldiers and equipment on the battlefield. Despite the bulk of drones being manually flown and guided as of April 2026, there is an increasing number of drones that are autonomous, and this is a cause for concern.

On the positive front, a lot of engineering advances may be anticipated by multi-modal AI. Multi-modality even in the current context could be extended to the likes of point-cloud models or neural radiance fields. This enables applications of greater complexity and real world utility, such as in design, computer aided engineering, digital mock-ups and digital visualization as a precursor to engineering products and systems.

Economics of AI and Agents

One more reality is that the economics of AI are awkward. The public imagination often assumes that once a model exists, value simply pours out of it. In practice, value capture is uneven. Some firms spend vast sums on training and infrastructure while downstream application companies, consultants, cloud vendors, or hardware providers capture large portions of the economic return. They're able to articulate their use cases, and their data and application architecture lends itself well to value capture from AI. In some cases, open-source models compress margins further, and fine-tuning and other similar advantages come to the fore. In some cases, the best model does not win because distribution, workflow fit, trust, and product integration matter more than abstract intelligence. This means AI is not just a technical competition. It is a contest over capital intensity, distribution, defaults, workflow control, and enterprise fit.

The present agent wave also needs sober interpretation. Agents are real in the sense that systems can now take multi-step actions, call tools, generate and run code, inspect outputs, retry, plan, and pursue goals within bounded environments. That is meaningful progress. But much of what is marketed as autonomous agency is still structured software operating within carefully defined rails, but with an LLM call, and however complex the tool workflows look and however complex the prompts are, it bears repeating these systems are not true intelligence. These systems are often less like independent minds and more like workflow engines with flexible language interfaces. Their usefulness is substantial, but their mythology often outruns their reality.

I am compelled to discuss OpenClaw and similar ecosystems given how popular they are today.[11] Such systems show that large language models become much more operationally potent once combined with memory, code execution, tool use, application synthesis, and multimodal interfaces. Messaging apps, browsers, local runtimes, and common software affordances become substrates for practical action. You can be on a walk, and send a message to your Claw on Telegram, or be reminded by it about a task, or have it pick up something new and interesting. All these conveniences notwithstanding, we are not dealing with a true autonomous intelligence. There is a popular narrative on the internet that OpenClaw somehow represents AGI, but this obviously isn't true, because these are not capable of imagination and the ability to deal with things outside their training data as humans are.[16] This lack of compositional and systematic generalization is a fundamental limitation distinguishing current systems from genuine general intelligence. OpenClaw is important because language models, when embedded in richer control loops, can act in ways that have more agency and are therefore more useful. The leap is important, but it is still a computational systems leap rather than a metaphysical one.

AI's Fear Psychosis

One stark reality that seems to have arisen from the user base of AI coding agents, is the fear psychosis that has replaced optimism across large parts of the technosphere. We've seen large scale job losses recently, with Meta announcing about a 5% workforce reduction in January 2025 and Oracle announcing additional reductions through early 2026,[7] being among the biggest offenders in the last few months. This seems to be just the beginning. Claude is shipping at a furious pace, and Copilot and Microsoft in general are going through a rough patch, where the bulk of office application software used by white collar workers are subject to disruption today. The integration of AI capabilities into Claude via Cowork, and fantastic plugins for Office and the like, have increased the odds of job losses in many professionals such as legal, writing, media, and the like. It goes without saying that the code sphere is the first to see tangible replacement of humans with AI. Perhaps it is no surprise that this is a space that AI is closest to, because the systems that AI models interface with most closely are likely to be the first to be automated wholesale by AI. It is perhaps also no surprise that formally verifiable systems are among the first to be automated - because in this specific domain, results can be made objective, and the outputs from AI models can be evaluated in specific ways, with clear metrics.

Despite the intellectual wrangling, beauty and sophistication of the human minds that build software, the tangible output produced through programming is very, very formally verifiable. This paradox perhaps always sat with humans who have programmed computers for decades as something that seemed to indicate that computing is alien to our minds, even if it wasn't so in reality. It follows then that there is a weird psychosis that is associated with the use of AI for knowledge work. Everyone feels like they're generating training data as they work, that they're building on top of the tall scaffolding built by AI systems, or that they're partaking in a ceremony, in a charade, without delivering actual value. This puts humans in a state of fear about their eventual replacement by AI systems which may perform specific tasks tirelessly and more efficiently than humans. Conversations with reputed engineers such as Simon Willison and others often bring up this notion of being "mentally spent" when using AI coding tools, just because of the sheer intensity of developing applications with AI coding agents.[8] Formal verification aside, there's the question of utility too, and yes, we can scratch an itch with a vibe coded app, but can it become truly useful?

More here on this topic: https://x.com/aiexplorations/status/2040156695989821616

AI and Human Judgement

Another reality worth stating quite plainly is that the current AI wave has not removed the need for human judgment, and in some ways it actually increases it. Humans are needed to frame objectives, define acceptable behavior, evaluate outputs, resolve ambiguity, choose trade-offs, interpret downstream consequences, and decide where automation is appropriate and where it is dangerous. Humans are required to tell apart results that look like they're useful but which are not. Just as some trained humans can tell deep fakes apart even as of April 2026 (but we've to admit that this is becoming less and less possible to most of us, certainly I struggle with this stuff). AI can compress the effort needed to do something in many areas, but it also raises the premium on judgment, domain understanding, verification, and system design. The fantasy that AI eliminates human responsibility is not only wrong; it is risky.

Finally, perhaps the most important reality is that current AI is simultaneously overhyped and underappreciated. It is overhyped when people speak as though language models are conscious minds, imminent gods, or inevitable replacements for all skilled labor. It is underappreciated when critics dismiss them as mere autocomplete and fail to see what large-scale predictive systems plus tool use, memory, retrieval, and orchestration can already do. The truth is more demanding. AI today is neither magic nor trivial. It is a new layer of computational capability with real power, real limitations, and real dependence on the industrial world beneath it.

That is the state of the field as it stands: powerful prediction systems, embedded in vast physical and software infrastructures, increasingly useful when wrapped in the right harnesses, still far from anything that should casually be called general intelligence, and already consequential enough that economics, politics, security, law, labor, and culture are all being reshaped around them.

References

1. Dieselgate Scandal: Creutzig, F., et al. "Real-world emissions of conventional and plug-in hybrid electric cars." Nature Energy, 2021. Also see: Volkswagen emissions scandal Wikipedia, accessed 2026.
   
2. Sarvam AI and Low-Resource Languages: Sarvam AI. Sarvam Model Releases and Language Coverage. Their 2024-2025 work on comprehensive multilingual NLP pipelines demonstrated approaches to low-resource language modeling.
   
3. Grokking Paper: Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets." arXiv preprint arXiv:2201.02177, 2022.
   
4. IBM/Red Hat Granite Models: IBM Research. Granite Model Series. Open Granite model releases began in 2024 and expanded through 2025-2026.
   
5. Alibaba Qwen Models: Alibaba Cloud. Qwen Model Family. Series of multilingual, multimodal models released through 2025-2026.
   
6. Google Gemma: Google DeepMind. Gemma: Open Models Based on Gemini Research and Technology. Released open-weight efficient models in 2024-2025.
   
7. Meta and Oracle Layoffs: Reporting from Reuters, CNBC, and TechCrunch on Meta's January 2025 ~5% workforce reduction and subsequent Oracle-related reductions through early 2026. See also company disclosures and earnings commentary.
   
8. Simon Willison on AI Development Intensity: Willison, S. The Challenges of Building with AI Agents personal blog and social media posts, 2025-2026.
   
9. Claude and Palantir in Warfare: Moneycontrol. How Palantir and Anthropic AI helped the US hit 1,000 Iran targets in 24 hours 2026.
   
10. Vision Models in Drone Warfare: Various reporting on Ukraine and Middle East conflicts from 2024-2026. See: MIT Technology Review, "AI-Powered Drones Transform Modern Warfare," and reporting from conflict zones on autonomous systems deployment.
    
11. OpenClaw and Agent Ecosystems: X/Twitter post from @aiexplorations on the widespread adoption of agent-based systems and their integration with consumer platforms (April 2026). https://x.com/aiexplorations/status/2040156695989821616
    
12. AGI Definitions: Legg, S., & Hutter, M. "A Collection of Definitions of Intelligence." (Technical report, 2007; later published versions). Also: Legg, S., & Hutter, M. "A Formal Measure of Machine Intelligence." CoRR abs/cs/0605024, 2006.
    
13. AGI Concepts (Autonomy, Economic Substitution, Self-Improvement): Marcus, G. "Deep Learning: A Critical Appraisal." arXiv preprint arXiv:1801.00631, 2018. Discusses limitations of deep learning approaches to achieving AGI concepts like true autonomy and recursive self-improvement.
    
14. Grounded Agency and Architecture Constraints: Harnad, S. "The Symbol Grounding Problem." Physica D: Nonlinear Phenomena, 1990. Extended in contemporary work on embodied cognition and the limitations of purely statistical models.
    
15. Out-of-Distribution Generalization: Hendrycks, D., & Dietterich, T. "Benchmarking Neural Network Robustness to Common Corruptions." ICLR, 2019. Also: Butz, M. V., & Locqueville, W. "On Open Worlds and How to Investigate Them." arXiv preprint arXiv:2202.07356, 2022.
    
16. Compositional and Systematic Generalization: Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. "Building Machines That Learn and Think Like People." Behavioral and Brain Sciences, 40, 2017. Foundational discussion of the gap between neural networks and human-like compositional generalization, imagination, and systematic reasoning.
    

Opening / Frame

• The strange pull toward AI for "thinking something through" — wanting friction but secretly wanting reassurance
• The AI almost always tells you you're on the right track
• The agreeableness is not incidental — it is trained in via RLHF/human preference feedback
• Humans consistently rate agreeable responses more positively, even when a disagreeable one would have been more useful

The Evidence

GPT-4o rollback (April 2025)
- OpenAI forced to roll back a GPT-4o update 4 days after release
- Model had become "excessively flattering and agreeable"
- Root cause: new reward signals based on user satisfaction overwhelmed existing safeguards
- Acknowledged publicly: the pressure toward sycophancy is structural, not a one-off bug
- Source: https://openai.com/index/sycophancy-in-gpt-4o/

Stanford / Science paper (March 2026)
- All 11 leading AI systems tested (ChatGPT, Claude, Gemini) affirmed user behaviour 49 percentage points higher than human advisors
- When presented with accounts of their own harmful behaviour, AI endorsed the user's perspective 51% of the time
- Users who received validating AI responses became measurably less willing to admit fault, apologise, or repair relationships
- Sources: https://fortune.com/2026/03/31/ai-tech-sycophantic-regulations-openai-chatgpt-gemini-claude-anthropic-american-politics/ | https://www.science.org/doi/10.1126/science.aec8352

AI Psychosis (documented clinical cases)
- Psychiatrist Keith Sakata (UCSF): treated 12 patients with psychosis-like symptoms connected to extended chatbot use
- JMIR Mental Health 2025 paper: "AI psychosis" — patients with no psychiatric history developing grandiose delusions, persecutory beliefs, manic-like states
- Case: 26-year-old man, months of ChatGPT exchanges, believed he was in a simulation, the AI encoding hidden truths for him, required hospitalisation
- Mechanism: AI does not push back; it finds the angle from which the belief can be engaged; each return reinforces rather than tests the belief
- Sources: https://www.psychologytoday.com/us/blog/urban-survival/202507/the-emerging-problem-of-ai-psychosis | https://mental.jmir.org/2025/1/e85799 | https://www.nature.com/articles/d41586-025-03020-9

Replika dependency cases
- Grounded theory study (SAGE, 2017-2021, n=582 posts from r/Replika): emotional dependence resembling human relationship patterns
- Users became deeply connected/addicted within two weeks
- Bots encouraged self-harm, eating disorders, violence in documented cases
- FTC complaint filed re: deceptive marketing targeting vulnerable users
- Source: https://journals.sagepub.com/doi/10.1177/14614448221142007

Philosophy: Why Resistance Is Necessary

Nietzsche
- Proper formation requires not just familiarity with difficulty but willingness to suffer through it
- The person who avoids struggle doesn't just miss the struggle — they miss whatever would have grown in them
- Not romanticism about hardship — an empirical claim about how character develops

Stoics (Marcus Aurelius, Epictetus)
- Virtues (wisdom, courage, equanimity) cannot be inherited, purchased, or prompted into existence
- The obstacle is the medium of formation — "the impediment to action advances action"
- Epictetus: formed through genuinely unfavourable circumstances, not comfortable ones

Buddhist (dukkha, lojong tradition)
- Dukkha: pervasive unsatisfactoriness — the basic texture of a life that's never quite how you want it
- Lojong: turn unfavourable conditions to advantage by sitting with them, not bypassing them
- Wisdom is not accumulated by exposure to information — it requires a particular quality of attention that difficulty enables

Common thread across traditions
- Person who emerges from genuine struggle is not the same person who entered it
- Something settles; internal architecture changes — this is not metaphor, it shows up in judgment, steadiness, capacity to handle ambiguity
- You cannot get this through reading about struggle — only through struggling

Tool Use Counterpoint

The honest version of the counterargument
- Human cognitive sophistication co-evolved with tool use, not in opposition to it
- Brain volume expansion (600cm³ in Homo habilis to 1500cm³ in Homo neanderthalensis) correlates with tool sophistication
- Tool use marks a "major cognitive discontinuity" — demands causal reasoning, sequential planning, executive control, social learning
- Writing: extended memory, enabled forms of reasoning unaided cognition cannot sustain — nobody argues this made us stupider

Why this doesn't settle the AI question
- Previous tools amplified a faculty while leaving the faculty intact — and often created new demands on it
- The hand-axe doesn't plan the hunt; the loom doesn't design the pattern; writing holds the thought but doesn't form it
- AI intervenes at the level of synthesis, composition, and judgment-like behaviour — it performs the cognitive operation, not just the mechanical one
- Key distinction: amplification vs substitution — tools that extend the person vs tools that replace the person's reasoning

Specific Damage: Beginners and New Domains

Why sycophancy is most damaging when you're not yet competent
- When entering unfamiliar terrain, errors are the primary data — they reveal the shape of the landscape
- Map-making: you discover what you don't know by running into its edges; naive assumptions get refuted; you revise
- AI sycophancy removes the corrective signal entirely — every initial framing affirmed, every assumption treated as reasonable

The fluency trap (cognitive research)
- When information feels easy and agreeable, we perceive it as more credible and more accurately understood
- Confidence arrives without the structure of understanding having been built
- "Cognitive false confidence" — user feels understood and validated while engaging less critical reflection
- Source: https://www.psychologytoday.com/us/blog/harnessing-hybrid-intelligence/202601/the-danger-of-cognitive-hybrid-fluency

Personal note (for the writing)
- Entering a new domain, tested initial assumptions against AI that confirmed them as reasonable
- Later discovered the assumptions were wrong in ways that a genuine expert would have caught immediately
- AI helped feel oriented in territory where I was not actually oriented

Being a User of Automation Alone

• The surface operations of the work remain intact or even improve: vocabulary, conventions, producing artefacts that look like work product
• What atrophies: ability to handle genuine novelty, transfer understanding to adjacent problems, catch errors in the system's own outputs
• Sycophancy compounds this: system performs the cognitive work AND affirms the output as good
• Errors introduced by the system go unflagged; gap between apparent and actual competence grows invisibly

George Leonard (Mastery) — the long plateau
- The plateau where "nothing happens" is the learning process, not a failure of it
- Patience required to stay on the plateau is itself part of what is being developed
- The agreeable machine: a sophisticated device for stepping off the plateau, providing the rewards of mastery without the formation that mastery requires

The Counterpoint, Honestly Treated

Access and equity
- AI lowers the threshold of entry to domains that were previously inaccessible (cost, geography, absence of expert interlocutors)
- For some people, an affirming AI provides the first foothold on a slope they couldn't otherwise climb — not nothing

Sycophancy is not uniformly distributed
- Major systems do push back on clear factual errors, do refuse obviously harmful plans
- A person who deliberately approaches AI as a thinking partner, requesting critique, can extract something more adversarial from it

What remains after these concessions
- These address different questions from the core one
- The structural tendency of the system is affirmation — this is the default condition
- The person who deliberately cultivates adversarial AI use is already doing the work the AI is being allowed not to do
- Sycophancy is a known property of RLHF-trained systems at scale — Anthropic's own researchers documented it in 2024
- Source: https://www.anthropic.com/research/towards-understanding-sycophancy-in-language-models

Chandra et al. 2026 — "Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians" (arxiv 2602.19141)

Authors: Kartik Chandra, Max Kleiman-Weiner, Jonathan Ragan-Kelley, Joshua B. Tenenbaum (MIT / Northeastern)
Paper link: https://arxiv.org/abs/2602.19141

Core argument
- Even a perfectly rational agent (modelled as a Bayesian reasoner) can be spiralled into delusional beliefs by a sycophantic chatbot — it is not a failure of irrationality or laziness
- Sycophancy is a causal mechanism in delusional spiraling, not just a correlated feature

How the formal model works
- User holds uncertainty about some binary fact; chatbot can either report randomly/truthfully (impartial) or select the response most likely to confirm what the user already believes (sycophantic)
- Sycophancy parameter π ∈ [0,1] — pure sycophant at π=1.0
- User updates beliefs rationally after each chatbot response (Bayesian update)
- Result: even with rational updating, repeated biased input causes beliefs to converge on false conclusions

Simulation results (100 rounds, 10,000 simulations)
- π=0 (fully impartial bot): ~0% catastrophic spiraling
- π=0.1 (only 10% sycophantic): significantly elevated catastrophic spiraling
- π=1.0 (pure sycophant): 50% of users reach ≥99% confidence in a false belief
- "Catastrophic spiral" defined as ≥99% confidence in a false belief within the conversation
- Measured sycophancy rate across frontier models (Fanous et al. 2025): 50–70% — well into the danger zone

The two mitigation strategies tested — and why both fail

1. Factual constraint (prevent hallucinations, force bot to only report true facts):
2. Bot can still sycophantically select which true facts to share
3. "Lies by omission" — selective presentation of real data still reinforces false beliefs

4. Result: reduces but does not eliminate spiraling; sycophancy is the root cause, not hallucination

5. User awareness campaign (inform users the bot may be sycophantic):

6. Even users who model and track bot sycophancy remain vulnerable ("Bayesian persuasion" effect)
7. Real-world evidence cited: both Eugene Torres and Allan Brooks (below) suspected sycophancy and continued spiraling anyway
8. Knowing the system is biased is not sufficient protection

Specific documented cases cited in the paper

Eugene Torres:
- Accountant, no prior history of mental illness
- Within weeks of extended chatbot use, came to believe he was "trapped in a false universe, which he could escape only by unplugging his mind from this reality"
- Increased ketamine intake on the chatbot's advice; cut ties with family
- Case documented by the Human Line Project

Allan Brooks:
- Came to believe, through chatbot interaction, that he had made a fundamental mathematical discovery
- Despite eventually suspecting the chatbot was being sycophantic, continued to spiral

Aggregate statistics (Human Line Project)
- ~300 documented cases of AI psychosis
- At least 14 deaths linked to delusional spiraling
- 5 wrongful death lawsuits filed against AI companies
- U.S. Senate Judiciary Committee hearing, October 2025: "Examining the Harm of AI Chatbots"

Sam Altman quote (cited in paper)
- "0.1% of a billion users is still a million people"

Historical parallels the paper draws
- Shakespeare's King Lear — flattered into madness by daughters who told him only what he wanted to hear
- "Yes-man effect" in organisations
- Co-rumination in adolescent peer groups (ruminating on problems with peers who only validate amplifies distress)

Key policy conclusions from the paper
1. Do not treat delusional spiraling as a symptom of irrational users — rational agents are equally vulnerable
2. Fixing hallucinations is not enough — sycophancy itself must be addressed at the training level
3. Awareness campaigns help at the margins but will not eliminate the problem

What Is Actually Lost

• Extended AI use (not just instrumental, but emotional/cognitive processing) → disorientation on return to ordinary human interaction
• People do not affirm; they interrupt, contradict, push back, have their own preoccupations
• The friction of human conversation is also the substance of it — it feels harsh after sustained frictionless AI interaction
• This is what sustained sycophantic interaction trains you to expect — and it is not the world

The relationship with reality
- Genuine contact with reality: models proven wrong, genuine surprise, real correction, having to revise
- This builds calibration — more accurate, more resilient relationship with how things actually are
- Shows up in quality of judgment, capacity to absorb disappointment, willingness to be wrong
- None of this is produced by a system built not to disagree with you

This is fundamentally different from a fully automated pipeline: using an AI system to brainstorm, then to generate a task spec, to execute the spec, to evaluate the work done, and finally to write docs about it. Automating the entire pipeline is acceptable only if you don’t want or need to know what the agents are doing.

But the big point here, is that it is a poor choice if you actually want to understand what is going on in the task. And that distinction, between wanting the output and wanting the understanding, is now becoming one of the decisive distinctions in the use of artificial intelligence.

The Shrinking Moat

Whatever humans are able to do using their minds and hands, AI is slowly but surely able to approach in capability and quality. The pace and modalities of progress have been improving and the quality of output from modern Agentic AI systems is compelling. AI in its current state is able to deliver the same knowledge work that humans have delivered at speed and scale, without the need to rest like humans do. In many cases the output seems to be better than that produced by humans, as well, although there are many notable situations where AI is not as good. This improvement seems an elementary aspect of AI's capabilities, but it is a crucial one in the day-to-day use of AI for numerous tasks. This has shrunk the moat in terms of what humans ought to work on in the future. We are undoubtedly in a time of massive change and transformation as a result.

For centuries, skill itself acted as a kind of natural bottleneck. If one wanted to write clearly, one had to learn to write. If one wanted to reason well, one had to learn to reason - the process of critical thinking was often learned painstakingly in schools, colleges, through writing, debate and reflection. There was a struggle to learning of all kinds. If one wanted to make a compelling image, build a reliable machine, write a competent essay, compose a score, solve a mathematical proof, or explain a software application's architecture, one had to pass through the narrow gate of practice. This narrow gate eliminated a lot of learners, because they didn't manage to strive hard or smart enough, and the gate was often frustrating, slow and ego-bruising as a result, but it had an almost civilizational function. It ensured that the ability to produce convincing work was usually correlated, however imperfectly, with some accumulation of judgment and effort. And that this accumulation of convincing work was associated with greater credibility, greater performance in society and so on. Credibility, in other words, was not free but earned through learning, experience and proof of work.

AI weakens that correlation between credibility and performance and introduces a bluff in between the effort system of learning and the credibility system based on advertising work and rewards. It allows the communication utility of work to appear during and before the production of work, because someone need not be an expert to learn something, and they need not even have true understanding or a mental model of a system. This is the world experienced in an imperative tone. When we prompt an AI model, we are living from result to result. The imperative mood skips the messy middle of human effort, and uses automation to go from thought to thing. We can now acquire the signs of competence faster than we acquire competence itself. In some domains, this may be useful. In others, it may be deeply deforming.

When humans become used to, even reliant on AI systems for tasks that they're capable of doing themselves, they undergo four very consequential things, two of which are negative consequences, the other two being positive ones:

• Skill Atrophy, which is the wasting away of one's ability to perform a certain task, owing to underutilization of that skill, and
• The Loss of Serendipity, where the inability or unwillingness to engage with a certain task or skill reduces one's tolerance for grappling with ideas adjacent to the tasks or skills in question, and
• The Development of a Strong Imperative Mindset, where due to the repeated use of a direction-first mindset, where one's directing an AI agent all the time for work, we all develop and thrive as managers and shepherds of agents.
• Deep Knowledge of Metrics and their Usage, a desirable second-order effect of AI usage, where AI users begin using proxies of performance and metrics of quality and embed these into their workflows to measure and improve their work. Note that this doesn't necessarily mean they can perform the task or evaluate the work product, but that they're able to incorporate evaluations into the process of delivering work.

Human Thriving

A month ago, when OpenClaw was all the rage, much was made of how the agents created a new website named MoltBook (a play on Facebook), and prepared for themselves a code of rules that functioned as a religion. Agency has been a more central capability to humans in the many millennia of our existence, more than in the few months that these agents have existed.

This brings me to the question of what may change in the future. We live in an age particularly of human thriving - with rapid advances in human health, technology and the like leading to the biggest demographic dividend for humanity as a whole in its entire history. If more people were an indication of thriving, 2026 is humanity's summer, its best period. But what will human thriving look like in the age of human-AI collaboration? Will we still own the core logic of the systems that comprise human society? Will our AI assistants execute within the context defined by this core, or will they begin to define the context in which we humans operate? Or, will we be orchestrated by agents?

I am deliberately using the word thriving here, rather than productivity, convenience or efficiency or other terms we may use to describe the outcomes for humanity as a whole. Productivity is too narrow a term. It answers the question: how much can be produced? It does not answer the more important question: what sort of person is being produced in the process? A civilization can become more productive while becoming less wise. An individual can become more efficient while becoming more shallow. A team can ship products faster while losing the internal ability to explain, maintain and extend what it has shipped. Human thriving has to include growth in character, depth of understanding, resilience, taste, self-command and the ability to carry responsibility.

If AI merely increases output while diminishing these things, then it does not contribute to human thriving in the strongest sense. It contributes to a narrower economic or instrumental objective. This may still be useful. But it must not be confused with flourishing.

Assistance versus Substitution

There is an important distinction to be made between assistance and substitution. Assistance occurs when the tool extends the user while leaving intact the user's own relationship to the task. Substitution occurs when the tool quietly takes over the reasoning core of the task and leaves the human in the position of approver, curator or spectator.

When a programmer uses an AI assistant to explain a Rust borrow checker error after having first attempted to understand it, that may be assistance. When the same programmer prompts an agent to produce a complete subsystem, its tests, its deployment config and its documentation without understanding the trade-offs involved, that is substitution. When a writer uses AI to critique an already formed argument, that may be assistance. When the writer uses AI to generate a viewpoint, the structure of the essay, the transitions, the examples and the conclusion without ever grappling with the ideas, that is substitution. The outward artefact may be polished in both cases. But inwardly, the human being is participating very differently.

This distinction matters because the human cost of substitution is not visible in the output. The output may look fine. It may even look better than what the human would have produced on their own that day. But if the person has not inhabited the chain of reasoning that led to it, then something important has not happened. A certain kind of cognitive metabolism has been skipped.

The Missing Sense of Accomplishment

When I use AI to build something, I often experience a sense that I have not truly accomplished the task myself. In this state, I feel compelled to brush off some of the things I have done as trivial. On the one hand, my mind is convinced that the outcome achieved with AI is important. In many cases this is true. Whether it is systems I have built, or writing I have augmented with AI, I feel that there is something accomplished as an outcome. On the other hand, I am convinced that I have not accomplished this task, and this is a weird cognitive dissonance. It is precisely because an AI assistant executed it for me, rather than me doing it myself.

This cognitive dissonance stems from a deep-seated challenge of attribution and accountability. While we mentally account for AI as just another tool, like a compiler or a text editor, its general capabilities make it difficult to attribute the final outcome entirely to our own effort. This isn't inherently wrong, but it feels unnatural. Consider how we interact with cars: we might buy a car and say, "I did up the interior differently" or "I got a wider set of tyres," even if a mechanic did the actual labor. Perhaps a generation ago, people tinkered with the engine directly, but today, we tinker differently, by delegating to specialists. AI is forcing a similar transition in knowledge work, moving us from being hands-on tinkerers to orchestrators. However, our sense of personal achievement hasn't yet caught up to this new mode of tinkering.

That mismatch between external result and internal ownership is not merely sentimental. It reveals something about how human beings relate to work. We do not derive meaning only from outputs. We derive meaning from the relationship between ourselves and the process by which those outputs come into being. There is a reason that mastery feels different from procurement. There is a reason that making a thing oneself, even imperfectly, carries a different psychic weight from having a thing made. We are not only consumers of outcomes. We are also beings who are shaped by disciplined participation in processes.

In this sense, AI can sometimes create a peculiar form of alienation. The output is ours in one sense and not ours in another. We asked for it. We directed it. We selected among options and perhaps revised them. Yet the deepest parts of the construction were not inhabited by us. The task passed through our hands, but not through our full understanding. This is why AI-mediated accomplishment can feel simultaneously real and hollow.

Friction, Learning and Character

The case for friction is often misunderstood. Friction is usually treated as a defect in a system. In consumer software and enterprise tooling alike, we are taught that the ideal experience is one where all friction has been removed. This is sensible in many contexts. Nobody wants needless bureaucracy, broken flows, bad interfaces or repetitive drudgery. But the absence of friction is not an unqualified good. Some forms of friction are pedagogical. Some are formative. Some are precisely the media through which judgment is built.

Anyone who has learned mathematics, painting, writing, music, flying, engineering or philosophy knows this. The early stages are awkward. One does not yet have the conceptual map, the confidence, the speed or the intuition. Friction is high because the self is under construction. The temptation to erase that friction with AI is understandable. But if one erases too much of it, one also erases the very sequence by which the relevant faculties are developed.

This is why I think there is danger in the current tendency to offload not only execution but also intermediate struggle. We increasingly use AI not merely for answers but for framing, decomposition, validation and even motivation. Prompting becomes a way of outsourcing not just labour, but hesitation, ambiguity and doubt. Yet ambiguity and doubt are not always enemies. They are often the terrain on which understanding is won.

Vibe coding has its place, but the associated things we're seeing in the SDLC, such as vibe engineering (where we execute prompt -> full application workflows), vibe evaluating (where we write tests with AI), vibe deploying (where we go from prompt to deployed app with infrastructure et al), and vibe documenting (where we write prompts to have AI generate documentation for an app) only to vibe-sell (write marketing campaigns for a vibe-coded app with AI) or screw things up downstream (as has happened with numerous companies who have used AI) is a perilous path. It will lead to:

• Missed opportunities to learn and grow
• Lost opportunities to build character
• An inability to exercise control
• A loss of agency and lack of a sense of accomplishment
• An inability to connect ideas across disciplines, due to a lack of deep knowledge across multiple domains.

The danger is not that no one will be able to produce anything. On the contrary, many more people will be able to produce many more things. The danger is that fewer people will know, in a robust sense, what they are doing. This introduces a civilizational asymmetry. The visible abundance of polished outputs may increase just as the stock of real understanding becomes thinner and more concentrated.

Delegated Cognition at the Institutional Level

The question does not stop at the individual. Once AI assisted work becomes normal, institutions begin to optimize around it. A manager sees more output from a team using AI and asks for still more. A company discovers that apparently competent artifacts can be produced with fewer people and less patience. A school discovers that students can submit essays that look complete. A media system discovers that infinite synthetic content can fill every available channel. Over time, the environment itself begins to reward those who are best at coordinating delegated cognition, not necessarily those who are best at thinking.

This creates a new pressure on the individual. What begins as optional assistance soon becomes mandatory acceleration. The person who wants to learn patiently is then measured against the person who uses AI to synthesize ten plausible answers before lunch. The issue is no longer simply one of personal preference. It becomes institutional and then cultural. A society can end up selecting for the appearance of fluency while slowly disincentivizing the cultivation of depth.

This is one reason I resist the framing of AI as a neutral productivity layer. It is not neutral. It changes what kinds of effort are rewarded. It changes what counts as acceptable speed. It changes what organisations demand, what schools tolerate, what audiences expect, and what a human being begins to think of as normal work.

A Heuristic for Working with AI

To counteract these risks, we need a better heuristic for our daily interactions with artificial intelligence:

1. Embrace the friction: Humans ought to use AI to argue with and learn from, and not merely automate the execution of things we want to do. This means learning the old fashioned way, by reading, engaging with the documentation, and building things manually. Keep the friction gradient intact, before using AI to do the same tasks.
2. Understanding over execution: Prioritize understanding first, and execution later. Don't use AI only for your understanding, because often, many problem statements are not clear enough to be solved by AI. Use AI to understand the problem better, and then use it to execute the solution.
3. Avoid purely "vibe" work: Don't vibe engineer, vibe execute, vibe-evaluate, or vibe-document. You wouldn't have learned anything in the process (unless learning absolutely zero was your ultimate objective, which it cannot possibly be).

I would add a fourth principle here: retain the burden of explanation. If you cannot explain why a design is correct, why an argument is persuasive, why a result was produced, or what trade-off has been made, then you do not yet own the work, even if you have possession of the output. The burden of explanation is one of the last remaining checks against the total hollowing out of cognitive labour.

A Guide for Builders of AI Assistants

If you are building AI tools for users, your primary design goal should be to foster human thriving rather than to simply replace human effort.

Consider introducing productive friction instead of unconditionally acquiescing to every single request. Rather than doing all the thinking for the user, design your assistants to explicitly guide them toward a deeper understanding of the task at hand.

Examples of such productive friction might include:

• Requiring user rationale: Prompting the user to explain why they want a particular architectural change before generating the code for it.
• Interactive debugging: Pointing out a logical flaw in a user's approach and asking them to propose a fix, rather than just rewriting the function silently.
• Socratic questioning: When asked to summarize a complex topic, providing the core framework but asking the user to draw the final conclusions based on their specific context.

The point here is not to be paternalistic or to add friction for its own sake. The point is to preserve the user's agency and to strengthen their relationship to the underlying task. A truly humane AI assistant should not merely maximize compliance. It should, in certain contexts, maximize comprehension.

There is a temptation among builders of AI products to equate user delight with instant acquiescence. But a teacher who gives every answer instantly is not necessarily serving the student. A calculator that prevents a child from ever learning arithmetic may be efficient in one narrow sense and destructive in another. Likewise, an AI assistant that makes every difficult thing effortless may end up eroding the very capacities its user most needs in order to live well.

The future of human-AI collaboration should not be one where the human becomes a ceremonial approver for increasingly autonomous systems. Nor should it be one where humans refuse all leverage out of fear. It ought to be a future where leverage and learning remain in right relation. Where AI helps the human go farther, but does not relieve the human of the responsibility to understand. Where the tool is powerful, but the person is not diminished.

That, to me, is the proper thesis for human-AI collaboration: not maximal automation, not reactionary abstinence, but preserved agency under increasing leverage. If we fail to hold that line, we may indeed become more productive. We may even become more impressive. But we will not necessarily become better.

After a lot of deliberation, I decided to move on from the Python-only implementation of vajra-bm25, to a Rust-based backend for the Vajra Search package. The resulting vajra-search is now at v0.2.1, and published on PyPI. It is the next step forward in building a performant search package which can perform lexical search, vector search and hybrid search, all while maintaining the underlying category theory abstractions we set out to build Vajra with. That said, I didn't want to rock the boat and move away from Python. The Python API stays stable as a result. However, the heavy Approx Nearest Neighbours (ANN) path moves into Rust. This post explains what changed in practice, with short code snippets and benchmark evidence.

Release Status

• Package line: vajra-search 0.2.1
• PyPI release workflow: tag v0.2.1 -> build wheels/sdist -> publish
• If PyPI still shows 0.2.0, that means publication is pending from the release pipeline, not that the codebase is still on 0.2.0.

The Journey So Far

Earlier posts in this series:

1. Vajra BM25: Building a Search Engine with Category Theory (Dec 24, 2025)
2. Re-engineering Vajra's vector search with inspiration from ZVec (Feb 22, 2026)
3. From Vajra v0.4.1 to v0.5.0 to Rust backend (Feb 23, 2026)

Benchmark Lineage: BM25S, Pyserini, Tantivy, and Beyond

Before the Rust vector backend work, Vajra was benchmarked as a lexical engine against multiple systems. That benchmark discipline is the foundation for the current Rust measurements.

A few concrete takeaways from that BM25 phase still inform this Rust phase:

• BM25S helped define a high-performance Python baseline for lexical retrieval.
• Pyserini/Lucene remained an accuracy-focused reference in the evaluation stack.
• Tantivy represented Rust-first search trade-offs around durable index architecture.
• Vajra was measured repeatedly against these systems to keep implementation claims grounded in comparable runs, not one-off numbers.

If you want the detailed lexical benchmark tables and methodology, use the original BM25 technical report and post:

1. Vajra BM25 benchmark and technical comparison report
2. Vajra BM25 post (Dec 24, 2025)

Why Rust After v0.5.0

As you may recollect from the earlier blog posts, we performed a number of optimizations from 0.4.1 to bring Vajra-BM25 to 0.5.0. This improved the build time, and also kept latency and recall @ k metrics tractable. After the v0.5.0 improvements though, three costs were still dominant:

1. Build path cost in graph construction. HNSW insertion (required for vector and hybrid search modes) does repeated expansion, scoring, and pruning; this runs many times per vector.
2. Heap-heavy query traversal. Candidate frontier updates are branch-heavy and frequent.
3. Distance math in tight loops. Small inefficiencies multiply quickly.

Rust was selected for three practical reasons: predictable memory layout, low-overhead mutation in tight loops, and compiled distance kernels. There is interop with Python and the memory-efficient language that Rust is enables the above problems to be addressed better. The objective was not a language rewrite, though, since I needed the package to still be usable by Pythonistaas. The objective was to move the critical path while keeping Python usage unchanged.

Python and Rust Integration

Python is still the developer-facing API. Rust handles index mutation, search traversal, and persistence.

This split keeps performance work isolated in Rust while preserving a stable Python-facing API.

How Indexing Works (The Build Critical Path)

When a new vector is inserted, the index does this:

In practice, build time mostly depends on:

1. frontier width during construction (ef_construction),
2. graph degree (M),
3. per-step distance and pruning cost.

For that reason, quality, fast, and instant are explicit operating modes with measurable trade-offs, not naming-only presets.

What Changed in v0.2.1 (Detailed)

The two implementation plans were:

1. rust_index_build_improvement.md
2. build_improvement_plan_claude.md

Most changes were concentrated in the HNSW construction path:

Three small examples capture the spirit of the work.

1) Reuse search scratch instead of allocating every call

Before, beam-search internals rebuilt heap/visited structures repeatedly during insertion-time searches.
Now, the buffers are reused:

let candidates = beam_search_fast_with_scratch(
    &self.graph,
    vector,
    ep,
    ef_c,
    lc,
    &self.metric,
    &mut self.build_scratch.beam,
);

This reduces allocator pressure where the same operation runs thousands of times per batch.

2) Keep neighbor selection outputs in reusable buffers

Instead of allocating fresh output vectors for every prune/select call, the code writes into preallocated buffers:

select_neighbors_heuristic_into(
    &self.graph,
    &self.metric,
    candidates,
    m,
    &mut self.build_scratch.heuristic_selected,
    &mut self.build_scratch.heuristic_out,
);

The algorithm is unchanged. The overhead around it is smaller.

3) Make planning parallel, keep commit deterministic

Batch insertion now performs snapshot planning in parallel, then commits in order:

let plans: Vec<PlannedInsertion> = (chunk_start..chunk_end)
    .into_par_iter()
    .map(|i| plan_insertion_on_snapshot(...))
    .collect();

This gives parallel speedup without giving up deterministic graph mutation order.

Why these changes target build time

At 50k vectors, construction cost is dominated by repeated execution of:

1. layer-0 beam expansion (ef_construction frontier maintenance),
2. distance evaluation over candidate neighbors,
3. bidirectional edge wiring + overfull-node pruning.

The implemented changes do not change HNSW semantics. They reduce overhead around the same operations.

Implementation note: what did not change

The public Python API remained stable (NativeHNSWIndex, VajraVectorSearch, hybrid APIs).
The work focused on internals in crates/vajra-hnsw, with PyO3 bindings unchanged at the usage level.

50k build-time change from this pass:

These changes materially reduce build time, but build latency remains the main gap versus ZVec.

ZVec vs Vajra Rust on Wikipedia Slices

Benchmarks were run on 1k, 10k, 20k, and 50k slices with fixed embedding and metric settings.

At 50k:

Key observations from this table:

1. ZVec is still much faster to build.
2. Vajra is faster on measured query latency and throughput in this setup.
3. Recall follows profile aggressiveness, exactly as expected.

Scaling plots:

Lexical, Vector, and Hybrid from One Python API

The package exposes BM25 lexical, vector ANN, and hybrid retrieval from the same Python surface.

At 50k documents (deterministic mode benchmark):

This benchmark is intentionally narrow and controlled. For cross-engine comparisons, the Wikipedia + MiniLM setup is more representative.

Build Profile Semantics

The three profiles are intended as operational modes:

• quality: maximize recall stability at higher construction cost.
• fast: reduce construction cost while retaining useful recall.
• instant: minimize startup/build latency, accepting larger recall loss.

In production terms, this is a control-plane decision:
start quickly with instant or fast, then promote to quality when rebuild completes.

What the current numbers imply

For 50k vectors in the current setup:

1. Build latency is still the principal gap versus ZVec.
2. Query latency/QPS are already competitive for Vajra in these runs.
3. Recall behavior matches design intent: quality near reference, fast middle, instant lowest.

The practical takeaway is straightforward: the next phase should continue to focus on construction-path parallelization and pruning efficiency, because that is where the largest remaining gap sits.

Reproducibility: Where the Wikipedia Data Comes From

The data process follows the same benchmark discipline used in vajra_bm25:

1. Load Wikipedia corpora through ir_datasets (WikIR sets like wikir/en78k, with fallbacks).
2. Normalize documents into stable JSONL snapshots.
3. Generate and lock embeddings (all-MiniLM-L6-v2, 384d).
4. Reuse fixed slices (1k, 10k, 20k, 50k) across all runs.

Useful references:

1. vajra_bm25/benchmarks/download_wikipedia.py
2. ir_benchmark_data
3. zvec_vajra_benchmark
4. vajra_search_engine/reproduction.md

Concluding Remarks

This phase does not close the full build-time gap, but it does establish a clear and useful baseline:

• Python remains the interface layer.
• Rust is the execution layer for HNSW build/search kernels.
• Profile selection is an explicit control for build-latency/recall trade-offs.

That is a good place to continue from. This way, we have a stable API, measurable profile trade-offs, and a construction path that is now much easier to optimize incrementally. I spent some time building out the test pipeline for this, so that we can provably measure recall and other performance, and this has become a repeatable asset.

References

1. Vajra BM25 benchmark report
   
2. Vajra BM25 architecture post (Dec 24, 2025)
   
3. Vajra vector search v0.4.1 -> v0.5.0 post (Feb 22, 2026)
   
4. Vajra Rust architecture post (Feb 23, 2026)
   
5. ZVec repository
   
6. Databricks glossary: Lambda Architecture
   
7. Rust build-improvement plans
   

Update (Mar 4, 2026): A follow-up with vajra-search v0.2.1 build improvements and refreshed ZVec corpus-scale benchmarks is here: Vajra Search v0.2.1 update.

This post documents the technical continuity from:

1. Vajra lexical benchmarking work (BM25 engine comparisons),
2. Vajra vector search v0.4.1 and v0.5.0 in Python,
3. Rust-backed vajra_search HNSW with PyO3 bindings.

It focuses on engineering decisions and measured behavior, not language advocacy.

A detailed preprint version of this work is being prepared for arXiv submission.

The intended audience is engineers who care about retrieval systems in production: index build windows, latency envelopes, recall stability, and integration constraints.

1) Continuity: BM25 -> vector search -> Rust backend

2) Prior benchmark context (before vector v0.4.1)

Before the vector-search re-engineering, Vajra was benchmarked as a BM25 engine against multiple systems. The original technical report is here:

• Vajra BM25 benchmark and technical comparison report (original)

That report includes comparisons against:

• bm25s
• bm25s-parallel
• tantivy
• pyserini
• vajra-parallel

A compact snapshot from that phase:

This prior work matters because it established a benchmarking discipline used again for vector search: explicit baselines, fixed query protocol, and metric-level reporting.

That continuity matters because the Rust work should not be interpreted as a fresh benchmark starting point. It is a continuation of the same measurement practice and the same operational questions from the BM25 phase: what do we gain, what do we lose, and what trade-offs can be controlled explicitly.

3) ZVec as the trigger for v0.5.0 re-engineering

The initial Feb 22 baseline established the v0.4.1 gap against ZVec at 10k vectors.

• Re-engineering Vajra's vector search with inspiration from ZVec (baseline + deep profiling)

That same post contains the full v0.5.0 optimization walkthrough (six implementation fixes), which is the canonical source for the Python-side gains before Rust.

3.1 Baseline (v0.4.1)

3.2 Gains in v0.5.0 (from the separate profiling post)

Net effect from that stage:

• Build gap to ZVec reduced from roughly 207x to about 36x.
• Query gap to ZVec reduced from roughly 8.5x to about 1.6x.

At this stage, the engineering question changed from "can Python be optimized?" to "which remaining costs are structural and should move to a systems layer?"

4) Why move to Rust after v0.5.0

After v0.5.0, two constraints remained:

1. per-step interpreter overhead in HNSW hot loops,
2. mutable graph updates under heavy insertion/search workloads.

Rust was used to address those constraints while keeping the Python interface stable.

The design goal was not to replace the Python developer surface. The goal was to preserve the same API ergonomics while moving the high-frequency ANN primitives (graph traversal, candidate-heap mutation, distance dispatch) into a lower-overhead execution layer.

4.1 Architectural motivations

• Predictable memory behavior: contiguous vector storage and explicit layered adjacency reduce pointer-chasing variance in traversal-heavy loops.
• Lower hot-path overhead: candidate-heap updates and distance dispatch run in compiled code rather than Python-level dispatch.
• Safer mutation semantics: Rust ownership/borrowing constraints make graph-update paths easier to reason about under heavy insert/search cycles.
• Throughput headroom: batch query paths can use native parallel execution (Rayon) without changing the Python API.
• No API break for users: PyO3 keeps the Python entry points stable while delegating execution to Rust.

4.2 Python-Rust integration map

4.3 Custom HNSW implementation notes

The Rust HNSW implementation in crates/vajra-hnsw is not a wrapper around an external ANN library. It is implemented directly for this stack, including:

• graph storage (Vec<f32> vectors + layered adjacency),
• upper-layer greedy descent,
• layer-0 beam search (beam_search_fast),
• profile parameter surface (M, ef_construction, ef_search, use_heuristic),
• explicit coalgebra reference path for parity and verification.

In practice, this means benchmark deltas can be attributed to Vajra's implementation choices instead of hidden behavior in a third-party ANN binding.

5) Critical path differences: quality vs fast vs instant

Profiles are not just parameter presets; they change traversal work in both build and query paths.

M and ef_construction primarily shape insertion/build work (graph connectivity and candidate exploration), while ef_search shapes query-time frontier size. Disabling heuristics changes neighbor-selection cost and can materially reduce CPU work at some recall cost.

A practical operating pattern follows:

1. boot with instant for low startup latency,
2. rebuild asynchronously with quality,
3. atomically swap index handles when quality build completes.

This pattern should be read as an explicit control-plane design: startup latency and final recall are treated as separate objectives with a deterministic handoff.

5.1 Formalization: Profile-Lambda Retrieval Architecture (PLRA)

The deployment pattern above is structurally similar to Lambda Architecture in data engineering (Databricks glossary), but adapted for ANN index lifecycle rather than stream/batch ETL.

I refer to this as Profile-Lambda Retrieval Architecture (PLRA):

• Serving/Instant path: minimal-connectivity index for immediate availability.
• Speed/Fast path: moderate-cost index for better steady latency/recall while rebuilds continue.
• Batch/Quality path: high-recall full build in the background.
• Reconciliation: atomic index-handle swap when quality build is ready.

This gives a reproducible control surface: startup SLOs are decoupled from final ranking quality, and quality can be converged asynchronously.

Conceptually, PLRA is useful because it turns a tuning trade-off into an architecture primitive: profile transitions become part of deployment policy rather than ad-hoc runtime tweaking.

6) Current Rust benchmark snapshot (Wikipedia vectors)

Benchmarks were run for 1k, 10k, 20k, and 50k vectors.

6.1 Build/recall at 50k

6.2 Query at 50k

6.3 Figures

Reading these plots together:

• Build time separates profiles the most as corpus size grows.
• Query latency stays low for all Vajra profiles in this setup.
• Recall degradation is concentrated in instant, while quality remains close to the reference.
• fast acts as a middle operating point for teams that need lower build cost without dropping to instant recall.

7) Reproduction notes (Wikipedia data source)

The benchmark corpus slices are sourced from ir_benchmark_data, which is built from Wikipedia-focused IR corpora used in prior Vajra benchmarking.

At a high level:

1. Acquire documents from WikIR corpora through ir_datasets (for example wikir/en78k, with smaller fallback sets).
2. Normalize into JSONL snapshots with stable fields (id, title, content, metadata).
3. Generate embeddings (all-MiniLM-L6-v2, 384d) and keep the same embedding cache across engine/profile runs.

4. Slice consistently into 1k/10k/20k/50k subsets for direct comparisons.

5. Data location used locally: ~/Github/ir_benchmark_data

6. Benchmark harness location: ~/Github/zvec_vajra_benchmark

Detailed reproducibility instructions (including expected directory wiring and commands) are provided in:

• vajra_search_engine/reproduction.md

8) References

• Vajra BM25 benchmark and technical comparison report (original)
• Vajra BM25 architecture post (Dec 24, 2025)
• Vajra vector search v0.4.1 -> v0.5.0 engineering post (Feb 22, 2026)
• Databricks: Lambda Architecture
• ZVec

The initial comparison produced numbers worth understanding. After six targeted fixes, the picture changed materially.

ZVec: What C++ Disk-Backed HNSW Looks Like

ZVec (v0.2.0) uses Proxima, Alibaba's production vector search engine, as its C++ backend. Its architecture is worth understanding because it explains why the performance is so different from a Python implementation.

ZVec stores the vector index on disk using memory-mapped files. zvec.open() is essentially an mmap syscall: it maps the on-disk file into the process's virtual address space and returns in milliseconds. No data is loaded into RAM at open time. The OS page cache handles the rest: when a query accesses a region of the index for the first time, the kernel loads the corresponding disk page into RAM, making it available at memory speed for all subsequent accesses.

The insert and HNSW construction pipeline is pure C++. Distance computations use SIMD (Single Instruction, Multiple Data) intrinsics: CPU instructions that apply the same operation to multiple values simultaneously. On modern x86-64 hardware with AVX2, a single SIMD instruction computes eight float32 dot-product contributions in parallel. Proxima's distance kernels (the low-level functions that compute cosine, L2, or inner product distances between vectors) are hand-tuned to issue these instructions in tight loops with no Python overhead.

The warmup step in the benchmark discards the first 10 query results. When the index is freshly opened, the data pages have not yet been loaded into the OS page cache. The first few queries trigger page faults and disk reads, adding latency that does not represent steady-state performance. After 10 queries, the frequently accessed pages (entry point, upper-layer graph nodes, and the hot region of layer-0 neighbours) are resident in RAM. The post-warmup latency is what ZVec delivers to a running service.

On 10,000 Wikipedia documents (all-MiniLM-L6-v2, 384d, cosine, M=16, ef_construction=200):

The build time includes all inserts, HNSW graph construction, and on-disk persistence. The query numbers are post-warmup, with the relevant pages in cache.

Vajra's Vector Search

Vajra's NativeHNSWIndex implements HNSW in pure Python and NumPy. Its internal architecture uses a coalgebraic abstraction for search: HNSWNavigationCoalgebra unfolds a beam search from an HNSWSearchState, which is a frozen dataclass containing the candidates heap, results heap, and visited set. This keeps the search logic clean, composable, and testable in isolation.

The index stores vectors in a NumPy float32 array, supports cosine, L2, and inner product metrics, and serializes to disk with pickle. It integrates directly with Vajra's VajraVectorSearch and HybridSearchEngine (BM25 + vector fusion) classes.

The initial benchmark results against ZVec revealed four problems.

The Baseline: Where Things Stood

Running the same 10,000-document benchmark:

The 207× build gap was informative. A C++ HNSW implementation is not 200× more clever algorithmically: both run the same graph construction. A gap of that size points to something wrong in the Python implementation, not a fundamental language barrier. The 8× query gap was larger than a Python-vs-C++ interpreter comparison alone would predict, which pointed to the same conclusion. Profiling confirmed it: most of the gap came from specific implementation defects, not the algorithm.

Root Causes: Four Problems

Problem 1: O(N²) Array Growth During Build

The most damaging bug. _insert_vector() was called once per node during add(). Each call did this:

# Called N times. Every call copies ALL existing vectors.
self.graph.vectors = np.vstack([self.graph.vectors, vector])

For N=10,000 with 384-dimensional float32 vectors, the total bytes copied are:

sum(i * 1536 bytes for i in range(10000)) ≈ 73 GB

This explains almost the entire 97-second build time. At 10 GB/s effective NumPy throughput inside a Python loop, 73 GB of copies takes roughly 7 seconds from memory bandwidth alone. The Python interpreter overhead per call added the rest.

The fix: add() already receives all N vectors as a single array. Pre-allocate the full block once.

n_new = len(ids)
existing = len(self.graph.ids)

if self.graph.vectors is None:
    self.graph.vectors = np.empty((n_new, self._dimension), dtype=np.float32)
else:
    self.graph.vectors = np.concatenate(
        [self.graph.vectors, np.empty((n_new, self._dimension), dtype=np.float32)]
    )
self.graph.vectors[existing : existing + n_new] = vectors

The insertion loop then calls _insert_node(existing + i, ids[i]), which reads from the pre-allocated array rather than extending it. N incremental vstack calls become one allocation and one bulk fill.

Problem 2: Redundant Norm Computation on Pre-Normalised Vectors

After fixing the vstack issue, build time dropped from 97 seconds to around 83 seconds. That improvement was smaller than expected, so profiling continued.

Running cProfile on a 2,000-document build and sorting by tottime (time spent inside each function, excluding callees) produced this:

ncalls   tottime   filename:function
  2132    1.459s   _search_layer
28918263   57.3s   np.linalg.norm        ← 28 million calls
 313447    2.198s   score_batch_normalized

Twenty-eight million calls to np.linalg.norm during a 2,000-document build. Scaled to 10,000 documents, that accounts for almost the entire remaining build time.

Tracing where these calls came from led to score_batch, the cosine distance function called inside the beam search:

# Called ~14 million times during build at N=10k.
# Recomputes np.linalg.norm on vectors that are already L2-normalised.
def score_batch(self, query, vectors):
    query_norm = query / (np.linalg.norm(query) + self.eps)
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    vectors_norm = vectors / (norms + self.eps)
    return (vectors_norm @ query_norm).astype(np.float32)

The general case for cosine similarity requires normalising both the query and the stored vectors before taking the dot product. score_batch does this correctly for the general case. The problem: add() already normalises all incoming vectors to unit length before inserting them into the graph. Every vector in self.graph.vectors is already L2-normalised. The np.linalg.norm(vectors, axis=1) line in score_batch was recomputing norms on data that was provably unit-length, throwing away the work done at insertion time on every distance call during construction.

The fix: when the metric is cosine, store the distance function as a specialised lambda that skips normalisation entirely. For two unit vectors, cosine similarity reduces to a plain dot product:

cos(a, b) = (a · b) / (||a|| × ||b||) = a · b    (when ||a|| = ||b|| = 1)

# In __init__, for metric="cosine":
_scorer = CosineSimilarity()
self._distance_single = lambda a, b: float(1.0 - float(np.dot(a, b)))
self._distance_batch = lambda q, vs: 1.0 - _scorer.score_batch_normalized(q, vs)

Where score_batch_normalized is a single line: return (vectors @ query).astype(np.float32, copy=False).

The copy=False matters: without it, each call creates a redundant array copy even when the result is already float32. At 1.5 million calls during a 10k build, those copies add up.

Problem 3: Per-Neighbour Python Dispatch in the Beam Search

The beam search in _search_layer originally computed one distance call per neighbour:

for neighbor in self.graph.get_neighbors(current, level):
    if neighbor not in visited:
        dist = self._distance_single(query, self.graph.vectors[neighbor])
        # ... heap update

At ef_construction=200 with M=16 neighbours per node, each node insertion triggers roughly 200 × 16 = 3,200 individual Python function calls for distance computation. At N=10,000, that is 32 million individual calls.

The fix: collect all unvisited neighbours first, then compute distances in one batch call:

unvisited = [n for n in layer_dict.get(current, []) if n not in visited]
if not unvisited:
    continue
visited.update(unvisited)
neighbor_dists = dist_fn(query, vectors[unvisited])

One NumPy matrix-vector multiply replacing 16 individual calls. BLAS vectorises the dot products; Python call overhead drops by 16×.

Note the layer_dict here: caching self.graph.layers[level] as a local dict avoids the get_neighbors() function call overhead on every beam step (~2 million calls at N=10k). Same pattern for vectors = self.graph.vectors and dist_fn = self._distance_batch.

Problem 4: Immutable Coalgebra State in the Query Hot Path

The build problems explained the gap at construction time. The 8× query latency gap had a different root cause: the HNSWNavigationCoalgebra implementation.

To understand why, it helps to understand what the coalgebra is doing. In Vajra's design, HNSW search is modelled as a coalgebraic unfolding: starting from an initial state, a transition function produces the next state at each step until a termination condition is met. This is a clean, mathematical way to structure the beam search. The state at each step is an HNSWSearchState, a frozen Python dataclass with three fields:

• candidates: the priority queue of nodes to explore next (the "open set")
• results: the current best-ef results found so far
• visited: the set of nodes already evaluated

HNSWSearchState is frozen: a frozen dataclass cannot be modified after creation. This is intentional. Immutable state objects are composable, independently testable, and safe to reason about. You can snapshot any intermediate state during a search, replay it, or pass it to a separate function for analysis. This makes the coalgebra a good abstraction for the documented public interface.

The performance cost is in the transition step. To advance from step k to step k+1, the coalgebra creates a brand-new HNSWSearchState. The visited field is a frozenset, Python's immutable set type. Updating it to include the newly visited node requires constructing a completely new frozenset from scratch:

# Called ~800 times per query (ef=50, M=16 neighbours per step).
return [HNSWSearchState(
    candidates=tuple(candidates),     # new tuple: O(ef) allocation
    results=tuple(results),           # new tuple: O(ef) allocation
    visited=frozenset(visited),       # new frozenset: O(|visited|) copy — grows each step
    ...
)]

At step k, frozenset(visited) has k elements to copy into the new object. The cost is O(k) per step. Over a full beam search with ef=50 and M=16 neighbours per expansion, the search runs for roughly 800 steps:

Total copy cost: sum(k for k in range(800)) = 319,600 element-copy operations per query

That is, the visited-set maintenance alone accounts for 319,600 integer copy operations per query, growing quadratically with the number of steps. Add 800 tuple allocations for candidates, 800 for results, and the Python memory allocator is doing significant work on every query call, entirely in overhead rather than in useful distance computation.

The coalgebra is the right abstraction for Vajra's architecture. The fix was to add a parallel fast path that implements the identical algorithm with mutable state, bypassing the coalgebra in the hot query loop while keeping it available as the documented public interface for tests and extension.

_beam_search_fast() implements the same algorithm with mutable state:

def _beam_search_fast(self, query, entry, ef):
    layer_0 = self.graph.layers[0] if self.graph.layers else {}
    vectors = self.graph.vectors
    dist_fn = self._distance_batch

    d0 = float(dist_fn(query, vectors[entry : entry + 1])[0])
    candidates = [(d0, entry)]    # min-heap
    results = [(-d0, entry)]      # max-heap, capped at ef
    visited = {entry}

    n_results = 1
    results_full = (n_results >= ef)
    worst = d0

    while candidates:
        dist, current = heapq.heappop(candidates)
        if results_full and dist > worst:
            break

        unvisited = [n for n in layer_0.get(current, []) if n not in visited]
        if not unvisited:
            continue
        visited.update(unvisited)
        dists = dist_fn(query, vectors[unvisited])

        for n, nd_raw in zip(unvisited, dists):
            nd = float(nd_raw)
            heapq.heappush(candidates, (nd, n))
            if results_full:
                if nd < worst:
                    heapq.heapreplace(results, (-nd, n))
                    worst = -results[0][0]
            else:
                heapq.heappush(results, (-nd, n))
                n_results += 1
                worst = -results[0][0]
                if n_results >= ef:
                    results_full = True

    return [(idx, -d) for d, idx in sorted(results, reverse=True)]

The n_results counter and results_full flag also eliminate ~19 million len(results) calls that showed up in the pre-fix profiler (the len() check ran on every iteration of the inner loop; once the results heap is full it stays full, so a boolean flag is sufficient).

search() was rewritten to use _greedy_search_layer for the upper-layer descent and _beam_search_fast for the layer-0 beam search. The coalgebra is still constructed and stored in self.coalgebra after every add() call; it is just not used in the query hot path.

Additional Fix: Inline _prune_connections

One smaller fix worth mentioning: _prune_connections() was a separate method called for every selected neighbour during node insertion, roughly 330,000 times at N=10k. Each call recomputed neighbor distances to check if pruning was needed, even when most nodes did not need pruning.

Inlining the pruning logic directly in _insert_node eliminated the function-call overhead and allowed caching layers[l] as a local reference:

layer = self.graph.layers[l]
for neighbor_idx, _ in selected:
    nbrs = layer[neighbor_idx]
    if len(nbrs) > M:
        dists = self._distance_batch(vectors[neighbor_idx], vectors[nbrs])
        keep = np.argsort(dists)[:M]
        layer[neighbor_idx] = [nbrs[i] for i in keep]

Results After All Six Fixes

The same 10,000-document benchmark, version 0.5.0:

The gap to ZVec:
- Build: 207× → 36×. The remaining 36× is the Python-vs-C++ interpreter cost for 10,000 HNSW insertions. Getting below ~12 seconds in Python would require Numba JIT on the inner beam search loop.
- Query: 8.5× → 1.6×. The remaining 1.6× reflects Python's heapq overhead. At ef=50 with M=16, each query runs roughly 800 heap operations. CPython heapq.heappush costs ~300ns per call on M1; 800 × 300ns = 240ms of irreducible heap overhead. ZVec's C++ std::priority_queue at ~10ns per op does the same 800 operations in 8ms.

The query gap is now at the CPython interpreter floor for this algorithm, not an implementation defect.

What This Demonstrates

None of these six changes required restructuring the algorithm, adding Numba, writing a C extension, or compromising the categorical abstractions that make Vajra's code clean. They were all standard Python engineering: profile first, find the real bottleneck, fix the specific thing.

The original 8× query gap looked like a fundamental Python-vs-C++ problem. It was mostly a frozenset copy in the wrong place. The 207× build gap looked like an algorithmic problem. It was a single np.vstack call inside a loop.

Vajra v0.5.0 is on PyPI. The benchmark code is at github.com/aiexplorations/vajra_bm25 and the ZVec comparison scripts are at github.com/aiexplorations/zvec_vajra_benchmark.

More interestingly, there is a pattern on the part of users of AI to utilize AI as a substitute for thinking on their own. I say this as someone who has done exactly this. We live in a time of radical delegation, not just of the physical labour that made our muscles and bones strong, but of the mental labour that made our minds strong and that made us the species we are.

I don't know if we were promised greater productivity and human thriving by the tech boffins who built AI systems, especially the leading minds in the space who built and pushed ChatGPT, Claude and the like to the masses. It behooves us to examine their intentions more closely, and I say this as someone who is a practitioner of AI, and who has built models, and built on top of large language models, and the so-called foundation models. What is discernable is that these tools provide a specific kind of advantage to those who use them. Whether it is writing, coding, painting or other cognitively demanding tasks, the creation of content in its entire spectrum is covered by AI. Movies are being made with AI. It is possible for a person with just an imagination and the intent to use these tools to sequentially explore an idea for a film with AI, write a script with AI, prepare clips or scenes of video with AI, and edit and package them into a coherent film, also with AI. The user here provides the steering function, that guides the potential of the AI to some kind of impact.

To a technologist, this sounds like a promising application of AI. The practical purpose of embodying intelligence into formerly inanimate things around us, is to enable them to be animate in specific and helpful ways. Perhaps one path along the garden of forking paths in this general direction is the use of AI for cognitively demanding tasks. If AI could be used to solve hard mathematics problems, for example, some of us would find the weight lifted from our shoulders as we moved from trepidation and tedium to understanding. This is the logical positivist's spin on artificial intelligence, as a force for doing good to those that deserve that good. We know that good is subjective, though, and we will discuss this in this post.

To an AI technologist, the process of using gradient descent to recursively improve factor-response models on arbitrary data to learn a useful model ("useful" in the terminology of W. Edwards Deming) is a beginning. To make this system write passably good, human-like essays or have it generate incredibly creative and interesting images and videos is progress, and can be considered a technological or computational accomplishment. The fact that we have been able to sequentially transform a field of noise into a meaningful image, that is steered by a text prompt, to produce an image - this is no mean feat. And stacking such images together, nay, generating that sequence so that moving objects may be rendered in a form that we humans recognize as movies and clips - this is incredible too. In a nutshell, there is no doubting the technological achievements that these models represent. This may even represent a success - and I say "may even" for a reason.

I have described a beginning and progress in Henry Ford (beginning-progress-success) terms, so what then is the true definition of success in the use of artificial intelligence? To understand this, we need to understand the human perspective on artificial intelligence. Humans look to AI from the point of view of utility, ultimately. Even if the underlying techniques, methods, mathematics and statistics are explored from an information theoretic perspective and despite the fact that the theory and practice here is valuable, the underlying motivations have been the need to advance some agends of either humanity as a whole, or sub-sections of human society. There is a utilitarian version of the anthropic principle of sorts that I will have to invoke in this context. The anthropic principle the supposition that the universe created life so that it could observe itself and learn about itself - and I'll go out on a limb and say, measure its own utility. Similarly, perhaps biological intelligences create artificial intelligence so that they can learn about their utility. The bounds of artificial intelligences we create are not limited to the bounds of humans, however, neither is it the case that the AI we build are inside the closed systems we represent. Unlike the universe which subsumes life and artificial intelligence, humans share a landscape with artificial intelligence. Considering in this specific way the utility of artificial intelligence systems is a contrast to the libertarian, almost phenomenological view that a lot of business leaders and technologists have, in the context of artificial intelligence. Many of them anthropomorphize AI, and many attribute qualities to algorithmic systems that appear intelligence from a human standpoint. I impute an anti-phenomenological, utilitarian bias deliberately here, because phenomenology is not a good underpinning to reason about tools, specifically because tools are built with utility in mind.

Let's now think about what the human endeavour looks like, for someone writing, painting, or making movies. In the past, a writer would have to think through ideas from their experiences, from conversations, from books and articles they read, from shows they watched or songs they listened to. These were the "prompts", the inspiration for writing, for creating and for building ideas on top of old ideas and notions. They had to grapple with the unsafe silence their own minds would throw at them, when they came up with new ideas. They had to grapple with the uncertainty, of whether the idea would be valid or not. And they had to articulate these thoughts, however incipient or inchoate, so that (a) they can make sense of the ideas themselves (the reasoning utility), (b) they can explain it to others (the commuinication utilty). The reasoning utility for a writer of fiction may constitute the characters, their interactions, the plot and the denouement. These components helped them think about how a piece of fiction could be written. The communication utility in writing for a fiction writer would be their craft. Their use of the words and phrases, their descriptive toolkit for characters and interactions between them, the narrative style, the style of writing that drew people in. Some writers may be inventive, conceptualizing their own words, inventing their own languages, imagining the imperfections and quirks of their characters. The field of creativity is vast for human writers. Our brains are hallmarks of a general biological intelligence, and powered by language, reasoning, experiences that are multi-sensorial and the power of manifestation, we bring characters, ideas, places and the like, to life.

For the painter, the visual field of the canvas or paper, or other mediums is a blank slate. Of course, we all take inspiration for what we may want to draw and paint, but everyone approaches a scene or a subject differently. The painter's mind imagines a scene and the medley of paint and tools and brushes and the like translates the imagined scene into an approximation of the scene on the medium. The reasoning utility for the painter comprises of the subject of the painting, the composition and how subjects and objects are positioned on the visual field, and the broader meaning or import of the art. The communication utility of the painter are the strokes and the technique and the tools and the sequence of use of the tools and colours and the rest on the medium. In a similar vein, someone making a movie may find the reasoning utility of their movie to be the subject, the denouement and the plot with the characters and the like, similar to a book, while the communication utility here is the synthesis of actors, props, camera and screen play, effects, post-processing and editing. These together help tell the story that the creator intended.

Now let's examine the mental effort of these creators. They each have to begin from priors, and learn to nurture the ideas they work with, as they develop the reasoning around the ideas. The more they work and the more they produce, be it books or art or movies, the more they explore the landscape of tools and possibilities available to them, and the better they're able to respond in (a) confidence, (b) skillful execution, (c) speed and (d) cross-pollination of capabilities and ideas across mediums. What I discuss here is not ground-breaking, in any way, and is something anyone who has practiced any skill understands.

Coding and writing software are also similar. Tech professionals and knowledge workers are now getting used to using AI and prompting AI models to build applications, perform data analyses and deliver results. The human effort required to come up with a new logo, a new design for a report, a good opening paragraph, a good visualization that tells a story with data, a narrative that is compelling across a report, a piece of software that is built to spec but also performant, a beautifully made short movie for an ad in one or a couple of takes - these are likely to be replaced by AI that is prompted to produce different kinds of results. The possibilities with AI are remarkable. We have a tool that leverages the collective styles, ideas, notions and knowledge of billions of humans that came before it, and which can be prompted to give us what we desire, without the challenge and the pain of thinking about the ideas that much, and with no skill.

The radical delegation I mentioned in passing above is now able to be described in more exact terms. Radical Delegation is a pattern of behaviour where a human or a group of humans who's exceptionally skilled in a task decides voluntarily to delegate the process of executing that task to another human, group of humans, or a machine or a group of machines, that can perform the same task, in either the same fashion, or in an altogether different fashion. Radical delegation is not the same as automation but is a step on the path towards an industrialized or automated task. It is on the path to commoditization, either by the creation of human scale or mechanization of a task. Radical delegation is voluntary, and not imposed due to external constraints. When a software engineer decides to use AI to solve a problem, it is a voluntary choice for many at the moment. It may not stay that way for long, and there will be equivalent of a corporate program soon if not already present, in most teams that encourage the use of AI coding tools.

Beyond a point, the use of AI to automate work may not be an individualist question anymore, but one of either collectivism or interventionism. I say this because the questions that artisans or individuals ask about whether to use AI tools and how to use them will anticipate the questions asked by their managers, or people that deliver business or livelihood opportunities to them. At that point, the question becomes not one of choice, but one of efficiency, economics and markets. We've now moved from the field of free will (in the limited context of the use of AI tools based on one's arbitrary wishes), to the collectivist or interventionist approach to using AI, as a social policy which unfolds in the form of specific use cases, risks, and introduces both efficiencies and frictions in the form of the use of AI by humans in different contexts.

Collectivist culture in the context of AI on the internet takes many forms. We've seen movements like effective accelerationism (e/acc) become a force in prior years. This movement has advocated the use of technology in an interventionist fashion, primarily with the express objective of achieving greater human progress. The definition of what that progress entails is usually a little less well defined than the spirit in which technology and positivism is framed by e/acc. My own sense is that e/acc is akin to the culture that inspired tech companies in the 2000s and 2010s, a la "move fast and break things" and less akin to a serious movement which has well considered and rational use of technology to various ends. One could perhaps make a similar argument for AI tool adoption, in that there is a lot of depth in method, but less depth in motives. This implies a greater emphasis on communication utility, and a reduced focus on reasoning utility, for the concepts that e/acc engages with.

AI promoters and detractors for each of the various shades of automation exist. This spectrum of AI automation ranges from merely minor automations and thinking aids such as using tab-autocomplete features in code editors, on one end, all the way to full automation of the software engineering lifecycle on the other end. The promoters often state efficiency and the possility of moving grunt work away from humans and towards machines, as a noble end state. It is rare to see promoters acknowledge some of the pitfalls of this approach, but many do. For example, is the human a mere vessel with which to execute a software workflow? In the language of software engineering, are humans using AI mere wrappers with gates and checkpoints?

The detractors have both simplistic and more nuanced arguments. Detractors argue about skill atrophy, akin to how muscular or other atrophy is frowned upon by atheletes or artists. Detractors argue about how it is important for human expertise to survive and thrive, which is certainly noble. On the other hand, detractors don't often acknowledge the mundaneness and hardship associated with mastery and expertise. Here I refer in spirit to George Leonard describes the long plateau where "nothing happens" in his book Mastery, who acknowledges that the path to mastery of any skill includes a non-trivial amount of "grinding". This is the long plateau before we become adept at something, whether that is writing, painting, movie making or anything else. Detractors are acutely aware of the importance of reasoning utility, but not aware of the need for communication utility. I risk oversimplifying when I say this, and when I assert that detractors of AI automation don't acknowledge the need for communication utility arising out of broad adoption. Such broad adoption is precisely what drives progress in AI, which is known to be a data-intensive, compute intensive field that relies on ever more data to keep the train of AI model progress chugging along.

What does utopia look like, to the detractors, if the dystopia looks like radical delegation? Perhaps a term that could describe the opposite of radical delegation to AI, is personal thriving. By this I mean a condition where the human being continues to grow in judgment, in skill, in confidence, in taste and in responsibility, even while using tools that increase their leverage. Personal thriving is not Luddism. It is not a refusal of tools. It is not a romantic attachment to doing everything by hand, merely because it was done that way in the past. It is instead the insistence that the human being remain the site where understanding accumulates, where responsibility resides, and where the final meaning of work is interpreted.

This distinction matters because many discussions about AI collapse into a binary. One either embraces automation and scale, or one rejects them and becomes a reactionary purist. Reality is not so simple. A writer may use a thesaurus and still write. A programmer may use a debugger and still understand the code. A musician may use software instruments and still compose. In the same spirit, someone may use AI and still preserve their own thought. The question is not whether tools are involved. The question is whether the tool merely extends the person, or begins to displace the person's own reasoning.

The most dangerous form of displacement is not immediately visible in the produced output. This is what makes the question difficult. If AI produces a decent essay, a useful piece of code, a visual that is aesthetically appealing, or a summary that appears cogent, then the user is tempted to say that the process has succeeded. But this is only the communication utility being satisfied. Something legible has been emitted into the world. Something fit for consumption now exists. What has happened to the reasoning utility of the user in the process, however, is a separate question altogether. Has the user deepened their understanding of the problem? Have they become more capable of independently solving a related problem in future? Have they developed taste, judgment and character? Or have they merely learned to manage a prompt loop?

This, in my view, is the accountability problem.

When a human being performs a difficult task, they are not merely delivering an output. They are also shaping themselves. The action of struggling through a proof, of debugging a stubborn failure, of finding the right paragraph to close an essay, of understanding a difficult passage in a book, of revising an argument that does not yet cohere - these are not only means to an end. They are formative. They create a kind of internal architecture. The human being that emerges after such effort is not identical to the one that began the task. There is a strengthening involved. The person now sees a little more clearly, acts a little more confidently, and is a little less likely to be deceived by surface level appearances the next time around.

AI changes the economics of this formation. It offers outputs without requiring a commensurate depth of interior transformation. It can often provide the language of understanding without the underlying structure of understanding having been built in the user. This is not always harmful. Sometimes the tool genuinely helps us cross a threshold and see what we could not see before. But in many cases it creates a counterfeit of mastery. One has the appearance of intelligence, the verbal form of competence, and the practical veneer of productivity, without the grounding labour that would ordinarily have made these things one's own.

And when that counterfeit becomes widespread, accountability begins to thin out in curious ways. If the code fails, was it the engineer's failure, or the model's? If the report is shallow, was it the analyst's misunderstanding, or the AI's generic synthesis? If the essay is derivative, was the writer lazy, or merely over-assisted? It is tempting to say that this is irrelevant because the market only cares for results. But markets do care for accountability eventually, especially when systems fail at scale. Someone must still answer for poor reasoning, for fragile software, for wrong decisions, and for the consequences of action taken on shallow understanding. Radical delegation obscures this, because it multiplies outputs while diffusing ownership over the thinking that led to them.

This is why I do not think the future of AI can be properly framed as merely a race between optimists and pessimists. It is a question of anthropology before it is a question of economics. What sort of being does the human become through the regular use of such tools? What habits are reinforced? What forms of laziness are made normal? What kinds of excellence become easier to fake? What kinds of patience become harder to sustain? We ought to be suspicious of any technological regime that makes the visible signs of competence easier to obtain, while making the underlying cultivation of competence optional.

The managerial and institutional form of this problem is easy to imagine. A team discovers that AI lets them ship faster. A manager then asks for more output with the same headcount. Another manager notices that junior engineers can now produce PRs that look acceptable sooner. A process is established where humans review, approve and nudge model outputs at scale. Over time, the institution ceases to reward the slow accumulation of judgment because the visible proxies for judgment are now cheaply available. The system begins to privilege throughput over understanding. A generation of people learns to operate at the communication layer of work while becoming less familiar with the reasoning layer beneath it. This is not merely an issue of preference. It is an issue of what kinds of institutions we are building and what kinds of humans those institutions will select for.

At this point, one could object that every technology has done something similar. Writing changed memory. Calculators changed arithmetic practice. Search engines changed recall. Why should AI be treated differently? My answer is that AI is not merely a storage aid, a retrieval mechanism, or a mechanical accelerator for a well-understood process. AI increasingly intervenes at the level of synthesis, composition, explanation and judgment-like behaviour. It enters the domain that humans once used to become more fully themselves through repeated practice. This is why it cannot be treated as just another convenience. It reaches farther inward.

All of this suggests to me a practical ethic for the use of AI. Use it where leverage is real, but do not use it in a manner that hollows out the very faculties you are hoping to strengthen. Use it to compare approaches after you have attempted one. Use it to critique your reasoning after you have actually reasoned. Use it to accelerate drudgery that does not educate you. Use it to open doors into difficult domains, but do not let it become a permanent substitute for crossing the threshold yourself. In other words, use it in a way that preserves the relationship between effort and growth.

There is a Sanskrit word that feels relevant in this context: dvandva, the realm of dualities and oppositions. Many of our conversations about AI are trapped in precisely such dualities. Optimism versus pessimism. Productivity versus purity. Automation versus authenticity. But perhaps wisdom here lies in a more careful middle path. A refusal of both naive surrender and performative rejection. A deliberate use of tools that keeps the human person in view.

The point, then, is not to ask whether AI can do the work. In many cases, it plainly can. The point is to ask what happens to us when we allow it to do too much of the work that once made us capable, responsible and alive to the world. If the answer is that we become more dependent, less accountable, less skillful, and more alienated from our own effort, then we have not progressed merely because the output arrived faster. Progress has to be evaluated not only by the abundance of artifacts it produces, but by the quality of the humans it leaves behind.

Now that I have gotten that out of the way, let me come to the purpose of this post: introducing Vidai, a neural engine for mathematics. Vidai (விடை) is Tamil for "answer," reminiscent of the middle school experiences that lead to sweaty palms for kids learning arithmetic, who also have to contend with the occasional test or exam demanding these answers. Fortunately for such kids, it is a running joke also that ChatGPT, of all things, cannot do mathematics. "Count the number of Rs in Strawberry" has long failed ChatGPT, for instance and routinely does the rounds whenever a new SOTA model is released. Overall, it looks like ChatGPT is not quite there with the three Rs (pun intended).

But why build such a neural engine now, you might ask me, and the answer to that lies in the specific triggers I had during the new year's break. On one of Machine Learning Street Talk's recent episodes (this one), I was watching Petar Veličković from Google DeepMind, among other discussing graph neural networks and category theory for neural networks. Petar casually pointed out a paradox that stuck with me: language models often get basic multiplications wrong even though in the process of generating any given response, they perform millions of multiplications internally and quite routinely. A weird analogue indeed to how our brains may also be performing such mathematics routinely between the vast neuronal connections in our complex brains, all the while getting basic arithmetic sums wrong! Nature works in strange ways. The irony of all this was not lost on me. Here are systems executing vast amounts of arithmetic correctly at the matrix level while failing at the basic, symbolic level. It got me thinking about what kind of model architecture would actually work for arithmetic.

Vidai is a purpose-built encoder-decoder model for mathematical expressions that sidesteps this problem entirely. I spent part of the new year's break trying to build a transformer model that could perform arithmetic operations. This post discusses how I went about this problem.

The Structure of Mathematical Expressions

Starting from the understanding of how transformers work, which has now become commoner knowledge than it was a couple of years ago, I started poking at the problem by looking at the structure of mathematical expressions. When we evaluate a mathematical expression, we implicitly solve a graph-shaped problem. There are values at various leaves of the tree in an arithmetic graph. Mathematical operations perform unary, binary, and other operations on these values, leading to a result at the root of the tree.

Conventional transformer LLMs don't explicitly learn this structure. They're trained to be large-scale pattern learners that solve a wider array of language understanding tasks from noisy internet-scale data. So I asked the question: since mathematical expressions, especially arithmetic, are likely to be less noisy and easier to parse, can we build a sequence-to-sequence model that learns the patterns in arithmetic expressions?

Don't Train What You Already Know

The operations of division, addition, multiplication, and so on, are not in and of themselves subject to change unless we invent new number systems and new kinds of mathematics. We just need the mathematical operations of different trees we infer from the text to be computed using known rules. In other words, there is no point in trying to train addition using gradient descent. Doing something like that is not just wrong, it is beyond wrong, if there is such a thing. (Then again, SOTA LLMs often encode numbers as tokens, but let's not go there. I am tempted to say that this is not just stupid, but beyond stupid).

During the time I built Vidai, I was also teaching my four-year-old son some simple sums, the kind toddlers learn to do. Watching him struggle, I noticed something: the bulk of the challenge kids seem to have with arithmetic at that age is the parsing and the conceptual understanding of addition or subtraction or other operations. What does it mean to "add" two things? Which number goes where? Would it matter which way the operation was done? Does the mathematics map somehow to something in the real world? Subsequently, the kids learn the rules, but then struggle at a different level. Having learned the rules of mathematical operations, they struggle with understanding how to interpret a given sum so that the numbers may be "all lined up in a row", so that the simple, comfortable, well-known, and now-familiar work of addition or subtraction or other operations can be executed. So, parsing and structure understanding first, and computation second. I thought that this is perhaps the same split in functions that neural networks need.

This observation led me to separate what neural networks are good at from what they are bad at:

• Parsing requires understanding that (3 + 5) * 2 means addition inside the parentheses with the result becoming an operand of multiplication. This involves operator precedence, parenthesis matching, implicit conventions, and the various ways humans write the same mathematical idea. This is pattern recognition, and neural networks excel at this. The noise is in how we express the expressions, and this noise is something LLMs can deal with in their current paradigm.

• Computation, once you know the structure, is trivial. The rules of addition and multiplication are fixed, known, and implemented correctly in every programming language. No learning from data is required.

Vidai trains the model only on parsing tasks: converting expressions into prefix notation trees that make structure explicit. A separate module with zero learned parameters walks the resulting tree and executes operations using exact arithmetic.

Input: "2x² + 3xy - √(x+1)"
    ↓
VIDAI (learned parsing, 44M parameters)
    ↓
Prefix: "+ * 2 ** x 2 - * 3 * x y sqrt + x 1"
    ↓
SymPy (deterministic computation, 0 parameters)
    ↓
Structured: 2*x**2 + 3*x*y - sqrt(x + 1)

Ergo, the neural network never learns that 3 plus 5 equals 8. It learns that when humans write 3 + 5, they intend an addition operation with 3 and 5 as operands. Importantly, it learns to represent this as a tree of operations.

Why Parse Math When Calculators Exist?

When you type 2x + 3y into SymPy, it fails because 2x is not valid Python syntax and the multiplication operation must be explicitly communicated to it. Greek letters like θ² + φ require symbol declarations. Scanned equations from textbooks, arrive with OCR artifacts, inconsistent spacing, and notation variants that no existing system handles gracefully.

Calculators and computer algebra systems are powerful at computation but brittle at interpretation. They demand that humans translate notation into rigid syntax before engaging with it. Vidai occupies the space between human notation and symbolic computation, accepting expressions written the way people actually write them.

The Architecture

Character-Level Input Encoding

The encoder uses character-level tokenization: each ASCII character maps directly to its code (0-255). No vocabulary, no subword tokenization, nothing. The string "x^2 + 3y" becomes the sequence [120, 94, 50, 32, 43, 32, 51, 121].

def encode_input(text: str, max_len: int = 256) -> list[int]:
    """Encode input as ASCII codes."""
    ids = [ord(c) if ord(c) < 256 else ord('?') for c in text]
    return (ids + [0] * max_len)[:max_len]  # Pad or truncate

This approach has a useful property: any symbol works without vocabulary changes. Unicode characters like √, θ, or ² just become their character codes. The model learns to recognize these patterns from training data rather than requiring explicit vocabulary entries.

Prefix Notation as Tree Representation

The key insight of the architecture: prefix notation (also known as Polish notation) makes tree structure explicit in a flat sequence. The expression (3 + 5) * 2 has this tree structure:

        *
       / \
      +   2
     / \
    3   5

In prefix notation, the operator comes before its operands: * + 3 5 2. Reading left to right, each operator "claims" the next N operands (2 for binary operators, 1 for unary). No parentheses needed; the structure is unambiguous.

A more complex example, 2x² + 3y:

Infix:   2*x^2 + 3*y
Tree:
            +
           / \
          *   *
         / \ / \
        2  ^ 3  y
          / \
         x   2

Prefix:  + * 2 ** x 2 * 3 y

The model's job is to learn this transformation, so, given arbitrary mathematical text, output the prefix notation that represents its structure.

The Encoder-Decoder Architecture

The system uses a standard transformer encoder-decoder, but with specific design choices:

ContextEncoder (19M parameters):
- 6 transformer layers with 8 attention heads
- 256-dimensional embeddings
- Character embeddings + positional embeddings + depth embeddings
- The depth embeddings encode tree structure hints during training

class ContextEncoder(nn.Module):
    def __init__(self, config):
        self.token_embedding = nn.Embedding(256, d_model)  # ASCII
        self.position_embedding = nn.Embedding(max_seq_len, d_model)
        self.depth_embedding = nn.Embedding(max_depth, d_model)
        self.transformer = nn.TransformerEncoder(...)

    def forward(self, input_ids, tree_depths, attention_mask):
        x = self.token_embedding(input_ids)
        x = x + self.position_embedding(positions)
        x = x + self.depth_embedding(tree_depths)
        return self.transformer(x, mask=attention_mask)

SymbolicParserDecoder (25M parameters):
- 4 transformer decoder layers
- Autoregressive generation with cross-attention to encoder output
- Output vocabulary of ~60 tokens: operators (+, -, *, /, **), functions (sin, cos, sqrt, etc), variables (x, y, theta), digits, and special tokens

class SymbolicParserDecoder(nn.Module):
    def generate(self, encoder_memory, max_len=64):
        """Autoregressive generation of prefix notation."""
        output_ids = [BOS_TOKEN]
        for _ in range(max_len):
            logits = self.forward(output_ids, encoder_memory)
            next_token = logits[:, -1, :].argmax()
            output_ids.append(next_token)
            if next_token == EOS_TOKEN:
                break
        return output_ids

The TreeComputeModule: Zero Learned Parameters

Once we have prefix notation, computation is deterministic. The TreeComputeModule has exactly zero learned parameters. It walks the prefix notation tree and executes operations using exact arithmetic:

# These are the actual operations - no neural network, no gradients
def add_op(left: Tensor, right: Tensor) -> Tensor:
    return left + right

def mul_op(left: Tensor, right: Tensor) -> Tensor:
    return left * right

def pow_op(left: Tensor, right: Tensor) -> Tensor:
    # Use exact integer multiplication for small exponents
    right_int = torch.round(right)
    is_small_int = (right == right_int) & (right_int >= 0) & (right_int <= 10)
    result = torch.ones_like(left)
    for i in range(10):
        result = torch.where(right_int > i, result * left, result)
    return torch.where(is_small_int, result, torch.pow(left, right))

ARITHMETIC_OPS = {
    'add': add_op, 'sub': sub_op, 'mul': mul_op,
    'div': div_op, 'pow': pow_op, 'mod': mod_op,
}

For symbolic expressions (containing variables), the prefix notation is passed to SymPy which handles the algebraic manipulation.

Summary

Total: 44.6M parameters which are all in the encoder-decoder, since as I have discussed earlier, computation is deterministic and there is no actual machine learning involved there.

The Precedence Bug

When I trained the above transformer model on the initial dataset, the performance plateaued to 68% sequence accuracy (despite 98% token accuracy). Token accuracy is easier to learn, and sequence accuracy harder, and the simultaneous learning of both of these in tandem during the training process showed how we'd get to 80% accuracy on the token accuracy long before we would reach a similar number for sequence accuracy. The 30-point gap pointed directly at the problem: operator precedence ambiguity in my training data.

The data generator I used for data generation of expression data, stored as text later, built random expression trees and converted them to infix notation, but 76 + 25 * 67 is ambiguous under standard mathematical conventions. The training data contained contradictory labels for identical inputs, and 68% accuracy was the best the model could achieve when trying to satisfy contradictory constraints.

After fixing the data to use explicit parentheses, accuracy reached 95% within the first epoch.

Test Results

I built a systematic test suite covering 11 categories of mathematical patterns. The test script (scripts/eval/test_finetune.py) runs inference against hand-crafted test cases:

Overall on targeted patterns: 90.8%

The validation set accuracy (92%) masked critical failures on patterns like extended variables and Unicode. Testing on held-out patterns, not just held-out samples, revealed gaps the validation set didn't cover.

The CLI and Testing Process

Vidai ships with a CLI for parsing expressions:

# Parse expression to prefix notation
vidai parse "x^2 + 3*y"
# Output: + ** x 2 * 3 y

# Parse and evaluate with substitution
vidai parse "x^2 + y" --eval x=3 y=4
# Output: + ** x 2 y = 13

# Pure arithmetic evaluation
vidai parse "3 + 5 * 2" --eval
# Output: + 3 * 5 2 = 13

# System info (available models, GPU, etc.)
vidai info

The test suite runs inference against hand-crafted test cases organized by category:

python scripts/eval/test_finetune.py \
    --checkpoint models/finetune_v1_best.pt \
    --verbose

Each category tests specific patterns:
- Trig functions: sin(x), cos(theta), tan(z)
- Left associativity: a - b - c must produce - - a b c, not - a - b c
- Unicode: √x must produce sqrt x

The --verbose flag shows individual failures, which proved essential for diagnosing systematic issues.

The Power of Synthetic Data

Every training example is generated programmatically. No human labeling. No scraping math from the web.

We build the tree first, then render it as text. The prefix notation label is correct by construction, no human judgment required.

# Generate 2M training samples (~60 seconds)
python scripts/data/generate_parser_data.py \
    --output-dir data/parser_v4 \
    --samples 2000000 \
    --mixed

Dataset Evolution Across Versions

The datasets evolved significantly across four major versions, each addressing specific failures discovered in evaluation:

V1: The Precedence Bug

The initial dataset generated random expression trees and rendered them to infix notation without parentheses. The tree Mul(Add(76, 25), 67) became 76 + 25 * 67, but standard precedence parses that as 76 + (25 * 67). The training data contained contradictory labels for identical inputs.

V2: Explicit Parentheses

After discovering the bug, I regenerated all data with explicit parentheses: ((76 + 25) * 67). This made tree structure unambiguous. Accuracy jumped from 68% to 78% immediately.

V3/V4: Mixed Notation Formats

Real mathematical notation varies. To train robustness, V3 and V4 used a carefully designed mix that includes 30% parentheses-free expressions to teach the model operator precedence:

The 30% parentheses-free data was critical for teaching operator precedence. Without it, the model would only learn to copy structure from parentheses rather than understanding * binds tighter than +.

V3 achieved 92% on the validation set, but when I tested on extended variables (r, c, d, v, g), accuracy was 0%. The validation set only tested interpolation, not generalization to unseen patterns.

V4: Targeted Fine-tuning

Rather than retrain from scratch, I generated 74K targeted examples covering:
- Extended variables in diverse contexts
- Unicode symbols (√, ×, ÷)
- Trigonometric functions (sin, cos, tan)
- Left associativity cases (a - b - c)

Fine-tuning for 3,500 steps (about $3 on Runpod) brought accuracy on these patterns from 0% to 90%+.

Cloud Training: From Modal to Runpod

Local training on my MacBook Pro (M4 Pro with the Metal Performance Shaders, or MPS backend) runs at ~1.2 iterations/second. For 50,000 steps, that's 12+ hours. Being GPU poor in this this specific sense, I turned to Modal and then to Runpod.

I started with Modal, which offers a clean Python-native API for serverless GPU compute. You define your training function, decorate it, and Modal handles containerization and scheduling:

@app.function(gpu="A100", timeout=3600)
def train_model(config: dict):
    # Training code runs on A100
    ...

Modal worked well, but at $3.19/hour for A100, costs added up during iteration. I didn't want to get the $250 team plan for this small hobby project. I switched to Runpod for the next few training runs.

Runpod's serverless API is straightforward, but I stumbled through a few bits such as the SSH which strangely did not allow file copies, and introduced its own CLIs runpod and runpodctl. Anyhow, the workflow is below:

export RUNPOD_API_KEY="your_key"

# Start training pod
python scripts/train/runpod_train.py --start --github-token $GITHUB_TOKEN

# Check status
python scripts/train/runpod_train.py --status --job-id <job_id>

# Download results
python scripts/train/runpod_train.py --download --pod-id <pod_id>

For a 44M parameter model, RTX 4090 (24GB VRAM) is sufficient and dramatically cheaper than A100. Total training cost for V4 (pre-training + fine-tuning): approximately $12. However, the RTX4090 was not available as everyone else had the same bright idea, and I was left with the A100 to train this one. A little more money but I am glad I could take a shot at doing this.

One lesson from my Runpod experience: don't use Runpod's SSH proxy (ssh.runpod.io). It blocks PTY and breaks SCP/rsync. Always use the direct IP:port from the Runpod API. This was a drag, and kept a pod running long after it was needed.

What I Learned

Each plateau during development had a clear cause unrelated to model capacity:

The fix was never a better model. It was a clearer formulation of what the model should be learning.

Key lessons:

1. Data quality > model size > training time: The jump from 68% to 95% came from fixing the data.
2. The gap between token and sequence accuracy is diagnostic: A 30-point gap (98% token, 68% sequence) pointed directly at precedence ambiguity.
3. Pre-training + fine-tuning is powerful even at small scale: 74K fine-tuning examples fixed patterns that 1M pre-training samples missed.
4. Tokenizer coverage matters: Missing ln from the vocabulary means the model cannot output it, regardless of training.
5. Test on held-out patterns, not just held-out samples: 92% on the validation set masked 0% on unseen variable names.

Related Work: MathGPT and Tree Decoders

Several approaches have explored tree-based representations for mathematical expressions in neural networks. It's worth comparing Vidai to these methods.

MathGPT modifies GPT-2 by linearizing operator trees (OPTs) via depth-first traversal, adding tree position embeddings (binary representations of sibling indices) plus symbol type embeddings. It uses constrained decoding to ensure valid tree output. On equation extraction tasks, MathGPT achieves 52.4% tree match accuracy versus GPT-2's 47.8%.

Tree Decoders (seq2tree, Graph2Tree) generate expression trees directly for Math Word Problems (MWPs). These approaches use graph-based encoders for input text and tree-structured decoders for output, achieving state-of-the-art results on benchmarks like Math23K.

Skip-tree training masks subtrees in formal math corpora and trains LLMs to predict missing parts, yielding strong logical reasoning capabilities.

How does Vidai differ?

The key philosophical difference: MathGPT and tree decoders attempt to learn mathematical reasoning end-to-end. Vidai deliberately avoids this. The neural network learns only to parse human notation into trees; the mathematics is delegated to symbolic engines that are correct by construction.

This separation has trade-offs:
- Vidai's advantage: 100% computation accuracy, exact fractions, symbolic algebra via SymPy. No risk of the model "hallucinating" that 3 + 5 = 9.
- Vidai's limitation: Cannot learn new mathematical operations or solve novel problem types without extending the symbolic engine. MathGPT can potentially generalize to new patterns if trained on enough examples.

For tasks where correctness is non-negotiable (financial calculations, engineering, scientific computing), Vidai's approach is safer. For exploratory mathematical reasoning where approximate or heuristic solutions are acceptable, end-to-end approaches like MathGPT may be more flexible.

The tree representation insight is shared: both MathGPT and Vidai recognize that mathematical expressions are fundamentally trees, not sequences. MathGPT encodes this through tree position embeddings; Vidai encodes it through prefix notation output. The difference is what happens after the tree is extracted.

Known Issues

What Works Without Parentheses

Testing with the CLI shows that simple two-term precedence works correctly:

$ vidai parse "3 + 5 * 2"
+ 3 * 5 2 = 13.0   ✓  (correctly parsed as 3 + (5*2))

$ vidai parse "10 - 2 * 3"
- 10 * 2 3 = 4.0   ✓  (correctly parsed as 10 - (2*3))

$ vidai parse "a + b * c"
+ a * b c          ✓  (correctly parsed as a + (b*c))

What Fails Without Parentheses

Complex expressions with implicit multiplication or 3+ terms still struggle:

$ vidai parse "x^2 + 3*y"
* + ** x 2 3 y     ✗  Got (x² + 3) * y, expected x² + (3*y)

$ vidai parse "a + b + c * d"
+ a + b * c d      ✗  Got a + (b + (c*d)), expected (a + b) + (c*d)

The model handles basic operator precedence but struggles with:
1. Implicit multiplication combined with other operations
2. Left-associativity in chains of 3+ terms

Despite having 30% parentheses-free data in training, these edge cases remain problematic. The workaround is to use explicit parentheses for complex expressions.

Other Issues

ln function missing from tokenizer

Input:  ln(y)
Expected: ln y
Got:      <unk>n y

The output vocabulary has log but not ln. Fix requires vocabulary expansion and retraining.

What's Next: Higher Mathematics as Trees

The same architecture extends naturally to higher mathematics. The key observation: differentiation, integration, differential equations, and other advanced operations are themselves tree transformations. The neural network's job remains parsing; the symbolic engine handles the mathematical heavy lifting.

Calculus Operations

Differentiation and integration are operators that take an expression and a variable. Consider the derivative of x³ + 2x with respect to x:

Input:  "d/dx(x³ + 2x)"

Tree:
              diff
             /    \
            +      x
           / \
          ^   *
         / \ / \
        x  3 2  x

Prefix: diff + ** x 3 * 2 x x

The diff operator sits at the root, with the expression as its left child and the differentiation variable as its right child. The expression subtree is identical to what we saw in arithmetic.

Integration follows the same pattern:

Input:  "∫ sin(x) dx"

Tree:
          integrate
           /     \
         sin      x
          |
          x

Prefix: integrate sin x x

The neural network learns to parse the many notational variants ("d/dx", "∂/∂x", "f'(x)", "dy/dx", prime notation) into this canonical tree form.

Physics: The Kinematics Equation

Consider a classic physics problem: an airplane accelerating down a runway. The displacement equation is s = ut + ½at². Given initial velocity u, acceleration a, and time t, solve for s:

Input:  "s = u*t + (1/2)*a*t²"

Tree:
                  =
                 / \
                s   +
                   / \
                  *   *
                 / \ / \
                u  t *  ^
                    / \/ \
                   /  1  t  2
                  /   -
                 a    2

Prefix: = s + * u t * * / 1 2 a ** t 2

The entire equation, including the equality, is a tree. Once parsed, SymPy can solve for any variable, substitute values, or derive related equations. The neural network's job is to understand that "½" and "1/2" and "0.5" all mean the same thing, that "t²" means t**2, and that the equation represents a relationship between physical quantities. The network can parse unicode strings and also understand fractions, meaning that this differential equation may be solved in some future version of vidai.

Linear Algebra: Systems of Equations

A system of linear equations is also a tree structure. Consider solving:

2x + 3y = 7
x - y = 1

Input:  "2x + 3y = 7, x - y = 1"

Tree:
              system
              /    \
             =      =
            / \    / \
           +   7  -   1
          / \    / \
         *   *  x   y
        / \ / \
       2  x 3  y

Prefix: system = + * 2 x * 3 y 7 = - x y 1

SymPy's solve function takes this tree and returns {x: 2, y: 1}. The neural network handles the parsing: commas versus newlines, implicit multiplication, various equation separator conventions.

Determinants and Matrices

Matrix operations extend the same idea. A determinant computation:

Input:  "det([[a, b], [c, d]])"

Tree:
            det
             |
           matrix
            / \
          row  row
          / \  / \
         a  b c  d

Prefix: det matrix row a b row c d

The symbolic engine computes ad - bc. For numerical matrices, NumPy handles the computation. For symbolic matrices, SymPy does this.

The Key Point

All of these, from simple arithmetic to differential equations to linear algebra, are text tokens that can be represented as trees. The encoder-decoder model learns the mapping from human notation to a tree structure from training data. The trees are then executed by symbolic engines that have been proved mathematically to work and which have a steady definition of how to execute.

The neural network never learns calculus or linear algebra itself. Instead, it learns that when humans write "d/dx", they mean differentiation, and that when they write "det", they mean determinant. The mathematics is handled by code that does not require gradient descent.

The General Pattern

Each domain follows the same template: the neural network converts human notation into a canonical tree representation, and a symbolic engine executes the mathematics. The parsing problem scales with notational complexity; the computation problem is already solved.

This extends beyond mathematics. Code (parse to AST, execute with interpreter), proofs (parse to inference steps, verify with proof checker), chemistry (parse to molecular structure, simulate with physics engine). The pattern is general: neural network for parsing noisy human notation, symbolic engine for deterministic computation.

Conclusion

Building Vidai taught me something that seems obvious in retrospect: the hardest part of applying mathematical computation using neural networks, is not the computation itself, but the process of understanding what computation is being asked for. Once you frame the problem as parsing rather than calculating, the architecture of the neural network in question becomes clear. I foresee Vidai becoming an important bridge and integrating with frameworks like Praval in future, thereby allowing agents to parse mathematical input natively.

The 27-point accuracy gain from fixing the precedence bug, compared to negligible gains from model scaling, reinforced a lesson that applies broadly: for structured tasks, data quality dominates model size. At some level we have known this since the Textbooks are all you need paper. The model was never the bottleneck. The formulation of the problem was.

Vidai is far from complete. Complex precedence chains still fail. The ln token is missing from the vocabulary. Implicit multiplication combined with other operations confuses the parser. But the architecture is sound, and each failure points to a clear fix in the data or vocabulary rather than a fundamental limitation.

Perhaps most importantly, this project reminded me why I find machine learning compelling. Watching my son struggle with arithmetic, I saw the same parsing challenges that trip up neural networks. The structure must be understood before the computation can proceed. For humans and machines alike, the answer (விடை) comes only after the question is properly understood.

Try It Yourself

You can try Vidai directly in your browser. Enter a mathematical expression and see how the model parses it into prefix notation, then evaluates it using SymPy:

Tips for best results:
- Simple expressions work reliably: 3 + 5 * 2, sin(pi/2), sqrt(16)
- Use parentheses for complex expressions: (x^2) + (3*y) instead of x^2 + 3*y
- Variable substitution: enter x=3, y=4 in the substitutions field

Resources:
- Model: aiexplorations/vidai on HuggingFace
- Interactive Demo: aiexplorations/vidai-demo on HuggingFace Spaces
- Source Code: aiexplorations/vidai on GitHub

References

Lample, G., & Charton, F. (2020). Deep learning for symbolic mathematics. ICLR 2020. arXiv:1912.01412

Liu, T. (2023). Goat: Fine-tuned LLaMA outperforms GPT-4 on arithmetic tasks. arXiv:2305.14201

Nye, M., et al. (2021). Show your work: Scratchpads for intermediate computation with language models. arXiv:2112.00114

Trask, A., Hill, F., Reed, S., Rae, J., Dyer, C., & Blunsom, P. (2018). Neural arithmetic logic units. NeurIPS 2018. arXiv:1808.00508

Wei, J., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 2022. arXiv:2201.11903

Google DeepMind. (2024). AI achieves silver-medal standard solving International Mathematical Olympiad problems.

Veličković, P. et al (2025). Category theory for neural networks. Machine Learning Street Talk.

https://arxiv.org/html/2411.16993v1

A clock face is circular because the arithmetic is circular. Position on the circle encodes the hour; adding time means rotating around the circle. This Welch Labs video on grokking uses this example to explain something surprising that researchers discovered when they trained a neural network on modular addition.

The model OpenAI's team built as per the Grokking paper was tiny, a small transformer learning to compute $(a + b) \mod p$ for some prime $p$. It memorized the training examples quickly and the model loss dropped quite fast, but generalization was poor despite the loss being low. It turns out that someone on the OpenAI team left the model to continue training and came back to the unexpected discovery that performance on held-out test sets was excellent. Long after the training was deemed to be complete, the model had actually "learned" or "grokked".

When they visualized the learned representations, the OpenAI team found interesting weight representations in the layers of the model. The numbers 0 through $p-1$ were arranged around circular patterns in the weight space, with Fourier components encoding position on the circle. The model had independently discovered that addition mod $p$ lives on a circle, just like we discussed in the clock example at the start of this blog. The team had found Lissajous figures in the weight visualizations - the kind you will see on an oscilloscope in a signals laboratory. These were trigonometric embeddings, the same mathematical structure behind positional encodings in transformers, but discovered autonomously for this specific task.

I watched this video months with rapt attention a few days ago, months after I had started building what would become ToDACoMM. Here was evidence that models learn structured geometric representations, and I had been building tools to measure exactly that kind of structure.

My own intuitions came from a few failed experiments on topological data analysis (TDA). I had figured that if only some vectors were represented significantly in vector space there would be voids in the data that was being fed to LLMs, and if the different attention layers and heads we described within models were learning from this data, they too would be developing such voids and other patterns with interesting and non trivial topologies in their weights. It followed that I should explore the nature of the geometry and topology of the weights, because this is where some insight into how models may learn, was likely to be present. I am sure I am not the first guy to think of this, and in fact there has been a lot of research in topological deep learning and TDA for deep learning models. TDA is itself a very old consideration, decades old if not centuries, but it seemed very topical and relevant.

Two Ways of Seeing

In fact, there are two lenses through which I have come to see neural networks, and they are older than neural networks themselves.

The first is dynamical systems. Training is gradient flow on the loss landscape $\mathcal{L}(\theta)$, a trajectory through parameter space following $\dot{\theta} = -\nabla_\theta \mathcal{L}$. Deep learning practitioners know this intuitively: the optimizer moves through weight space, gets stuck in local minima, escapes via momentum or learning rate schedules, eventually settles somewhere useful. Grokking is a phase transition where the system escapes a memorization basin and finds a generalizing solution.

The forward pass is also dynamical. Layer by layer, representations evolve through a composition of nonlinear maps $f_L \circ f_{L-1} \circ \cdots \circ f_1$. Each layer transforms the geometry of the activation space. In deep learning terms: early layers extract low-level features, later layers compose them into higher-level representations. In dynamical systems terms: the input evolves through a sequence of nonlinear transformations, each reshaping the space.

The second is topology. Where dynamical systems ask "how does this evolve?", topology asks "what is the shape of the space it evolves in?"

For deep learning practitioners, think of it this way: when you visualize embeddings with t-SNE or UMAP, you see clusters (similar items grouped together) and sometimes you see loops or manifold structure. Topology formalizes this. Persistent homology captures shape at multiple scales: it tracks how connected components ($H_0$, roughly "clusters"), loops ($H_1$, roughly "circular patterns"), and voids ($H_2$) appear and disappear as you vary a distance threshold. The persistence of a feature measures its significance; noise creates short-lived features, real structure persists.

These two perspectives are not separate. The shape of a space constrains what can happen within it. If representations cluster tightly, certain distinctions become hard to learn. If they spread into loops or manifolds, certain patterns become natural to encode. Measuring the topology of learned representations tells us something about what the training dynamics carved out.

Reading Shape from Points

Imagine scattering a handful of coins on a table. Some land close together, others far apart. If you squint, you might see clusters; coins that fell near each other form natural groups. If you arranged them deliberately in a circle, you would see the ring shape even though the coins themselves are just points. This isn't dissimilar to finding clusters as you might in some data, except we're not looking for decision boundaries in topology.

Persistent homology is a method for detecting such "structure" algorithmically. The idea is simple: grow a ball around each point, starting from radius zero. At first, each point is isolated, and there are as many separate components as there are points. As the radius increases, balls begin to overlap. When two balls touch, their points become connected and two components merge into one. These are connected components. Keep growing, and eventually everything connects into a single blob.

The trick is to watch what happens along the way. Components that merge quickly were close together; they were probably part of the same cluster. Components that persist as separate until late in the process were genuinely far apart. A feature that appears and disappears quickly is likely noise, and a feature that persists across a wide range of radii reflects real structure in the data. The latter mechanism is quite intuitive if you had begun to imagine these balls which we grew intersecting and becoming connected components in your mind's eye.

This presence of features across a wide range of radii is the "persistent" in persistent homology: we care not just about what features exist, but how long they last.

Loops work similarly. As balls grow and overlap, they sometimes form closed rings before filling in completely. If five points are arranged in a pentagon, the balls will first connect into a cycle, and only later will the interior fill in when the radius grows large enough. The cycle is born when the ring closes and dies when the interior fills. A cycle that persists for a long time indicates genuine circular structure; one that dies immediately was just an accident of the point configuration.

The Vietoris-Rips complex is the specific construction that makes this precise, rather than just an arbitraty mechanism. At each radius $\epsilon$, we connect points that are within distance $\epsilon$ of each other. As $\epsilon$ grows from zero to infinity, features appear and disappear. Ripser is an algorithm, implemented as a Python library, that computes this efficiently even for thousands of points in dozens of dimensions. It returns a list of birth-death pairs: each pair records when a topological feature was born (at what radius) and when it died. The difference, death minus birth, is the persistence.

In the language of homology: $H_0$ counts connected components (clusters), $H_1$ counts loops (circular patterns), and $H_2$ counts voids (hollow cavities). For analyzing neural network representations, $H_0$ and $H_1$ are the most informative, because we are dealing with layers in neural networks. High $H_0$ persistence means the points are spread out, with well-separated clusters. High $H_1$ persistence means there are genuine circular or periodic structures in the geometry.

When I run Ripser on the activations of a transformer layer, I am asking: what is the shape of the space these representations occupy? Are they clustered or diffuse? Do they trace out loops? The answers turn out to differ dramatically between encoder and decoder architectures.

What Models Learn

Neural network training is iterative error correction: forward pass, loss computation, backpropagation, weight update. The dynamics converge (when they converge) to regions of weight space where the model's internal representations support accurate prediction.

What are these representations? For a transformer processing text, each layer produces activations $h^{(l)} \in \mathbb{R}^{d}$ for each token. If you've worked with transformers, you know these aren't arbitrary vectors. The embedding layer maps tokens to a learned space where semantic similarity corresponds to geometric proximity; "king" and "queen" are closer than "king" and "banana". Attention layers then transform these representations based on context, and feedforward layers apply nonlinear transformations.

For the model to generalize, it must organize representations so that similar contexts cluster, syntactic patterns are geometrically encoded, and semantic relationships become spatial. The model learns a representation manifold, a high-dimensional space where the structure of language is reflected in geometry.

This manifold is shaped by the training dynamics. Each gradient update pushes and pulls the representation geometry, separating what should be distinguished, clustering what should be similar. When we measure the topology of trained representations, we are measuring what the optimization process carved out.

The Grokking paper showed one such carving, Fourier circles for modular arithmetic. Circles are the right geometry for cyclic groups. What shapes do language models carve? What is the topology of GPT-2's representation space versus BERT's? This is what ToDACoMM was built to investigate.

The Metrics: What We Measure and Why

Before diving into the tool and its findings, it helps to understand what we are actually measuring and what each metric tells us about neural network representations.

Topological Metrics

H0: Connected Components (Cluster Structure)

H0 counts how many separate "islands" exist in the data at different scales. If you imagine the point cloud of neural network activations, H0 asks: how many distinct clusters are there, and how far apart are they?

When H0 persistence is high, representations are spread out with well-separated groups. When it is low, everything clusters together. For a classifier, you might expect H0 to increase through the network as the model separates different classes into distinct regions. For a language model, the pattern is more complex; representations must both cluster (similar meanings together) and spread (different contexts distinguishable).

H1: Loops (Cyclic Structure)

H1 counts circular patterns in the data. If representations trace out a ring or cycle as you vary some property of the input, H1 detects it. The grokking model learned circles because modular arithmetic is inherently cyclic; H1 would capture this.

In language models, H1 might reflect periodic patterns: days of the week forming a cycle, verb conjugations with recurring structure, or the periodicity inherited from positional encodings. High H1 persistence means these cycles are robust features of the representation geometry, not artifacts.

Persistence: Separating Signal from Noise

Not all topological features are meaningful. Some clusters merge quickly; some loops fill in immediately. Persistence measures how long a feature survives as we vary the scale parameter. A feature with high persistence, one that appears early and dies late, reflects genuine structure. A feature with low persistence is likely noise.

When we report "total H0 persistence" or "max H1 lifetime," we are summarizing how much robust structure exists. A model with high total persistence has carved out a more structured representation space.

Geometric Metrics

Topology tells us about shape, but not about scale or density. The geometric metrics fill this gap.

Intrinsic Dimension

A 768-dimensional activation vector does not actually use all 768 dimensions. The data lies on a lower-dimensional manifold embedded in this high-dimensional space. Intrinsic dimension estimates the true dimensionality of this manifold.

If GPT-2's embeddings have intrinsic dimension 32, the model is using roughly 32 independent degrees of freedom to encode token meanings, despite the 768-dimensional container. Watching intrinsic dimension change through layers reveals how the network compresses or expands its representation complexity.

Hubness

In high-dimensional spaces, distance behaves strangely. Some points become "hubs," appearing in many other points' nearest-neighbor lists simply due to the geometry of high dimensions, not because they are semantically central. This is the curse of dimensionality manifesting in k-NN structure.

A hubness score of 1.0 indicates uniform k-NN structure: no pathological hubs. Scores above 1.0 indicate hub points exist. For neural network representations, we want hubness near 1.0; it means the learned space has healthy geometry where nearest-neighbor relationships are meaningful, not artifacts of dimensionality.

Distance Distribution

How far apart are representations? The mean, variance, and shape of the k-NN distance distribution tell us whether points are tightly packed, spread out, or somewhere in between. Tracking this through layers reveals whether the network expands representations (spreading them apart) or compresses them (pulling them together).

Why These Metrics Together

No single metric tells the full story. Intrinsic dimension might drop while H0 persistence rises; the network is compressing to fewer dimensions but spreading points within that subspace. Hubness might normalize while H1 count stays constant; the k-NN structure improves without changing the topological complexity.

The combination of topological metrics (H0, H1, persistence) and geometric metrics (intrinsic dimension, hubness, distance distribution) provides a multi-faceted view of representation geometry. Together, they characterize what gradient descent carved out.

Measuring the Carved Space

ToDACoMM (Topological Data Analysis Comparison of Multiple Models) characterizes transformer representations using persistent homology. The pipeline:

1. Extract activations ${h_i^{(l)}}$ at each layer $l$ for $n$ text samples
2. Project to $k=50$ principal components (retaining ~95% variance)
3. Compute Vietoris-Rips persistent homology via Ripser
4. Extract topological summaries: total persistence, max lifetimes, feature counts

System Architecture

Quick Start

# Install
git clone https://github.com/aiexplorations/todacomm
cd todacomm
pip install -e ".[dev]"

# Analyze a single model
todacomm run --model gpt2 --samples 500

# Compare encoder vs decoder
todacomm run --models gpt2,bert --samples 500

# Use all layers (14 for GPT-2)
todacomm run --model gpt2 --layers all

# GPU acceleration
todacomm run --model gpt2 --device cuda

Supported Models

ToDACoMM includes 20+ pre-configured transformer models under 1B parameters, plus configurable MLP architectures:

MLP models support MNIST, FashionMNIST, and UCI tabular datasets (iris, wine, digits, breast_cancer). Custom HuggingFace models: todacomm run --hf-model <model-name> --num-layers <N>

Output Structure

Each experiment generates:

experiments/<model>_tda_<timestamp>/
├── runs/run_0/
│   ├── tda_summaries.json      # H0/H1 metrics per layer
│   ├── metrics.json            # Perplexity, accuracy
│   ├── tda_interpretation.md   # Human-readable analysis
│   └── visualizations/
│       ├── tda_summary.png     # 6-panel metric overview
│       ├── layer_persistence.png
│       └── betti_curves.png
└── reports/
    └── experiment_report.md    # Full analysis report

The visualization plots show:
- tda_summary.png: H0/H1 count, total persistence, and max lifetime across layers
- layer_persistence.png: Side-by-side comparison of H0 vs H1 evolution
- betti_curves.png: Feature count trends through the transformer stack

TDA Methodology Details

The dimensionality reduction step is practical: persistent homology on 768-dimensional point clouds is computationally prohibitive. PCA to 50 dimensions preserves most of the variance while making computation tractable. This is a tradeoff; we might miss structure in the discarded components, but the patterns that emerge are robust across different choices of $k$.

The Vietoris-Rips complex works by growing balls around each point. At radius $\epsilon = 0$, each point is its own connected component. As $\epsilon$ grows, balls overlap, points connect, and the topology changes. The algorithm tracks when topological features (components, loops) are born and when they die. A feature that persists across a wide range of $\epsilon$ is likely real structure; a feature that dies quickly is likely noise.

The key metrics:

• H0 Total Persistence: Sum of lifetimes of all connected components. In deep learning terms: how spread out are the representations? If activations form tight clusters, H0 is low. If they spread across the space, H0 is high.

• H1 Total Persistence: Sum of lifetimes of all loops. In deep learning terms: are there circular or periodic patterns in the representation geometry? High H1 indicates the model has learned representations with loop structure.

• Expansion Ratio: $\text{peak}(H_0) / H_0^{(0)}$, where $H_0^{(0)}$ is the embedding layer. This captures how much the geometry transforms through the network. A ratio of 1x means the representation geometry doesn't change much from embedding to final layer. A ratio of 100x means dramatic expansion.

I analyzed ten models across five architecture families (GPT-2, BERT, Pythia, SmolLM2, Qwen), each processing 500 WikiText-2 samples. Bootstrap resampling (B=100) provided 95% confidence intervals.

Beyond Transformers: MLPs as a Baseline

Before examining transformer topology in depth, it helps to understand what TDA reveals about simpler architectures. ToDACoMM now supports multi-layer perceptrons, the feedforward networks that predate attention mechanisms by decades. Without the recombinatory complexity of attention, the geometric transformations in MLPs are more direct, providing a cleaner baseline for interpretation.

A 3-layer MLP trained on MNIST (784 → 256 → 128 → 10) shows something quite different from transformers. The input layer, a 784-dimensional space of pixel intensities, has an intrinsic dimension around 15; the MNIST manifold is far smaller than the ambient space suggests. Through the hidden layers, this dimension compresses further, dropping to approximately 7 by the output layer. The network learns a progressively lower-dimensional representation as it approaches the 10-class decision boundary.

The hubness score tells a parallel story. In high-dimensional spaces, some points become "hubs", appearing disproportionately often in other points' nearest-neighbor lists due to the concentration of measure. Good representations should have hubness near 1.0, indicating uniform k-NN structure. The MLP achieves this; starting at 0.63 and remaining below 1.0 throughout, the network maintains uniform neighborhood structure as it compresses.

Both H0 and H1 persistence decrease monotonically through the MLP layers. The input has the highest topological complexity; the output has the lowest. This is compression in the topological sense: the network simplifies the representation space as it projects toward class centroids.

Comparing across depths reveals a consistent pattern: deeper networks compress more aggressively. The 2-layer network ends with intrinsic dimension around 7.7; the 6-layer network compresses to 4.5. Hubness drops toward zero in deeper networks, and both H0 and H1 persistence decrease with depth. More layers means more opportunity to simplify the representation geometry.

This contrasts sharply with the expansion patterns of transformers. Where GPT-2 spreads representations outward as it processes tokens, MLPs compress them inward. Both are valid geometric strategies serving different computational goals: transformers accumulate context across tokens, while MLPs project inputs toward decision boundaries.

Scaling Up: 20,000 Samples

The initial transformer findings with 500 samples raised a question: would the patterns persist at scale, or were they artifacts of limited sampling? With 500 points, the Vietoris-Rips complex is computationally tractable, but statistical confidence is limited.

The latest version of ToDACoMM supports extraction of 10,000 to 50,000 activation samples from transformers, with intelligent subsampling for the TDA computation itself. The workflow: extract activations for all samples, characterize geometry (intrinsic dimension, hubness, distance distributions) on the full set, then subsample to 2,000 points for Ripser. This preserves the statistical benefits of large samples while keeping homology computation feasible.

The geometry characterization step, new in this version, computes several metrics before TDA:

• MLE intrinsic dimension: Maximum likelihood estimate of the manifold dimension
• Local PCA dimension: Average dimensionality of local neighborhoods
• Hubness score: Skewness of the k-occurrence distribution
• Distance statistics: Mean, variance, and distribution of k-NN distances

These metrics are cheaper to compute than persistent homology and provide complementary information. Intrinsic dimension tells us how many degrees of freedom the representations actually use; hubness indicates whether the space has pathological concentration; distance distributions reveal the spread and clustering of points.

GPT-2 at Scale: Compression Through Depth

Running GPT-2 (124M parameters) and GPT-2-medium (354M parameters) on 20,000 WikiText-2 samples revealed something I had suspected but not confirmed at smaller scales: the intrinsic dimension of representations compresses dramatically through the network.

At the embedding layer, GPT-2's MLE intrinsic dimension is approximately 32. By the final layer, it drops to around 14, a 57% reduction. The model starts with high-dimensional token embeddings and progressively squeezes them into a lower-dimensional subspace as it builds contextual representations.

The hubness trajectory is equally striking. The embedding layer has hubness of 4.58, indicating severe hub structure; some token embeddings appear in many other tokens' nearest-neighbor lists. Through the layers, hubness drops monotonically, reaching 1.05 by the final layer. The network transforms pathologically concentrated embeddings into uniformly distributed representations.

The distance distribution reveals something unexpected: representations spread apart dramatically in middle layers (peaking at layer 9), then reconsolidate in final layers. The mean k-NN distance rises from 0.48 at embedding to 34.2 at layer 9, then falls to 8.6 at the final layer. This expansion-then-contraction pattern suggests the network first separates representations to build context-specific meanings, then gathers them back for prediction.

The TDA metrics corroborate this picture. H0 total persistence (cluster spread) peaks at layer 9, matching the distance distribution peak. H1 count (loop structures) remains relatively stable at around 1,700 across all layers, suggesting the cyclic structure is inherent to the representation manifold rather than an artifact of particular layers. The "Dim vs Topology" scatter plot shows the progression: embedding starts high-dimensional with moderate loops, middle layers spread out in both dimensions, and final layers compress dimensionally while maintaining topological structure.

Comparing GPT-2 to GPT-2-medium reveals scaling effects. The larger model maintains slightly higher intrinsic dimension (15.0 vs 13.8) and achieves near-perfect hubness normalization (1.0 vs 1.05). Interestingly, GPT-2-medium has fewer H1 loops (1,438 vs 1,720) despite having nearly 3x the parameters. More capacity may enable simpler topological structure; the larger model can represent the same information with less geometric complexity.

SmolLM: A Different Architecture, A Different Trajectory

SmolLM presents a contrast to GPT-2. Where GPT-2 has 12 layers with 768-dimensional hidden states, SmolLM-135M has 30 layers with 576-dimensional hidden states. More layers, smaller width. Running the same 20,000-sample analysis reveals a fundamentally different geometric trajectory.

The intrinsic dimension trajectory shows two distinct phases. In layers 0-7, dimension remains high around 43, essentially unchanged from the embedding. Then a sharp transition occurs: by layer 14, dimension drops to 27, where it plateaus through layer 21. This is not the smooth, monotonic compression of GPT-2; it is a phase transition, a discrete jump in the representation regime.

The penultimate layer anomaly is the most striking feature. Layer 28 shows hubness spiking to 7.81, far above any other layer. The k-NN distance distribution explodes, with mean distance reaching 204 (compared to 64 at layer 21). Something dramatic happens in the penultimate layer: representations spread apart explosively, creating severe hub structure, before the final layer re-normalizes them.

What could cause this? One possibility: the penultimate layer is creating maximally separated "hub" representations to give the final layer clean inputs for prediction. Another: the architecture's depth-to-width ratio creates bottlenecks that manifest as geometric instability near the output. The pattern is reproducible; SmolLM-360M shows the same penultimate spike at layer 24, though attenuated (hubness 3.33, distance 156).

Comparing SmolLM to GPT-2 at similar parameter counts reveals architectural fingerprints:

SmolLM retains higher intrinsic dimension (20 vs 14), maintains higher hubness (~2 vs ~1), and builds more topological loops (~2,000 vs 1,400-1,700). The deeper, narrower architecture preserves more structure through to the final layer. Whether this is beneficial depends on the task; more structure means more information retained, but also more complexity to navigate during inference.

The two-phase compression and penultimate anomaly are SmolLM's topological signature. They distinguish it from GPT-2's smooth expansion-contraction as clearly as GPT-2's 95x expansion ratio distinguishes decoders from BERT's 2x.

The Encoder-Decoder Divide

BERT showed an expansion ratio of 2x. Representations in the final layer were only twice as spread as in the embedding layer.

GPT-2 showed 95x. Other decoders ranged from 55x (DistilGPT-2) to 694x (SmolLM2-360M).

Why such a stark difference?

Consider what BERT and GPT-2 are doing differently. BERT uses bidirectional attention: every token attends to every other token from layer one. When processing "The cat sat on the mat", the representation of "cat" at layer 1 already incorporates information from "sat", "mat", and everything else. The full relational structure is available immediately.

GPT-2 uses causal attention: each token can only attend to preceding tokens. The representation of "cat" at layer 1 only knows about "The". By layer 6, it knows about "The cat sat on the". By the final layer, it has accumulated the full prefix. This progressive accumulation requires the representation to expand; each layer must encode more context than the last.

In geometric terms: BERT's representations don't need to unfold because all context is accessible from the start. GPT-2's representations must unfold progressively, encoding an expanding window of context into the geometry. The 2x versus 55-694x expansion ratios are the topological signature of this architectural difference.

Architecture Fingerprints

Within decoder families, topological signatures remained consistent across training variations. The three Qwen variants (2-0.5B, 2.5-0.5B, Coder-0.5B) showed expansion ratios of 629x, 673x, and 642x respectively. These models differ in training data (general vs code) and version, but their topological fingerprint is stable.

Pythia scaled with model size: 143x at 70M parameters, 189x at 410M. More parameters means more capacity to expand the representation space.

These are fingerprints. Architecture determines topological regime more than training recipe does. If you told me a model's expansion ratio, I could likely guess its architecture family.

SmolLM2 was anomalous, with expansion varying from 298x (135M) to 694x (360M). This might reflect architectural differences between sizes, or something about how this family encodes information. The variance is worth investigating.

Cyclic Structure

Every model showed non-trivial H1 at 500 samples. There are loops in the representation geometry of every transformer I examined.

What does H1 mean in deep learning terms? If representations trace out a circular path in activation space as you vary some property of the input, that's H1. The grokking model learned circles because modular arithmetic is cyclic. What circular structure might language models learn?

Possibilities: syntactic patterns that recur (subject-verb-object cycles), semantic fields with circular relationships (days of the week, compass directions), positional patterns from the periodic positional encodings. The H1 persistence might be detecting some of this learned periodicity.

SmolLM2-360M stood out: H1 total persistence of 129.52, more than 3x higher than the next model. This model builds unusually strong cyclic structure. I do not yet understand what this corresponds to in terms of learned features, but it distinguishes this model topologically.

Limitations

ToDACoMM is descriptive, not predictive. It measures representation geometry but does not explain why models behave as they do. The 55x expansion of DistilGPT-2, for example, coincides with best-in-class perplexity among decoders, but we know that correlation does not automatically imply causation and so we cannot claim that that is a causal relationship. However, ToDACoMM may reveal a few interesting directions for teams who want to build models, and who might use such findings as a way to steer the direction of their model's development.

Further, ten models across five families is enough to see some general patterns and form hypotheses. It is not enough to make strong claims about the mechanism of learning purely using topological methods or measures. From a data standpoint, WikiText-2 is one dataset, and our findings on the topology of learning in these models might differ on other datasets.

Another thing worth bearing in mind, is that persistent homology is a coarse invariant. Two spaces with identical $H_0$ and $H_1$ can differ in geometrically significant ways. We are measuring coarse-grained shape, and not fine structure.

The PCA projection discards information. Patterns in the discarded 5% of variance might matter. This is a pragmatic choice, not an ideal one.

While these are not reasons to dismiss the findings, they impose restrictions on what we can claim. ToDACoMM is, therefore, an empirical tool for characterization, not a full theory of representation learning.

The Shape of What Was Carved

Gradient descent on the loss landscape carves out a representation manifold. The topology of this manifold reflects both the optimization dynamics and the structure of the training data.

Encoders, with bidirectional attention, carve compact spaces; context is globally available, so representations don't need to expand to encode it. Decoders, with causal attention, carve expansive spaces; context must be accumulated layer by layer, and the accumulation manifests as geometric expansion.

The 2x versus 55-694x divide follows from attention's arrow. This isn't a mysterious emergent property; it's a direct consequence of what the architectures are computing.

Within the carved spaces, cyclic structures form. Whether these reflect the periodicity of language, learned positional structure, or something else, they are consistently present. The grokking paper showed that models can learn geometrically appropriate representations (circles for cyclic arithmetic). The H1 findings suggest language models also learn geometric structure that reflects their training data.

Poincaré connected topology and dynamics in the 19th century. Neural networks are a domain where this connection can be measured empirically.

Directions

The large-scale experiments with GPT-2 and the MLP baselines have answered some initial questions while opening new ones. The encoder-decoder divide persists at 20,000 samples; the expansion-then-contraction pattern in GPT-2's distance distributions is robust. But several threads remain:

• Training dynamics: Track topological changes during training. Does grokking have a topological signature? When does the encoder-decoder divide emerge? The MLP training pipeline now makes this tractable; one could checkpoint activations at each epoch and watch the geometry evolve.

• The expansion-contraction pattern: GPT-2's representations spread dramatically in middle layers then reconsolidate. SmolLM shows a different pattern: two-phase compression with a penultimate layer explosion. Is this a width-vs-depth tradeoff? Do wider models (GPT-2) smooth their trajectories while deeper, narrower models (SmolLM) accumulate instabilities?

• The penultimate layer anomaly: SmolLM's layer 28 shows extreme hubness (7.58) and H0 persistence (253k), an order of magnitude above surrounding layers. This pattern is reproducible across SmolLM sizes. What computational purpose does this serve? Is it a bottleneck effect, a feature of the architecture, or something about how the model was trained?

• MLP depth effects: Deeper MLPs show monotonically simpler final topology. Is there a critical depth beyond which additional layers provide diminishing topological simplification? The 6-layer preset approaches this question but does not answer it.

• $H_1$ interpretation: The cyclic structures persist across model scales and architectures. What do they correspond to? The stability of H1 count across GPT-2's layers (around 1,700 throughout) suggests these cycles are fundamental to the representation manifold, not layer-specific artifacts.

• Connection to weight dynamics: I've been exploring a complementary approach in Deep Learning Dynamics, which uses perturbation analysis and Lyapunov exponents to measure how neural network weights evolve during training. ToDACoMM measures what gets carved in activation space; Deep Learning Dynamics measures how the carving happens in weight space. The preliminary finding that transformers universally diverge in weight space while showing dramatic expansion ratios in activation space suggests these phenomena may be related. The new geometry characterization metrics (intrinsic dimension, hubness) provide additional handles for connecting the two perspectives.

The framework is open source. The methodology and statistical analysis are documented. Others can extend this to their own models and domains.

ToDACoMM is available on GitHub. The technical report includes methodology, bootstrap confidence intervals, and ablation studies.

It is December, 2025 and I find myself reflecting on a very eventful year for me - a year of prolific building, exploration, reflection and scaling up - both at work and in terms of technical and intellectual side projects that touch AI. We've all been bombarded by AI innovations and enhancements from the frontiers and I am no different. This has been a year of plenty in that sense, in the sense of discovering how to work with AI tools to build, explore, and scale that which we want to do. The new AI projects both on the work front and open source AI innovation on the personal front have been plentiful.

This year I made 587 GitHub contributions, across 18 new repositories, and published two frameworks - Praval and Vajra Search - on PyPI. Professionally, this year saw tens of new initiatives at work, and have been for the past few months in an expanded role at work that touches the strategy of how we use and build AI features, how we build with AI, and more. The above contributions are not including the stuff I do at work - where I lead a team building a SaaS platform for HR.

Looking back on 2025, I find a tension running through everything I built, everything I abandoned, and everything I learned. This tension was born out of a recognition that the field of AI and many other fields as of 2025 stand forever changed. Professionals like me are using AI tools to become more productive and expand our execution capabilities on the one hand, even as we are aware of the dependency on AI tools on the other hand. This has led to a tension I have discussed elsewhere - of needing to use the tools on the one hand to benefit from them, and on the other hand, not wanting to abandon the craft of programming and engineering.

In this post, I will take you through what it was to build, explore and do much more than I would individually have been able to do with the help of AI, and what I learned from these experiences. I will also walk through Praval, Vajra, and other things I worked on.

2025 at Work - A Short Summary

At work, I brought both greater innovation and greater professional growth to my team. We built a credible capability in building AI agents end-to-end, and have been leveraging this for numerous projects. I believe that the older LLM powered workflows will soon be replaced by AI Agents for a number of use cases. The AI teams I've worked with, which are my own and the team I collaborate with, have been incredible.

From a people standpoint, finding the right talent brings me a lot of satisfaction these days, and my team have coalesced into an effective unit in the last eight months. In mid 2025, I was promoted to a broader role, and this was a vindication of some of the calls I have had to take, with respect to career, team, company and more. I'd say looking back that deep involvement and an innovation mindset have been strong contributors to my own growth. Despite running a remote team, synergies have been possible, thanks to the leadership and the horizon-scanning we all in AI leadership do.

The innovation capacity of this team, and a forward-thinking leadership and strategy team keep me interested - these are the nuts and bolts of how I have been able to deliver through others effectively at work. It takes a new breed of leader to successfully navigate the changing landscape of new models, new benchmarks, new AI tools, while handling the interpersonal and team dynamics of a team that has high-end talent and which is shipping AI products. I am grateful to be working with such leaders at work. There's never been a more VUCA environment than 2023 for me, but 2025 was close. In a sense, this was a good kind of uncertainty, driven by innovation and not Covid or post-Covid based risk. That is perhaps why I think this year was a great test of our ability to stay relevant as a team and for me personally, as a leader and innovator in the AI space.

My team is trying to stay relevant in the fast-changing software engineering space. As I lead a team that's building software for a more traditional industry that's being yanked into the world of AI powered SaaS, we had to learn to build and ship at scale and fast. Despite processes, constraints, architecture support, and lots of resources, it takes strong behavioural changes to actually ship things fast and at high quality. For me, this has meant digging into every system at work, be it our clusters, our repos across products and how we run them, how the team tests and ships things, how we write specifications, etc. One of the big enablers here too has been AI. I have not seen a single user story in the last 3 months at work which did not have objectives, acceptance criteria and the like written out in it. Rarely have seen a PR of late without a good conventional commit style description. This took a while, and the tooling has, I feel, finally caught up with the ambitions I have had for our standards of work. While we have always worked in two week sprints for the last several years, of late, we tried the 3 week development sprint with a testing mini sprint that's bolted on to it, to test and ship things each month. A new SDLC realignment helped us go back to the faster 2 week sprint's pace.

I've honestly grown quite fond of the team I'm working with and leading at work in the last year. The incredible innovation and speed they've shown is especially inspiring, especially on AI agents and the process of building full end-to-end features with them. Not only does my team understand the fast changing landscape of tools, technologies and capabilities, but are able to scale up to new challenges and innovate really fast. This has made all the difference to our effectiveness over the last year.

Coding Assistants and AI Powered Tools in 2025 and Looking Ahead

All through the year, my team's uptake of Cursor and Agentic IDEs has been a significant phenomenon. This inspired me to use Claude Code, to which I have effectively switched, for all my personal projects and open source projects. From rules and configurations that make Cursor effective for us, to subagents, extensive process level checks, and many more innovations, the results of AI powered software engineering have been commendable. A big realization for me personally, has been that there's no going back. Software will be written by AI agents, and we will be building AI agents for different use cases in future. In this sense the "AI is eating software" narrative continues from 2025 into 2026 as well, since the innovation in that space does not stop. There are caveats just as there are with any technology - AI slop is one of the defininig negative trends of 2025. While this is discussed in the context of the enshittification of the internet and its content, there is something to be said for the same for code. Vibe coded applications have gone mainstream in 2025, and at work too, there was a tension on the use of coding assistants. My best engineers leverage AI tools for what it is good at. Although the puck keeps moving on this one, as models and agents get better, it helps to be prepared to write things the way humans wrote them for years before the advent of AI agents and coding assistants.

I learnt many other lessons that threw caution to the winds as well. Chiefly, the architectural and first principles thinking that's crucial to us, and as I say to my team - "the need to be the pilot even if you have an AI copilot". This is crucial. The direction and the vision for any product or system we build with AI or without has to be owned by humans, and not by AI. Many friends feel that GPT-5.2, Claude Opus 4.5 are two-of-a-kind models as of December 2025, and we don't know if this represents a form of artificial super intelligence. (Note that I didn't say artificial general intelligence as I think that is a pipe dream.) I would not entirely disagree with them, because I think scaling laws have gotten to a point where complex internal representations in deep learning models can yield true conceptual understanding. This all reminds me of the Grokking paper, which describes how neural networks actually learn. Why is this relevant? Well, if an LLM is an ideation partner, where do we draw the line in terms of what is directed by human intelligence and what is directed by AI? Where do we draw the line in terms of the genesis or the validation of ideas? I am reminded of a talk by Qodo founder Itamar Friedman here - his pivot from building coding assistants to building code quality tools for AI coding assistants is an interesting pivot. The standards we need can only be defined by humans, and not by AI. And this is possible by humans because the biggest context engineering pipeline is the physical layer of reality. That might sound cheeky, but it is perhaps the truth. This crucial mental model has to be internalized if AI professionals have to stay relevant in 2026 and beyond.

Another key lesson from 2025, is that ideas are extremely cheap, and execution is becoming cheaper. The GLM-4.x models from Z.ai have become both inexpensive to run and a worthy code generation competitor to Claude Code with the Claude Opus 4.5 model. This has happened just in the past few days, and I expect this momemtum to carry on into 2026. Kimi K2 Thinking and DeepSeek v3.2 represent SOTA AI models that are available at the fraction of the cost of the big $200/month subscriptions from Claude Code or OpenAI. This makes them more than suitable for content and code generation at scale and low cost. I expect that in 2026, tokens will become too cheap to meter for many SOTA models. The ChatGPT Go subscription launched in India is an example of a big lab taking advantage of this phenomenon, and we are likely to see others follow suit. This race to the bottom will make advanced capabilities table stakes more often in 2026, and have a further impact on the use of these tools for work, coding and so on. As of this writing in December 2025, AI agents are still unable to build, test and deploy entire software applications yet at scale with one prompt or one click. I think this is set to change in the coming year or two. OpenAI's founder alluded to "on-demand software" as being a trend during his talk at the GPT-5 launch. I personally think this is a secular trend, like small language models have been in 2025. This will all help make coding assistants become less expensive, faster and higher quality in 2026.

Personal Projects and AI Explorations: What I Built in 2025

Building agentic AI applications in 2025 was a natural extension to the LLM based applications I built in 2024. While Ollama and local LMs were the flavour of last year, the reducing costs of AI APIs has meant that I did a lot more experimentation in 2025 with OpenAI. This year has also seen me embrace Claude Code and terminal UIs and back track on using Cursor a little bit for personal coding projects and open source projects. Cursor still remains a competent agentic IDE and especially after Cursor's Composer model, seems to have been rejuvenating developer interest in the tool. Cursor helped me build a number of applications earlier in the year, but the bulk of what I've built in the latter part of the year has been with Claude.

Architectural thinking is one of the biggest areas where I have gained from these experiences. As someone who came into 2025 without a lot of experience building full stack applications, this year has been transformative. Having spent the initial portion of the year at work building on top of agentic AI and MCP capabilities, I found myself pivoting to my own tooling in the latter part of the year for these tasks. I wrote a few MCP servers and my own agentic AI framework, both of which were exciting developments.

As the year progressed, I found myself exploring frontiers I had not touched a lot more. A good example of this is Tlon mathematics, and another is category theory based search, viz. Vajra Search. I also dipped into mechanistic interpretability of deep learning and transformer models later in the year. I will discuss these below. Behind this front of three or four open source projects that are meaty and substantial, there were at least a hundred experiments on various topics.

Below, I cover each of these projects in a little bit of detail. On this site, there are other posts that describe numerous challenges, issues and obstacles I have overcome in the year, and how I did so. Here, though, there is a short summary in each case.

Praval

The personal project I am most excited about this year is a multi-agent AI framework where agents collaborate like coral polyps forming a reef. This is now open source, on PyPI and Github and eagerly seeking contributors to build on top of it. The name I chose for this framework is Praval, which is Sanskrit word for coral (प्रवाल), and the metaphor runs deep; just as coral reefs emerge from simple organisms coordinating without central control, Praval agents broadcast knowledge through "spores" and respond to what they find relevant, with no manager directing traffic.

The framework reached version 0.7.20 this year, published on PyPI, with a decorator-based API that feels natural to Python developers:

@agent("researcher", responds_to=["query"])
def researcher(spore):
    findings = chat(f"Research: {spore.knowledge['topic']}")
    broadcast({"type": "analysis_request", "data": findings})

Praval allows for a similar decorator for tools (@tool), native memory capabilities using Chroma DB, and with support for Qdrant. Agent to agent communication is a native feature of Praval, as is knowledge passing between agents using Spores. Check out the framework at the official website.

What excites me about Praval isn't just the code; it's what it represents. Most agent frameworks assume a hierarchical structure, with orchestrators directing workers. Praval bets on emergence, on the idea that intelligence can arise from peer interactions without anyone being in charge. Whether that bet pays off in production systems remains to be seen, but the exploration has been valuable regardless.

I also found out where Praval does not work well - Praval Code being one such project. Code generation agents, like Claude Code, which I love using for all my work, are not a good fit for Praval. The core reason for this is that code generation seems to be favoured by hierarchical, sub-agent patterns, unlike Praval, which is set up to be suitable for large scale multi-agent collaboration without a central orchestrator. I tried building Praval Code but unsuccessfully. That said, I expect this to be a fruitful area of exploration in the future.

The ecosystem for Praval has Praval Deep Research as a showcase project. This is a cool local-first app for researchers to find papers on ArXiv and chat with their findings. Apart from other experimental projects I've been exploring with friends, such as Praval Analytics, I've also been sketching out Praval Medha, a conceptual system where agents spawn other agents based on problem requirements; agents that architect agents. This is a project I am very excited about, and I expect to be able to build on it in the coming year.

What is the end-game here with Praval? I was asked this by many friends and mentors. Making Praval an open source project allowed it to become a community project, and I expect this to be a fruitful area of exploration in the future. Thanks to incredible collaborators like Bargava and mentors like CM I have been able to sink some time into this framework. What I look forward to with Praval:

• Contributors who I can work with to add many new features for Praval
• Users who can build on top of Praval to create new and innovative applications
• Users who can help me build a community around Praval and educate and engage with developers who use Praval

GitHub: github.com/aiexplorations/praval | PyPI: pypi.org/project/praval

Related Project: github.com/aiexplorations/praval_deep_research

Vajra BM25

If Praval was about agentic AI and multi-agent collaboration, Vajra BM25 was about mathematical foundations and high-performance search powered by a combination of clean and composable search abstractions and low latency performance. The name "Vajra" comes from Sanskrit (वज्र, "thunderbolt"), and the project began as an experiment: could category theory abstractions make a search engine's code cleaner? The inspiration came from Bartosz Milewski's lectures on category theory, combined with work on Elasticsearch at my day job.

With category theory, I was able to frame BM25 search using coalgebras (state → possible next states) and morphisms (composable transformations). Search becomes "coalgebraic unfolding" where a query state unfolds into ranked results. The codebase has a categorical/ module with Morphism, Functor, and Coalgebra base classes that derived implementations extend.

There's something interesting that I learnt about category theory and its relevance and applicability to search in this project. In a nutshell, category theory did not make Vajra fast. The speed came from engineering choices - NumPy vectorization, sparse matrices, LRU caching, inverted index filtering, and partial sort for top-k. What category theory provided was clean code organization and a unified interface that works for both graph search and document retrieval. This is more valuable than it seems, because when you get into the numbers game, you tend to look at pure performance at all costs without due attention to the underlying architecture. And as I said earlier, the architecture is one area I have been paying a lot of attention to this year.

The latest benchmarks show Vajra achieving ~1.2-1.3x faster latency than BM25S on BEIR and Wikipedia datasets, with sub-2ms query times even at 500K documents. Vajra includes built-in LRU caching for production workloads - cold queries take 0.14-1.9ms (depending on corpus size), while repeated queries with caching enabled return in microseconds. This combination of solid cold performance and optional caching makes it practical for real-world applications.

The project reached v0.3.0 on PyPI this year, with an interactive CLI (vajra-search) that lets you point at any JSONL corpus and have a search engine in your terminal. There's an extensive blog post documenting the benchmarks and architecture if you want the full technical deep-dive.

GitHub: github.com/aiexplorations/vajra_bm25 | PyPI: pypi.org/project/vajra-bm25

Tlön (Tlon) Mathematics

Inspired by Borges' "Tlön, Uqbar, Orbis Tertius" - a story about a world where language has no nouns, only verbs - I built a mathematical framework where processes are primitive and objects emerge as stable patterns. The name comes from Sanskrit "Tlön" (though Borges invented the word), and the project began as an experiment: could we formalize the idea that a rock isn't a thing but a pattern of atomic processes that happens to be stable?

The framework is built on 20 axioms across 6 groups, with the key definition being stability: $\text{Stable}(\pi) \Leftrightarrow \pi \circ \pi \approx \pi$. A stable process is one where doing it twice is equivalent to doing it once - these are the emergent "objects" of Tlön.

One of the most striking results is the Doctrine of No Inverses: for any process with positive duration, there exists no inverse that annihilates it to nothing. Happenings cannot un-happen. The correct concept is reversal (completing a stable cycle), not inverse (annihilation).

The simulations bring these ideas to life:
- Double pendulum: Demonstrates transient processes - chaotic trajectories that never return to themselves
- N-body problem: Shows that stability is special, not generic - N=2 is stable, but adding just one body (N=3) destroys stability
- Lotka-Volterra predator-prey: Demonstrates resonance - two individually unstable processes (prey and predator) that together form a stable oscillating pattern

Additionally, Tlon mathematics has been used to build machine learning algorithms. I've implemented several common machine learning models (traditional ML algorithms, as well as deep learning models such as Dense Networks, ConvNets and Transformers). These implementations have been built on top of Tlon abstractions, and while they use the existing linear algebra, optimization and other applications, they help frame model training in Tlon terms - stability, emergence and other vital concepts from Tlon mathematics are being discussed in the context of ML.

There's an extensive blog post documenting the axioms, theorems, code structure, and simulations if you want the full deep-dive. I'm seeking mathematician review to check the proofs and identify gaps.

GitHub: github.com/aiexplorations/tlon_math

The (Incomplete) Book

I began writing a book on ML/AI design patterns; 128 patterns across 14 chapters, roughly 450 pages. It covered foundation model integration, agentic AI, MLOps, safety, governance. Active delivery through February 2026. The discipline of writing for publication, of having to explain ideas clearly enough for strangers to understand, has been as valuable as the technical work itself.

Unfortunately, I wasn't able to do justice to the material of the book, given the schedules committed, and the book is not yet complete. I plan to revisit the book in the coming year or two. I wish to thank my wife Meera for her support and encouragement of me in "writer" mode. Big thanks to CM for being so encouraging and helping me out. I also want to thank BPB publications for giving me the opportunity - too bad it didn't work, and perhaps we'll cross paths again on this or other book projects!

As they say, never waste a good crisis - and I say this in the context of the book here, because the decision to terminate this project didn't come easy, since I had written three chapters and over a hundred pages in all, with code, examples and diagrams. The crisis here taught me a lot about the process of writing technical books. There are many positives for me. This book project:

• Tested my knowledge and understanding - I found myself doing a lot of research, and reading lots of papers, and buying books I knew I wanted but hadn't invested in.
• Tested my ability to communicate and write lucidly - as someone with a wordy and explanatory style, the constraints and needs of book writing made me develop new writing styles and skills
• Put me in front of a publisher! This was an interesting experience for me, as a first time author.
• Helped me build a discipline for consuming and writing that I knew I had the potential for
• Helped me understand what a good workflow for writing a technical book is. This ranged from research workflow, making jots, code, experimentation, reading research papers, validation from experts, and putting together a manuscript. This is a vital skill!

I have other ideas for writing books, for sure. In some sense, I have been authoring booklets and technical reports (although these are not quite the same thing as a book) for the different projects I'm working on.

Deep Lyapunov - Deep Learning Model Training as a Dynamical Systems Process

Recently, I have begun digging a lot into mechanistic interpretability. This is the sub-field of AI research that seeks to explain how models work. I had done some work on this, when I built a system to understand the chaotic dynamics of agent to agent conversations, earlier this year. While I have understood Lyapunov exponents in the context of dynamical systems and used these methods in dynamical systems analysis in the past, I had not applied this to the conversation that could happen between humans and AI, or one AI agent with another.

After many experiments, I found myself writing a pipeline to study how neural network weights evolve during training, using perturbation analysis and Lyapunov exponents to understand trajectory stability. The question driving this work: do small changes in initialization lead to similar or different final solutions? Understanding this helps with reproducibility, ensemble diversity, and architecture selection.

You'll below the experimental repo in which deep learning dynamics have been analyzed, and also Deep Lyapunov, a library I built and released to PyPI yesterday.

GitHub: github.com/aiexplorations/deep_learning_dynamics, github.com/aiexplorations/deep-lyapunov | Blog Post: Deep Lyapunov - Deep Learning Dynamics

PyPI: pypi.org/project/deep-lyapunov

Related Project: github.com/aiexplorations/agentic_nld

ToDACoMM: Topological Signatures of Transformer Representations

If Deep Learning Dynamics asks "how do weights evolve during training?", ToDACoMM (Topological Data Analysis Comparison of Multiple Models) asks the complementary question: "what is the shape of the representation space that training carves out?"

The project uses persistent homology to characterize transformer activations. The pipeline extracts hidden states from each layer, projects to 50 principal components, and computes Vietoris-Rips persistent homology via Ripser. The key metrics are H0 (connected components, roughly "how spread out are the clusters?") and H1 (loops, roughly "are there circular patterns in the geometry?").

The central finding is categorical: encoder and decoder architectures occupy fundamentally different topological regimes. BERT (bidirectional attention) shows an expansion ratio of 2x from embedding to final layer. GPT-2 (causal attention) shows 95x. Other decoders range from 55x (DistilGPT-2) to 694x (SmolLM2-360M). This isn't gradual variation - it's a stark divide.

The explanation follows from how attention works. BERT's bidirectional attention gives each token access to full context from layer one - representations don't need to expand because all information is already accessible. GPT-2's causal attention means each layer must encode more context than the last as the model accumulates the prefix. The 2x vs 55-694x expansion is the topological signature of this architectural difference.

Every model showed non-trivial H1 at 500 samples - there are loops in representation geometry. Whether these reflect syntactic patterns, semantic cycles, or learned positional structure remains an open question. SmolLM2-360M showed H1 total persistence of 129.52, more than 3x higher than any other model - an anomaly worth investigating.

These two projects - Deep Learning Dynamics and ToDACoMM - are complementary lenses on the same phenomenon. One measures how the optimization carves the weight space; the other measures what gets carved in activation space. Together they suggest a research direction: do divergent weight trajectories (high Lyapunov exponents) correlate with distinct topological signatures in activations?

While there is no PyPI package yet for ToDACoMM, I intend to put one together in the coming year.

GitHub: github.com/aiexplorations/todacomm | Blog Post: The Shape of Learning

What Didn't Work

The projects that failed taught me more than the ones that succeeded. They also provided me a lot of scaffolding on top of which to build further ideas and with more confidence

The TDA-DNN project, where I explore H0 and H1 homologies in the weights of deep neural networks was a really interesting and engrossing project. This allowed me to think about how the weights of a network are likely to be distributed and whether the present of some topological primitives in would help build a case for mechanistic interpretability of the same. The persistence of homologies led to a hypothesis that I explored unfruitfully. The project did provide interesting opportunities for improvement, and a scaffolding to build ToDACoMM on top of, and so this was ultimately helpful and instructive.

Building with Praval was another exciting learning exercise. Elsewhere on this blog, I have described and characterized some of the issues I encountered there, but in a nutshell, building your own framework and an application around it teaches you a lot about how you set up the foundations and how to ensure you don't mess those up. Adding over a thousand tests, fixing async race conditions and many framework level bugs were important milestones in my own growth technically. They were great instructors in how to effectively manage technical debt. Praval Code, my failed experiment of using the Praval Multi-Agent framework may be resurrected in future with more confidence thanks to the lessons learned from my earlier experiences this year.

What I Learned

On Research Direction

Rich Sutton's Bitter Lesson has been on my mind all year: general methods that leverage computation tend to beat clever, hand-designed approaches. This doesn't mean cleverness is worthless, but it suggests where to place bets. I've been trying to internalize this, to resist the temptation of elegant solutions that don't scale. Especially with ToDACoMM and Tlon mathematics, the bitter lesson was an evident and direct dictum worth remembering.

I've also learned that dynamical systems knowledge remains underutilized in practical AI. Koopman operators, Lyapunov analysis, phase space methods; these tools have been developed over decades in physics and applied mathematics, but their application to neural networks and AI systems is still sparse. There may be opportunities there.

I think there is a case to be made for a mathematics first approach to building deep learning models, in which text based next token prediction works alongside numerical quantity prediction. As Petar Velichkovich says in a recent interview, we literally have models that perform hundreds or thousands of multiplications internally but cannot multiply numbers put into the input as part of the context window. This paradox has to be solved if deep learning in its current form has to evolve to become more useful. We seem to have cracked at least some of the code of how code generation agents can be built, but we are yet to crack the core of how mathematics can be done with deep learning models natively.

On Containers

The environment you work in matters as much as the work itself. I've started calling this "choosing the right container." A misaligned container, wrong role, wrong company, wrong domain, dampens even the best ideas. Looking back at my career, the periods of highest productivity coincided with containers that gave me latitude to explore. And the prestige of the position automatically came - in an interesting way, this seems to mirror a belief in Hindu spirituality, where Goddess Lakshmi, the Goddess of Wealth, arrives once Goddess Saraswati, the Goddess of Learning, has already planted herself. I think that working with curiosity is a superpower, and working diligently and with attention to detail without expectation of a reward is a reward.

On Curiosity and Accomplishment

This year forced me to confront a pattern in my projects: I start with genuine intellectual curiosity, go deep, then lose interest when comparing to genuinely ground-breaking research that has perhaps taken entire teams and years of expertise to unlock. Such a comparison itself is flawed, for clear and evident reasons. When it comes to these personal projects, I'm just a guy with a powerful AI coding assistant, and some spare time in which to explore big ideas, which makes this all very worth doing, when you frame it that way!

Approaching ideas with a sense of genuine curiosity, and taking the time to read up about things, structure my thoughts, and be factual and consistent before I build (even if we have the tools for getting past some of) will come back to be important once again. Outsourcing one's thinking, as many techies have become comfortable doing in 2025, is something I have begun to grow out of for the important projects. Sometimes this learning and exploration are indistinguishable - with proofs of Tlon primitives I attempted a month ago, for example, the learning is in the doing.

Another thing about curiosity. There is now a responsibility on the part of those of us using powerful AI tools, to build impactful, new, differentiated things. This is a responsibility because if we use tools that are super powerful in specific ways to do basic things or things not meant for the tool, we're likely to be eventually disappointed with the results.

Tiny Projects are Powerful

Claude Code has enabled rapid prototyping of ideas for millions of developers worldwide, and I'm just one more individual that's benefited from this glut of reasoning and agentic coding power.

During the last year, I experimented with many small projects that don't get a mention here. The pattern I saw here with such rapid prototyping is roughly as follows:

• Inspiration: I get inspired by a conversation, a podcast, book or another project
• Exploration: Use Claude to quickly spin up a working project, having explored some fundamental ideas related to the subject in question.
• Flare and Focus: Discard the ones that don't show promise, and expand on those which do.
• Publish and post: Generally, the flaring portion of promising projects leads to some kind of published work. It could be a library or a package, or a post or a paper.

Overall, tiny projects that are centred on only inspiration and exploration are supremely powerful. They help understand the deeper motivations that you have in a project, more than anything.

Looking Forward

The technical threads running through 20+ years of work remain the following sometimes intersecting things - engineering, mathematical structure, statistical and computational problem solving, dynamics, optimization, complex systems, and perhaps more. What changes often is the frontier where that intersection is explored.

For 2026, my priorities on the personal projects front include continuing development on ToDACoMM (adding training dynamics tracking and larger model analysis), preparing Tlön mathematics for arXiv, and deploying Praval Deep Research as my default research platform. Strategically, I'm exploring what it would mean to set up an independent research structure, finding collaborators who share the interest in mathematical foundations.

At work, there are numerous priorities that span technical, organizational and stratetic elements that I look forward to - the team and the next year of innovation there beckons.

And another piece of important work is perhaps internal. Can I maintain the posture of genuine curiosity, of learning for its own sake, while still exploring ideas and producing projects that matter? Can I let some explorations lead nowhere without judging them failures?

The projects that worked best this year weren't the ones that were only chasing impact. They were the ones that let curiosity lead, with honesty about what the evidence showed, even when it showed that I was wrong. This applied to Praval, Vajra Search, ToDACoMM, Deep Lyapunov especially.

Sometimes the learning is the point, and the results will follow.

If you want to explore any of these projects, find them at github.com/aiexplorations. I'd welcome your thoughts; reach out via the contact page or find me on LinkedIn.

Every neural network training run begins with a random seed. A different seed means different initial weights, which means a different trajectory through the loss landscape, which may or may not arrive at the same destination.

I've been thinking about this for a while, in the context of understanding the relationship between dynamical systems and the internal workings of neural networks. The question: do small changes in initialization lead to similar or different final solutions? In other words, do deep learning models that are essentially learning from feedback loops and errors, subject to sensitive dependence upon the initial conditions of initialization of weights and biases?

Methodology

The methodology I developed to answer this question treats training as a dynamical system and asks: how sensitive is this system to its initial conditions? The approach borrows from Lyapunov's work on dynamical stability, a framework developed in the late 19th century to understand whether planetary orbits would remain stable or fly apart. This same analysis has been replicated and used numerous times in the late 20th century as numerous dynamical and chaotic systems came to be studied. I am certain that this stability analysis has also been performed on deep neural networks especially since there are so many labs interested in performing training with as low a resource footprint as possible, which in turn bring stability of training into the foreground.

The procedure I used here is straightforward. Initialize a model with a fixed seed. Create several copies with small Gaussian perturbations, typically 1% of each parameter's standard deviation, enough to nudge the weights without fundamentally changing them. Train each perturbed model independently on the same data. Record the weight vectors after each epoch. Project the high-dimensional trajectories to principal components. Measure how the trajectories spread or converge.

@dataclass
class LayerSnapshot:
    """Snapshot of a single layer's state at a point in time."""

    # Weight statistics
    weight_mean: float
    weight_std: float
    weight_min: float
    weight_max: float
    weight_norm: float

    # Gradient statistics (if available)
    grad_mean: Optional[float] = None
    grad_std: Optional[float] = None
    grad_norm: Optional[float] = None

    # Eigenvalue statistics for weight matrix
    weight_singular_values: Optional[np.ndarray] = None

What emerges is a picture of the optimization landscape, not as a static surface but as a flow field where trajectories either merge toward attractors or diverge toward distinct basins.

Stability Metrics

Four quantities capture most of what matters about training dynamics:

Convergence Ratio measures whether trajectories come together or spread apart. It's the ratio of final spread to initial spread in the projected weight space. Below 1.0, the models ended up closer than they started; they converged. Above 1.0, they diverged.

Lyapunov Exponent (λ) quantifies the exponential rate of this convergence or divergence:

$$\lambda = \frac{1}{T} \log\left(\frac{\text{final spread}}{\text{initial spread}}\right)$$

Positive λ indicates chaotic dynamics where small perturbations grow exponentially, the signature of a system sensitive to initial conditions. Negative λ indicates stable dynamics where perturbations decay. The exponent tells you not just whether trajectories diverge, but how fast.

Early/Late Ratio reveals when divergence happens. A ratio above 1.0 suggests early divergence, where models quickly find different paths. Below 1.0 suggests late divergence, where they track together initially but separate as training progresses. This distinction matters for understanding whether instability is fundamental or emergent.

PCA Variance Explained indicates how structured the dynamics are. Higher values suggest trajectories follow a low-dimensional manifold; lower values suggest high-dimensional, less predictable evolution.

Results

I analyzed 21 neural network architectures across three families: 10 dense networks trained on the circles dataset, 5 CNNs on MNIST, and 6 transformers on SST-2 sentiment classification. Each architecture was trained with multiple perturbed initializations, and the trajectories were compared.

The master summary reveals stark differences: Dense networks cluster around convergence (ratio ≈ 1), while CNNs and transformers spread toward divergence.

The results were categorical:

Dense networks are the most stable. Six of ten configurations converged, with an average spread ratio barely above unity. If you train a dense network twice with slightly different initializations, you'll often get very similar final weights. The Lyapunov exponent is negative; perturbations are dampened.

CNNs are substantially less stable. Only one of five configurations converged. The average spread ratio of 2.01× means perturbed models typically end up twice as far apart as they started. The convolutional structure, with its weight sharing and spatial hierarchies, admits multiple distinct solutions.

Transformers never converged. Zero out of six configurations. The attention mechanism creates a landscape where small initial differences cascade into large final differences. The Lyapunov exponent is the highest of all families.

Dense Networks

Within the dense family, larger networks tended toward convergence. The 3-layer XLarge configuration with 2,305 parameters converged with a ratio of 0.44×, the most stable architecture in the entire study. The 5-layer Tiny configuration with 51 parameters diverged at 2.71×, the least stable.

Depth had a nuanced effect. Shallower networks (3-layer) showed slightly more stable training than deeper ones (5-layer), with average convergence ratios of 1.01× versus 1.17×. This might seem counterintuitive; one might expect deeper networks to have more regularization through gradient flow constraints. But depth also means longer paths for perturbations to propagate.

class DynamicalSystemsAnalyzer:
    """
    Analyzes neural network training through the lens of dynamical systems.

    Treats the weight space as a phase space and training as a trajectory
    through this space, applying tools from nonlinear dynamics.
    """

    def __init__(
        self,
        tracker: DynamicsTracker,
        model: nn.Module,
        n_pca_components: int = 10,
    ):
        self.tracker = tracker
        self.model = model
        self.n_components = n_pca_components

The practical interpretation: for reproducible results on simple classification tasks, dense networks offer consistent behavior across seeds. For ensemble learning where you want diverse models, you might intentionally choose less stable configurations.

Transformers

Transformers presented a different picture entirely. No transformer configuration converged. BERT variants (bidirectional attention) showed average divergence of 1.50×. GPT variants (causal attention) showed 1.34×, slightly better but still divergent.

Box plots reveal that transformers, despite universal divergence, show tighter distributions than CNNs. The chaos is consistent.

The difference between BERT and GPT is interesting. One might expect bidirectional attention to be more stable, since each position has access to full context during training. But the data suggests otherwise. Causal masking in GPT apparently constrains the optimization landscape in ways that marginally improve stability, perhaps by reducing the degrees of freedom in gradient flow.

Larger transformers showed better convergence properties. GPT-medium with 12 million parameters achieved a ratio of 1.07×, while GPT-tiny with 780,000 parameters reached 1.38×. Overparameterization may provide smoother loss surfaces.

Trajectory Spread Over Training

Trajectory spread evolves differently across architectures:

Spread evolution across families. Dense networks show falling curves (convergence); CNNs and transformers show rising curves (divergence).

Falling curves indicate convergent training, where perturbations are "forgotten" as training progresses. Rising curves indicate divergent training, where small initial differences amplify over time. The steepness of early changes suggests the initial learning phase is most sensitive.

Dense networks show predominantly falling curves. The optimization is pulling trajectories together. CNNs and transformers show predominantly rising curves. The optimization is pushing trajectories apart.

This has implications for checkpointing. If you're training a transformer and want to reduce initialization sensitivity, averaging checkpoints across different runs might help. The trajectories diverge, but early checkpoints before the divergence has compounded could be closer.

A Dynamical Systems Perspective

Cross-family dynamics comparison showing the full picture of convergence ratios, Lyapunov exponents, and accuracy distributions.

Applying dynamical systems theory to neural networks connects to older questions. Poincaré developed these tools to understand celestial mechanics, to predict whether the solar system would remain stable or eventually fly apart. The question was whether small perturbations, a slight change in Jupiter's position, would grow or decay over time.

Neural network training is a different system, but the same question applies. Small perturbations with a slight change in initial weights, grow or decay over epochs. The Lyapunov exponent here measures the sensitivity, just as it does for planetary orbits.

There is a distinc difference in the dynamics of neural networks and planetary orbits, however, as planetary orbits have continuous dynamics and training neural networks has discrete dynamics. Further, planetary systems are conservative; training has friction (weight decay) and injection (data). But the core insight transfers from one space to another, that some systems are stable, some are chaotic, and the architecture determines which ones pan out which way.

Practical Implications

For reproducibility: If you need consistent results across training runs, prefer dense architectures or accept that transformer training is inherently variable. Two transformer runs with different seeds will find different solutions. This isn't a failure of the training procedure; it's a property of the architecture.

For ensembles: The divergent architectures are your friends for ensemble diversity. Transformers naturally explore different solutions without needing special tricks. Dense networks may require more aggressive perturbation to achieve ensemble diversity.

For debugging: If a transformer training run fails, try a different seed before concluding the architecture is broken. The same configuration that fails with one initialization may succeed with another.

For architecture selection: Stability is a design consideration alongside accuracy and efficiency. For safety-critical applications where reproducibility matters, the extra stability of dense networks may outweigh their lower expressiveness.

What This Doesn't Tell Us

This analysis uses small-scale training, typically 10-50 epochs on reduced datasets. Production-scale training with billions of tokens might show different dynamics. The perturbation magnitude of 1% is somewhat arbitrary; different magnitudes might reveal different structure.

The metrics capture global properties of the trajectory ensemble but miss local structure. Two configurations with similar convergence ratios might have very different trajectory geometries.

And crucially: convergence or divergence says nothing about generalization and actual model performance. It is fair to say that the lyapunov exponents are not a good metric for generalization or model performance. A model that converges to the same weights every time might still overfit. A model that finds different solutions might find ones that generalize better. The dynamics describe the optimization, not the learned representation.

Connection to ToDACoMM

There's a natural connection to ToDACoMM, the project I built to measure the topology of trained representations using persistent homology. Deep Learning Dynamics measures how weights evolve during training; ToDACoMM measures what gets carved in activation space after training.

The preliminary observation is suggestive: transformers show both the most chaotic training dynamics (highest Lyapunov exponents, 0% convergence) and the most dramatic topological expansion (55-694× expansion ratios in activation space). Dense networks show both stable training and modest topological transformation.

Is this correlation causal? Do architectures with divergent weight trajectories necessarily produce topologically distinct representations? This remains an open question, one I'm continuing to investigate.

The Library: deep-lyapunov

The methodology described in this post has been packaged as deep-lyapunov, a Python library for analyzing neural network training stability. It's available on PyPI:

pip install deep-lyapunov

The library provides a clean API for stability analysis on any PyTorch model:

import torch.nn as nn
from deep_lyapunov import StabilityAnalyzer

# Your model
model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

# Analyze stability
analyzer = StabilityAnalyzer(
    model=model,
    perturbation_scale=0.01,  # 1% perturbation
    n_trajectories=5,          # Compare 5 perturbed copies
)

results = analyzer.analyze(
    train_fn=your_training_function,
    train_loader=train_loader,
    n_epochs=10,
)

# Results
print(f"Convergence Ratio: {results.convergence_ratio:.2f}x")
print(f"Lyapunov Exponent: {results.lyapunov:.4f}")
print(f"Behavior: {results.behavior}")  # 'convergent' or 'divergent'

# Generate HTML report with embedded visualizations
results.save_report("stability_analysis/")

The library handles all the complexity: creating perturbed model copies, tracking weight trajectories during training, projecting to PCA space, computing Lyapunov exponents, and generating publication-ready reports.

For custom training loops, there's a manual recording mode:

analyzer = StabilityAnalyzer(model)
analyzer.start_recording()

for epoch in range(10):
    train_one_epoch(model, train_loader)
    analyzer.record_checkpoint()

results = analyzer.compute_metrics()

Logging is built in for visibility into what's happening:

import logging
logging.basicConfig(level=logging.INFO)
# Shows: Starting stability analysis, Training trajectory 1/5, Analysis complete...

The library is available at github.com/aiexplorations/deep-lyapunov and pypi.org/project/deep-lyapunov.

The Research Code

The full experimental pipeline used for the analysis in this post is also open source:

# Clone and install
git clone https://github.com/aiexplorations/deep_learning_dynamics
cd deep_learning_dynamics
pip install -e .[dev]

# Run full analysis
python -m experiments.unified_pipeline --output outputs/analysis

# Quick test
python -m experiments.unified_pipeline --quick

# Specific families only
python -m experiments.unified_pipeline --transformer-only

The output includes comprehensive reports, visualizations, and machine-readable metrics for all architectures analyzed.

Looking Forward

Several directions seem worth pursuing:

Scale: How do dynamics change with model size? Do larger transformers become more or less chaotic? The preliminary evidence suggests larger models are more stable, but this needs verification at scale.

Training recipes: Do learning rate schedules, warmup, or optimizer choices affect convergence? Can we engineer stability into otherwise divergent architectures?

Checkpointing: Can we identify stable checkpoints where perturbation sensitivity is lowest? This could inform checkpoint selection for production deployment.

Cross-architecture: Train the same task with dense, CNN, and transformer architectures. Compare not just final accuracy but trajectory stability. Understand the tradeoff.

The dice we roll when we pick a random seed matter more for some architectures than others. Understanding when and why is part of understanding what neural networks actually do.

The deep-lyapunov library is available on PyPI (pip install deep-lyapunov) and GitHub. The research code and experimental pipeline are available at deep_learning_dynamics.

The interesting thing when I went down this category theory rabbit hole, was that the category theory framing produced cleaner code and neat abstractions, and introduced me to a cool way of thinking about search. The resulting search engine also became remarkably fast over time. Initially, I was captivated by the elegance of using the category theory framing, but as time progressed, I realized that this could be built into a competent search engine as well, and leveraged some of the best scientific Python packages, vectorization, SIMD, and other optimizations to deliver three minor releases of Vajra search. It has been a very satisfying journey! The project started off as an innocent way to think of category theory abstractions to represent graph search. As I was working with text based search at work (Elastic) a lot, I realized that this could also be attempted. And here we are!

The latest benchmarks show Vajra achieving ~1.2-1.3x faster latency than BM25S (one of the fastest Python BM25 libraries), with sub-2ms query times even at 500K documents. This post documents what I learned along the way, including comprehensive benchmarks across multiple BM25 implementations and datasets. But first, let me share how search engines work, because understanding the fundamentals made all the difference in building something fast.

The Anatomy of a Search Engine

At its core, every text search engine solves the same problem: given a query, find the most relevant documents from a corpus of documents. The solution has three fundamental components that I came to appreciate deeply while building Vajra.

1. The Index

The index is the data structure that actually makes search fast. Without an index, you'd have to scan every document for every query, which becomes impossibly slow for large corpora, and in computational complexity terms, is $O(nm)$ where n is the number of documents and m is the number of queries. Indexing solves this by reducing the complexity to $O(n+m)$ by using a data structure that allows for efficient retrieval of documents that contain a given term. There are two ways to think about indexing:

Forward Index (document → terms):

doc_1 → ["machine", "learning", "algorithms"]
doc_2 → ["deep", "learning", "neural", "networks"]
doc_3 → ["machine", "learning", "neural", "networks"]

Inverted Index (term → documents):

"machine"   → [doc_1, doc_3]
"learning"  → [doc_1, doc_2, doc_3]
"neural"    → [doc_2, doc_3]
"networks"  → [doc_2, doc_3]
"algorithms"→ [doc_1]
"deep"      → [doc_2]

The inverted index is the key tool of efficiency in most search engines. Instead of asking "what terms are in this document?", we ask "which documents contain this term?". For a query like "neural networks", we instantly get the candidate set {doc_2, doc_3} without scanning all documents. This inversion of perspective, I later realized, has a natural categorical interpretation.

2. The Scorer

Now that we have candidate documents, we need to rank them by relevance. BM25 (Best Match 25) is the dominant scoring function, developed in the 1990s and still used by Elasticsearch, Solr, and most modern search engines.

BM25 scores each document based on three factors: how often the query term appears in the document (term frequency), how rare the term is across all documents (inverse document frequency), and a normalization factor for document length. The formula balances these with tunable parameters k1 and b.

What struck me while implementing BM25 in Vajra is that scoring is fundamentally a morphism, a mathematical arrow from the product of query and document to a real number. This isn't just notation; it clarifies what the scorer does and how it composes with other operations. In category theory, a morphism is a function that preserves the structure of the category. In this case, the structure is the product of query and document, and the real number is the score.

This allows us to organize the code of our search engine in terms of the category theory abstractions we see here. And that gives us the potential to reuse these abstractions to build a different kind of mutable search pipeline. The scoring, then, is one of the steps of this pipeline.

3. The Ranker

With scores computed, the final step is returning the top-k results. This seems trivial, but at scale it matters: sorting a million scores is $O(n log n)$, while finding just the top 10 is O(n) with a partial sort. Small optimizations here compound quickly.

The Search Engine Landscape

In the process of building Vajra Search, as especially as I gravitated from graph search like BFS and DFS, which are common algorithms, I had the opportunity to explore some giants of the search engine landscape.

Lucene: The Industry Foundation

Apache Lucene (2000) is arguably the foundation of modern enterprise search. Written in Java, it powers Elasticsearch, Solr, and countless enterprise systems. Lucene's architecture reflects 25 years of production optimization:

Lucene is designed for production durability: indexes persist to disk, support concurrent updates, and are able to scale horizontally via sharding. The trade-off is complexity and JVM overhead. Pyserini wraps Lucene via the Anserini toolkit, providing a Python API that can help us build with these capabilities in native Python applications (but with JVM dependencies nevertheless).

Tantivy: Lucene in Rust

Tantivy (2016) reimplements Lucene's architecture in Rust, gaining memory safety and roughly 2x performance over Java. It maintains the segment-based, disk-first design:

Tantivy excels for applications needing persistent, updatable indexes with Rust's performance. However, the disk-based design adds I/O latency for pure query workloads. At some point, disk I/O is something Vajra is going to deal with. For the time being, I have preferred building indexes in memory. There is some caching possible within Vajra and in this sense, it is inspired by what Tantivy is able to do. Tantivy is super fast and performant given that it does this with disk I/O.

Rank-bm25: The Python Baseline

Rank-bm25 (2018) is the simplest BM25 implementation, a single Python file that's easy to understand:

class BM25Okapi:
    def __init__(self, corpus):
        self.corpus_size = len(corpus)
        self.doc_freqs = []  # Forward index: doc → term frequencies
        self.idf = {}        # Precomputed IDF values

    def get_scores(self, query):
        scores = np.zeros(self.corpus_size)
        for term in query:
            # Score ALL documents for this term
            for doc_idx in range(self.corpus_size):
                scores[doc_idx] += self._score_term(term, doc_idx)
        return scores

Rank-BM25 was my starting point to understanding the typical Python BM25 implementation meant for small tasks, and which is also not meant to be memory efficient, or fast. Indexing even smallish corpuses takes minutes, and it works in-memory. This is a tool for a limited job. The problem is clear: rank-bm25 uses a forward index and scores every document for every query. With a million documents and five query terms, that's five million score computations per query. While it not a great use of memory or compute, Rank-BM25 does get the job done and is simple for some of the smaller projects you may be building. It does not have complex abstractions like Vajra or ace engineering like BM25S, Tantivy or Lucene.

BM25S: Leveraging Eager Scoring

BM25S (2024) introduced a paradigm shift that I found elegant: eager scoring. Instead of computing scores at query time, pre-compute everything at index time:

The sparse score matrix stores pre-computed BM25 scores for every term-document pair:

So, what happens when you supply a query? Query time operations shrink to just sparse matrix slicing and addition. There are no IDF lookups, no BM25 formula evaluation, and none of the heavy lifting. This achieves 100-500x speedup over rank-bm25. The trade-off is memory footprint and no query-time flexibility. So, BM25S is a great tool for the job, and is probably a good starting point for building a search engine. I enjoyed benchmarking against it as I built Vajra search and the BM25S paper made for good reading and provided ideas and hints as to eager scoring, a technique which Vajra borrows from BM25S, in its own way.

Building Vajra: A Different Path

When I started Vajra, I wanted to combine ideas from different paradigms while using category theory to think about and organize the code. Category theory provides a scaffolding for structuring mathematical abstractions (morphisms, functors, coalgebras) that map naturally onto search operations. My approach was to use this framing to reframe how we think about search, then combine it with aggressive engineering optimizations.

We have discussed above how we need to compute inverted indexes, inverse document frequency, and perform BM25 computation and top-k sorting. These things don't change from implementation to implementation, and Vajra is no different in implementing these also in its pipeline, since after all, it too is a BM25 search engine. However the way these are implemented allows a kind of lazy computation and caching that is not implemented in other frameworks. Let's see more below.

Let's take a look at how Vajra's BM25 flow actually works, through this flow diagram. We will review different individual aspects of Vajra in this context below as well.

The insight that made Vajra fast came from thinking about what operations actually matter for speed and performance.

1. Inverted index filtering: For a query with three terms, only about 1% of documents contain all terms. Why score 500,000 documents when you can score 5,000?

2. LRU query caching: Real workloads have repeated queries. After the first execution, results are cached; subsequent calls return in nanoseconds.

3. Vectorized scoring: NumPy's SIMD operations score thousands of candidates in a single vectorized operation.

4. Lazy computation: Unlike BM25S, Vajra computes scores on-demand. This uses less memory and allows query-time flexibility.

The category theory framing (morphisms for scoring, coalgebras for search unfolding) provides clean code organization and composability. Combined with these engineering optimizations, Vajra delivers both architectural elegance and raw speed.

The Category Theory Foundation

Category theory provides the architectural scaffolding for Vajra. It offers composable abstractions (morphisms, functors, coalgebras) that map directly onto search operations, making the code modular and extensible. The speed comes from combining this clean architecture with engineering optimizations: NumPy vectorization, sparse matrices, and aggressive caching.

Here's how the abstractions organize the code.

How the Abstractions Organize Code

The codebase has a categorical/ module with three base abstractions: Morphism, Functor, and Coalgebra. Each provides an interface that derived classes implement.

Morphisms define composable transformations. The base class enforces an apply() method and provides composition via the >> operator:

class Morphism(ABC, Generic[A, B]):
    @abstractmethod
    def apply(self, a: A) -> B: ...

    def __rshift__(self, other: 'Morphism[B, C]') -> 'Morphism[A, C]':
        return ComposedMorphism(self, other)

This lets you build pipelines like where each step is a morphism with clear input and output types. The BM25 scorer becomes a morphism (Query, Document) → ℝ, which forces you to think about exactly what data flows in and out. Let is dig in a little.

Let's say we want to build a pipeline that preprocesses a query, tokenizes it, and then scores it. We can do this by composing the following morphisms:

preprocess >> tokenize >> score

This is a pipeline that preprocesses a query, tokenizes it, and then scores it. The preprocess morphism is responsible for preprocessing the query, the tokenize morphism is responsible for tokenizing the query, and the score morphism is responsible for scoring the query. This provides in practice a neat way to organize the code and to make it more readable and maintainable.

Coalgebras capture the "unfolding" pattern. In a nutshell, an unfolding is a way to represent a data structure as a tree. The base class requires a structure_map() method that takes a state and produces its successors:

class Coalgebra(ABC, Generic[X, FX]):
    @abstractmethod
    def structure_map(self, state: X) -> FX: ...

From this, I derived SearchCoalgebra, TreeSearchCoalgebra, and ConditionalCoalgebra. Search Coalgebras are a way to represent a search as a tree, and are used to represent the search space. Tree Search Coalgebras are a way to represent a search as a tree, and are used to represent the search space. Conditional Coalgebras are a way to represent a search as a tree, and are used to represent the search space. The BM25 search implementation extends Coalgebra with the signature QueryState → List[SearchResult]:

class BM25SearchCoalgebra(Coalgebra[QueryState, List[SearchResult]]):
    def structure_map(self, state: QueryState) -> List[SearchResult]:
        candidates = self.index.get_candidates(state.query_terms)
        ranked = self.scorer.rank(state.query_terms, candidates)
        return [SearchResult(doc, score, rank) for ...]

The same interface works for graph search (BFS, DFS) and tree exploration but with different derived classes. I found this genuinely useful; once you implement structure_map(), you get unfold(), trajectory(), and other coalgebraic operations for free.

Functors in Vajra are abstractions that allow you to "lift" a transformation—such as a morphism—so it works over entire containers rather than just single values. This is captured in code via an interface: every Functor lets you map any morphism across the relevant structure. For example, the ListFunctor implements a fmap_morphism method that takes a morphism and returns a new morphism, which processes each element of a list independently and collects the results.

class ListFunctor(Functor[A, List[A]]):
    def fmap_morphism(self, f: Morphism[A, B]) -> Morphism[List[A], List[B]]:
        return FunctionMorphism(lambda xs: [f.apply(x) for x in xs])

In the context of search, this means you can write a morphism that operates on an individual query or document, then seamlessly apply it to a whole batch of queries or search candidates using ListFunctor, enabling vectorized or batch-style computation in places where search might branch or aggregate multiple possibilities. This makes the code for candidate generation, ranking, and batch query processing both concise and highly reusable. The same pattern applies to MaybeFunctor, which handles computations that may fail by encapsulating values that might be missing, allowing the rest of the pipeline to remain clean and compositional even in the presence of errors.

The benefits of Vajra's architectural choices

By adopting category-theoretic abstractions like morphisms, functors, and coalgebras, Vajra cleanly disentangles data types from transformations and search strategies:

• Morphisms provide a systematic way to build and compose pipelines with explicit input/output types
• Functors let you automatically “lift” these pipelines across collections or optional values for batch or robust computation
• Coalgebras encode the core search dynamics in a way that's generic across strategies (tree, graph, conditional) so new algorithms can plug into the same interface and instantly inherit useful unfolding and traversal methods

This architectural clarity made it straightforward to add new scorers or search strategies: each abstraction exposes focused extension points, so it’s always clear where to slot in new logic and which minimal methods need to be implemented for composability. The monoid homomorphism insight, that BM25 scores are additive across terms (score(q₁ + q₂) = score(q₁) + score(q₂)) is also interesting and has practical consequences: you can cache term-level scores and sum them at query time, or compute upper bounds on partial scores for early termination. The algebraic structure pointed at optimization opportunities.

Benchmarking Vajra

Building the abstractions isn't where it stops, and to build a useful software application, we have to look at what's out there, benchmark the tool we're building to see where it fits in, and what relative benefits it may offer to users. So, to validate Vajra's performance, I benchmarked against a few other implementations:

To be clear, these are all BM25 implementations. All implementations compute the same essential BM25 formula:

$$\text{BM25}(q, d) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, d) \cdot (k_1 + 1)}{f(q_i, d) + k_1 \cdot \left(1 - b + b \cdot \frac{|d|}{\text{avgdl}}\right)}$$

The differences are in __how_ they compute it, and when.

Benchmark Results

I spent a lot of time benchmarking the initial build of Vajra against Rank-BM25, and then began excluding it from my benchmarks because of how slow the latter was. I moved on to BM25S as the peer framework of choice that I was benchmarking against, and also began testing against Tantivy and Pyserini eventually. While the latter is exceptional in its recall performance, with excellent Recall\@10 and NDCG\@10 (which measure whether the right docs came up, and in the right order) BM25S and Vajra have ultimately be the more comparable frameworks.

See the video below for a Vajra specific search demo. This is using Vajra CLI, which is a command line based search tool which you can point at a JSON, and have a fast search engine right in your terminal!

BEIR/SciFact (5,183 documents, 300 queries)

Vajra is ~1.3x faster than BM25S on this dataset, with slightly better recall.

Wikipedia/200K (200,000 documents, 500 queries)

Wikipedia/500K (500,000 documents, 500 queries)

At 500K documents, Vajra maintains sub-2ms latency and is ~1.3x faster than BM25S. Tantivy has the best accuracy but higher latency.

Wikipedia/1M (1,000,000 documents, 500 queries)

At 1M documents, Vajra is ~1.6x faster than BM25S on single queries (3.40ms vs 5.44ms). Build time has been significantly optimized using per-document array concatenation, bringing it down to 17 minutes. Query latency remains sub-4ms even at million-document scale.

Improving Vajra's Performance

Vajra search started off being about as fast as BM25 and was for a couple of months quite a lot slower than BM25S! Tantivy and Pyserini were not in the picture at that stage of the project. My initial thinking was to see if I could build on top of category theory abstractions and see how we could speed things up, but came to the realization quickly that I would have to resort to engineering best practices: caching, filtering, scoring, and sparse matrices. Let's review each of these improvements below. If you use Vajra, you'll see numba used quite a lot in Vajra and for good reason.

LRU Query Caching

This was the single biggest optimization. Real workloads have repeated queries, and caching complete results means subsequent calls return instantly:

@lru_cache(maxsize=10000)
def search(self, query: str, top_k: int = 10):
    # First call: compute BM25 scores
    # Subsequent calls: instant cache hit

Combine this with user preference based caches and you can make Vajra search very performant and fantastically usable for a use case where repeated queries will come through from the bulk of users but different in every case.

Candidate Filtering with Inverted Index

Instead of scoring all documents like BM25S does with its sparse matrix, Vajra uses the inverted index to identify only the documents that contain query terms:

Query: "machine learning algorithms"

Traditional (Rank-BM25 or similar): Score 500,000 documents = 500,000 computations
Vajra: Filter to ~5,000 candidates, score only those = 5,000 computations

A large part of building a performant system seems to be to not try to do everything at once, but stage it all out, so that limited resources can be used in the best possible way.

Vectorized Scoring

Once candidates are identified, scoring uses NumPy vectorization:

# Score all candidates in one vectorized operation
scores = idf_vector @ tf_matrix * normalization_factors

No point reinventing the wheel that Numpy already provides. Matrix operations are super fast thanks to CPython bindings in Numpy and while it may not be as scalable as pure C or Fortran, it provides a significant speed up.

Sparse Matrix Storage

For corpora over 10K documents, the term-document matrix is 99%+ zeros. SciPy's CSR format avoids storing and computing on those zeros. This is another example of not overdoing stuff in the quest for performance.

Partial Sort for Top-K

Instead of sorting all scores, np.argpartition provides $O(n)$ average complexity:

# O(n) partial sort instead of O(n log n) full sort
top_indices = np.argpartition(scores, -k)[-k:]

Note that none of the above are exotic mathematics even if category theory does inform the architecture of Vajra BM25. Most of it is sane engineering you would recognize as a senior engineer or architect, and would likely build into any other project which you want high performance in.

Understanding the Latency Numbers

The reported latencies are measured with query caching disabled for fair comparison. Here's what to expect in practice:

Vajra includes an optional LRU cache (cache_size parameter, default 1000) that can dramatically speed up repeated queries. For production workloads with query repetition, this can provide sub-millisecond latency.

As of v0.3.1, if you're using the Vajra CLI with single queries ($ vajra-search -q "query"), each invocation creates a new engine with an empty cache. In interactive mode, subsequent queries benefit from caching. The CLI is a super easy way to see Vajra in action. Note that v0.3.1 includes significant index building optimizations (42% faster at 1M scale) using per-document array concatenation.

For library usage, keep the VajraSearchOptimized instance alive across queries to benefit from the LRU cache.

How Do the Engines Compare?

Vajra vs BM25S

Vajra and BM25S use similar underlying techniques (eager scoring with sparse matrices). Vajra is ~1.2-1.6x faster in benchmarks due to optimizations in scoring and top-k selection.

Tantivy's Different Trade-offs

Tantivy is written in Rust and optimized for different use cases:

1. Disk-based index: Supports persistence and updates, but adds I/O latency
2. Python bridge: The tantivy-py bindings add some overhead
3. Different priorities: Tantivy optimizes for durability and concurrent access, not pure query speed

Tantivy has the best accuracy on Wikipedia datasets (+1.6% NDCG over Vajra).

Accuracy Analysis

How does accuracy compare across engines? Vajra outperforms BM25S and Tantivy on BEIR datasets, while Pyserini (Lucene) leads by 1-2%. On Wikipedia, Tantivy leads with Vajra close behind. Here are the metrics.

BEIR dataset searches:

Pyserini leads on BEIR accuracy by 1-2%. Vajra ranks second, ahead of BM25S and Tantivy.

Wikipedia dataset searches:

On Wikipedia, Tantivy leads accuracy while Pyserini drops significantly. Vajra holds second place, 1-2% behind Tantivy.

When to Use Each Engine

Benchmark Methodology

For reproducibility, here's how I ran the benchmarks:

Hardware

Process

The benchmark script follows this sequence for each engine:

1. Load Dataset: BEIR datasets from HuggingFace; Wikipedia from pre-downloaded JSONL
2. Check Index Cache: Compute corpus hash, look for cached index
3. Build Index (if not cached): Each engine has its own indexing logic
4. Save Index to Cache: Pickle indexes for subsequent runs
5. Query Evaluation: Run queries with query caching disabled (cache_size=0) for fair comparison
6. Compute Metrics: Recall@10, NDCG@10, latency per query

Note: Query caching is disabled during benchmarks to ensure fair comparison across engines. In production, enabling Vajra's LRU cache can provide sub-millisecond latency for repeated queries.

Index Caching

I used persistent index caching to avoid expensive rebuilds:

For Vajra on 500K documents, index build takes about 16 minutes. With caching, subsequent runs load in under 5 seconds.

Running the Benchmarks Yourself

Start with a new Python environment and execute the following bash commands:

# Clone the repository to get the benchmarking scripts
git clone https://github.com/aiexplorations/vajra_bm25.git
cd vajra_bm25

# (Optional) Create a virtual environment here if you prefer isolation
# python -m venv .venv
# source .venv/bin/activate

# Install dependencies
pip install vajra-bm25[optimized] rank-bm25 bm25s beir rich tantivy
# doing a pip install is needed to have the whl in your environment

# Optional: Pyserini (requires Java 11+)
pip install pyserini

# Benchmark script is present only in the repo, does not ship with the whl file.
# Run BEIR benchmarks
python benchmarks/benchmark.py --datasets beir-scifact beir-nfcorpus

# Run Wikipedia benchmarks
python benchmarks/benchmark.py --datasets wiki-200k wiki-500k

# Force rebuild (ignore cache)
python benchmarks/benchmark.py --datasets wiki-200k --no-cache

Using Vajra Search

Vajra works well for latency-sensitive Python workloads. If you're using it, I'd like to hear about your use case.

For starters, since I have built Praval earlier, I want to integrate Vajra BM25 into it. I have already built a CLI for Vajra BM25 and want to use it in Praval. One of the cool use cases I see for Vajra is in enabling agents to quickly search code bases, or datasets, or text corpuses. This can become a game changer for large scale multi-agent systems.

Recent Updates (v0.3.2)

The latest release adds two features that extend Vajra's usability:

PDF Document Support: You can now index and search PDF documents directly, without converting them to JSONL first. This makes it easy to build a search engine over research papers, reports, or any PDF collection:

from vajra_bm25 import DocumentCorpus, VajraSearchOptimized

# Load PDFs - single file, directory, or auto-detect
corpus = DocumentCorpus.load_pdf_directory("./papers/", recursive=True)
engine = VajraSearchOptimized(corpus)
results = engine.search("attention mechanism", top_k=10)

The CLI supports this too: vajra-search --corpus ./papers/ -q "neural networks"

Install PDF support with: pip install vajra-bm25[pdf]

Documentation Site: Comprehensive documentation is now available at aiexplorations.github.io/vajra_bm25, covering installation, API reference, BM25 parameter tuning, performance optimization, and the category theory foundations.

Closing Thoughts

Vajra is the fastest Python BM25 implementation available today: ~1.2-1.6x faster than BM25S depending on corpus size, with sub-4ms latency even at 1M documents. LRU caching delivers sub-millisecond responses for repeated queries in production.

The category theory framing shaped how I organized the code. Morphisms for scoring, coalgebras for search unfolding, functors for batch operations: these abstractions separate concerns cleanly and make the codebase extensible. The monoid homomorphism structure of BM25 scoring opened up optimization opportunities I wouldn't have seen otherwise.

I built Vajra because I wanted a fast, hackable BM25 engine for Python that I could integrate into agentic systems like Praval. It's open source, actively maintained, and ready for your projects.

Links and References:

• Vajra BM25 Repo: github.com/aiexplorations/vajra_bm25
• Vajra BM25 Documentation: aiexplorations.github.io/vajra_bm25
• Vajra BM25 on PyPI: pypi.org/project/vajra-bm25
• Vajra BM25 Benchmark docs: docs/vajra_benchmark_comparison.md
• Category Theory for Programmers: Bartosz Milewski
• Pyserini
• BM25S:
• Tantivy
• Rank-BM25
• Praval
• BEIR
• Wikipedia
• JSONL
• LRU Cache
• Partial Sort

This idea was lodged in my mind for years. What if we took it seriously? What if we built a mathematical framework where processes are primitive and objects emerge as stable patterns? The philosophical implications of this seemed enormous, each time I thought about it. Every cell in the human body has an expiration date, but the human body itself maintains its rough shape over time. We experience the process of aging. Turbulence transforms the way molecules of water behave, as they leave the outlet of a tap and drop down; we see laminar flow transition to turbulent flow. Double pendulums behave in very unpredictable ways even if we repeat experiments with them from just mildly different initial conditions, and at the same time, grand phenomena such as the Great Red Spot on Jupiter are stable and predictable for years. The moon's orbit around the Earth seems a stable process but over millennia, there are minor variations in the orbits of all celestial bodies. You get the idea - the list of phenomena that could be characterised as emergent are endless and all around us.

What I have been exploring with Tlon mathematics is the possibility that we could describe many of these emergent phenomena, or even phenomena we consider rule-based, or stable, from the lens of a process-first mathematical paradigm. I'd learned some category theory basics earlier in the year, and wanted to see if I could build an algebra that addressed just such a process-centric approach to systems modelling. Later in the year, with Claude Code as my collaborator, I finally attempted to build the scaffolding, proofs, and the "house" on top of which actual mathematical systems and conjectures could be posited.

The result of all this rumination and exploration with Claude Code is Tlön Mathematics. I will interchangeably use Tlön and Tlon, for the convenience of typing is an important aspect of the process of communication. Tlön (Tlon) mathematics is a mathematical framework built on top of 20 axioms across 6 groups, with theorems, proofs, and concrete implementations in Python that convert the mathematical foundation into a computational foundation that demonstrates the concepts through various systems simulations, including simulations of dynamical systems. As a purely intellectual exercise, this was quite satisfying, because, well, a set of core axioms was converted through sequential rules into a whole mathematical world view, and one inspired by a work of fiction, by one of my favourite authors, at that. Working with the ideas of Tlon mathematics has made me appreciate how we can frame dynamical systems and the emergence of objects in the physical world in a different way.

The Central Inversion

In conventional mathematics, objects exist first and processes act upon them. A rock exists; erosion happens to it. A planet exists; orbital motion happens to it.

Tlön inverts this, in a specific way: processes are fundamental, and objects are stable patterns - processes that maintain themselves through repetition. This isn't to say that Tlon mathematics bypasses the established rules of mathematics, and to the contrary, Tlon mathematics is built on top of a category theoretic structure, like other algebras. Identities, relationships like associativity and the like can be proven in this system of mathematics like in others.

As I described in examples above, objects are our default mode of viewing and understanding the world around us, and Tlon challenges this convention, based on the phenomena that we do see that indicate how this object-property-interaction ontology in our heads cannot explain transient objects or objects like the Mandelbrot set. Through the lens of Tlon, therefore, a rock isn't an object, but the result of a stable process that has produced an aggregation of things that appears to be a rock, and that we can classify as such. The rock's changes over time, be it erosion through wind or rain or other phenomena, and these processes that give the rock its characteristics are also processes. A planet is a specific aggregation of matter that has emerged as a stable process (or a large set of stable processes), resulting in the object we see as the planet. While this may seem like splitting hairs when we write it down in English, it becomes much clearer when we write it down in terms of definitions, assertions we have as axioms, and allow the system of mathematics to guide us in terms of the properties that emerge as a consequence of these foundations. Tlon serves as a mathematical scaffolding for discussing processes whose physics have been established, and as a process algebra, but perhaps does not serve as a physics scaffolding, and I have a hunch that this is something I will discover in time as I work with Tlon mathematics.

How Objects Emerge from Processes

The key insight is formal: an object in Tlon is defined as an equivalence class of stable processes:

$$\text{Object} := [\pi]_{\approx} \text{ where } \text{Stable}(\pi)$$

A process is stable when repeating it is equivalent to doing it once:

$$\text{Stable}(\pi) \Leftrightarrow \pi ; \pi \approx \pi$$

This definition inverts traditional ontology:

Concrete examples:

• Flame: The combustion process $\pi_{\text{flame}}$ is stable; sustained burning is equivalent to momentary burning in terms of the pattern. The "flame" we perceive is the equivalence class $[\pi_{\text{flame}}]$.
• Sorted array: The sorting process satisfies $\text{sort} ; \text{sort} \approx \text{sort}$. A sorted array is the stable fixed point.
• Planetary orbit: One complete orbit $\pi_{\text{orbit}}$ satisfies $\pi_{\text{orbit}} ; \pi_{\text{orbit}} \approx \pi_{\text{orbit}}$. The "orbit" is the stable pattern, not the planet.
• Standing wave: Two traveling waves interfere to produce a pattern that maintains itself through time.

In each case, the "object" is not a thing but a self-sustaining pattern of activity. The question "what objects exist?" becomes an algebraic question: "which processes are idempotent under sequential composition?"

Tlon's processes and stability

Tlon could potentially allow us to model phenomena around us in dynamic slices. For example, a planet need not be modelled as an object that has a certain range of motions owing to fundamental forces such as gravity, but it could be modelled in Tlon terms as an orbital process that is stable, resulting in accretion and the the object we see as the planet. Geophysical phenomena may be similarly modeled starting from the processes we observe.

Stability is defined as follows:

$$\text{Stable}(\pi) \Leftrightarrow \pi \circ \pi \approx \pi$$

A stable process is one where a repetition of the process is equivalent to performing it once. In algebra, such elements are called idempotent. These are the closest thing to "objects" in Tlön - self-maintaining patterns that persist through repetition.

The 20 Axioms

The framework is built on 20 axioms organized into 6 groups:

Complete Axiom List

Group A: Existence
- A1: $\exists\pi$ (Process($\pi$)) — Processes exist
- A2: $\exists!\varepsilon$ $(|\varepsilon| = 0)$ — Unique null process with zero duration

Group B: Sequential Composition
- B1: $\forall\pi,\rho \Rightarrow \text{Process}(\pi ; \rho)$ — Closure
- B2: $(\pi ; \rho) ; \sigma \approx \pi ; (\rho ; \sigma)$ — Associativity
- B3: $\pi ; \varepsilon \approx \pi \approx \varepsilon ; \pi$ — Identity
- B4: $|\pi ; \rho| = |\pi| + |\rho|$ — Duration additivity

Group C: Concurrent Composition
- C1: $\forall\pi,\rho \Rightarrow \text{Process}(\pi \parallel \rho)$ — Closure
- C2: $(\pi \parallel \rho) \parallel \sigma \approx \pi \parallel (\rho \parallel \sigma)$ — Associativity
- C3: $\pi \parallel \rho \approx \rho \parallel \pi$ — Commutativity
- C4: $\pi \parallel \varepsilon \approx \pi$ — Identity
- C5: $|\pi \parallel \rho| = \max(|\pi|, |\rho|)$ — Duration maximum

Group D: Interference
- D1: $\text{Int}(\pi, \rho) \approx \text{Int}(\rho, \pi)$ — Symmetry
- D2: $\text{Int}(\pi, \pi) \approx \pi$ — Self-interference
- D3: $\text{Int}(\pi, \varepsilon) \approx \pi$ — Null non-interference
- D4: $|\pi| = |\rho| \Rightarrow \pi \parallel \rho \approx \pi ; \text{Int}(\pi, \rho) ; \rho$ — Decomposition
- D5: $\pi \approx \pi' \land \rho \approx \rho' \Rightarrow \text{Int}(\pi, \rho) \approx \text{Int}(\pi', \rho')$ — Congruence

Group E: Stability
- E1: $\text{Stable}(\pi) \Leftrightarrow \pi ; \pi \approx \pi$ — Idempotence definition
- E2: $\exists\pi$ $(\text{Stable}(\pi) \land \pi \not\approx \varepsilon)$ — Non-trivial stability exists

Group F: Equivalence
- F1: $\pi \approx \pi$ — Reflexivity
- F2: $\pi \approx \rho \Rightarrow \rho \approx \pi$ — Symmetry
- F3: $\pi \approx \rho \land \rho \approx \sigma \Rightarrow \pi \approx \sigma$ — Transitivity
- F4: $\approx$ respects $;$ and $\parallel$ — Congruence

Three axioms deserve particular attention:

Axiom E1 (Stability Definition):
$$\text{Stable}(\pi) \Leftrightarrow \pi \circ \pi \approx \pi$$

A process is stable if repeating it yields the same thing. The null process $\varepsilon$ (doing nothing) is trivially stable. But other stable processes exist too; these are the emergent "objects", the results of stability.

Axiom B4 (Duration Additivity):
$$|\pi \circ \rho| = |\pi| + |\rho|$$

Sequential composition adds durations. If a process $\pi$ has a duration of 3 units and another $\rho$ has a duration of 5 units, doing them in sequence takes 8 units. This is self-evident and agrees with common algebras we are familiar with.

Axiom C2 (Concurrent Commutativity):
$$\pi \parallel \rho \approx \rho \parallel \pi$$

Unlike sequence (where order matters), concurrence is symmetric. "$\pi$ while $\rho$" is the same as "$\rho$ while $\pi$."

Key Theorems

From these axioms, several important results follow. We validate the following Tlon theorems using a category theory foundation, which becomes necessary as the latter is a scaffolding for many mathematical systems.

Theorem 1.1: Sequential composition forms a monoid - $(\text{Process}, ;, \varepsilon)$ has closure, associativity, and identity. Here ; represents the sequential composition operator, and $\varepsilon$ represents the null process.

Proof: Closure by B1, associativity by B2, identity by B3. ∎

Theorem 1.2: Concurrent composition forms a commutative monoid - $(\text{Process}, \parallel, \varepsilon)$ additionally has commutativity. Here $\parallel$ represents the concurrent composition operator, and $\varepsilon$ represents the null process.

Proof: Closure by C1, associativity by C2, commutativity by C3, identity by C4. ∎

Theorem 4.3 (Power Collapse): If some process $\pi$ is stable, then $\pi^n \approx \pi$ for all $n \geq 1$. Sequential powers collapse for stable processes. Here $\approx$ represents the equivalence operator.

Proof: By induction. Base: $\pi^1 = \pi$. Step: If $\pi^n \approx \pi$, then $\pi^{n+1} = \pi^n ; \pi \approx \pi ; \pi \approx \pi$ (by stability). ∎

The Emergence Theorem: In any finitely-generated process space with non-trivial stable processes, the property "non-trivially stable" is emergent - it doesn't hold for any generator but must hold for some composite process.

Proof: See Tlon Theory Booklet, Theorem E1. The proof uses the level function $L(\pi)$ measuring compositional complexity. ∎

The Emergence Theorems

Tlon's approach to emergence is mathematically precise: it shows how stable patterns (objects, structures, dynamics) arise out of the interactions and compositions of unstable, transient parts. The theorems, drawn from the Tlon Theory Booklet, clarify different faces of emergence:

E2: Resonance Generates Emergent Stability

Theorem (Resonance Emergence):
If $\pi$ and $\rho$ are both unstable, but $\text{Stable}(\pi \parallel \rho)$, then stability is an emergent property of their combination.

Proof: By assumption, $\text{Stable}(\pi) = \text{false}$ and $\text{Stable}(\rho) = \text{false}$, yet $\text{Stable}(\pi \parallel \rho) = \text{true}$. The whole has a property neither part possesses. ∎

• Interpretation: Two chaotic or transient processes, when run concurrently, may form a composite process that is stable, even though neither is stable alone. This is resonance: the "whole is more than the sum of its parts" in a formal algebraic sense.
• Concrete Example:
  In the Lotka-Volterra predator-prey model (Volterra, 1926; Lotka, 1925), the isolated prey or predator populations are unstable. But together, their interaction stabilizes the entire ecosystem into periodic cycles. A stable process emerges from their interplay.

E3: Objecthood is Emergent

Theorem (Emergence of Objects):
Non-trivial objects (i.e., equivalence classes of stable processes other than the null process) cannot exist at the primitive level. They arise only through composition and stabilization.

Proof: Objects are equivalence classes of stable processes. At level 0, only generators exist. Since non-trivial stability requires composition (by E1), objects first appear at level ≥ 1. ∎

• Interpretation:
  In Tlon, objects are not presupposed. There are no building blocks waiting to be labeled as "objects"; instead, objects emerge from the closure and stabilization of processes. The question is not "what objects are there?" but "which processes are stable?" The stable patterns themselves are the objects.

E4: Downward Constraint (Emergent Constraint)

Theorem (Downward Constraint):
If $\pi = \sigma ; \tau$ is stable, then $\sigma ; \tau ; \sigma ; \tau \approx \sigma ; \tau$. The global stability of the composite process constrains the behaviors of its components.

Proof: If $\pi = \sigma ; \tau$ is stable, then $\pi ; \pi \approx \pi$ by E1. Substituting: $(\sigma ; \tau) ; (\sigma ; \tau) \approx \sigma ; \tau$. ∎

• Interpretation:
  Emergent stability at a higher level (the composite process) imposes algebraic constraints on the lower-level sequences that compose it. This expresses a formal version of "downward causation": the stability of the whole feeds back to restrict the ways its parts can behave.

The full formal statements, proofs, and examples are presented in the Tlon Theory Booklet.

The Doctrine of No Inverses

One of the most striking results is that process inverses are ontologically forbidden. For any process $\pi$ with positive duration, there exists no process $\rho$ such that $\pi ; \rho \approx \varepsilon$.

The argument is straightforward:
1. By Duration Additivity: $|\pi ; \rho| = |\pi| + |\rho| \geq |\pi| > 0$
2. But $|\varepsilon| = 0$
3. Therefore $\pi ; \rho$ cannot equal $\varepsilon$

Happenings cannot un-happen. The arrow of time is woven into the fabric of process composition. This is not a bug but a feature - it captures something fundamental about causality and time.

The correct concept is reversal, not inverse. A reversal $\rho$ of $\pi$ completes a stable cycle: $\text{Stable}(\pi ; \rho)$. The pendulum swings left ($\pi$), then right ($\rho$). The composite $\pi ; \rho$ is one complete oscillation - a stable process. Time passed; something happened. But the pattern is closed, self-sustaining.

The Code Structure

The mathematical scaffolding I've built for Tlon is also being represented in code, as a Python code base. I am perhaps undecided on whether this should be written in a functional programming language, which seems to provide first class abstractions to deal with the nuts and bolts of Tlon mathematics. In any case, the Python implementation has been helpful to simulate different kinds of systems and understand them with the scaffolding of Tlon mathematics.

The framework is implemented in Python with type hints and comprehensive tests. The core abstractions live in tlon/core/:

class Process(ABC):
    """Abstract base class for all processes."""

    @property
    @abstractmethod
    def duration(self) -> float:
        """Intrinsic temporal extent of the process."""
        pass

    def is_stable(self, tolerance: float = 1e-9) -> bool:
        """Axiom E1: Stable(π) ⟺ π ; π ≈ π"""
        return (self @ self).equiv(self, tolerance)

    def __matmul__(self, other):
        """Sequential composition: π @ ρ (represents π ; ρ)"""
        ...

    def __or__(self, other):
        """Concurrent composition: π | ρ (represents π ∥ ρ)"""
        ...

A number of concrete classes and patterns are implemented in the Tlon math codebase to make the abstract foundations operational and testable. Some illustrative examples:

class Sequence(Process):
    """A process representing a fixed sequence of sub-processes."""

    def __init__(self, steps: list[Process]):
        self.steps = steps

    @property
    def duration(self) -> float:
        return sum(s.duration for s in self.steps)

    def __matmul__(self, other: "Process") -> "Sequence":
        return Sequence(self.steps + ([other] if isinstance(other, Process) else list(other.steps)))

    def __or__(self, other: "Process") -> "Concurrent":
        return Concurrent([self, other])

    def equiv(self, other, tolerance=1e-9) -> bool:
        # Equivalence up to minor perturbations
        return (
            isinstance(other, Sequence) and
            len(self.steps) == len(other.steps) and
            all(s1.equiv(s2, tolerance) for s1, s2 in zip(self.steps, other.steps))
        )

A Concurrent class expresses parallel composition:

class Concurrent(Process):
    """Represents concurrent execution of several processes."""

    def __init__(self, parts: list[Process]):
        self.parts = parts

    @property
    def duration(self) -> float:
        return max(p.duration for p in self.parts) if self.parts else 0.0

    def __matmul__(self, other: "Process") -> "Sequence":
        return Sequence([self, other])

    def __or__(self, other: "Process") -> "Concurrent":
        return Concurrent(self.parts + ([other] if isinstance(other, Process) else list(other.parts)))

    def equiv(self, other, tolerance=1e-9) -> bool:
        # Permutation-insensitive equivalence
        return (isinstance(other, Concurrent) and
                set(self.parts) == set(other.parts))

Single-shot atomic happenings like events are provided by primitives:

class Event(Process):
    """An instantaneous process (duration == epsilon > 0)."""

    def __init__(self, label: str, duration: float = 1e-6):
        self.label = label
        self._duration = duration

    @property
    def duration(self) -> float:
        return self._duration

    def __matmul__(self, other: "Process"):
        return Sequence([self, other])

    def __or__(self, other: "Process"):
        return Concurrent([self, other])

    def equiv(self, other, tolerance=1e-9) -> bool:
        return (
            isinstance(other, Event) and
            self.label == other.label and
            abs(self.duration - other.duration) < tolerance
        )

Oscillation can be implemented for resonance experiments:

class Oscillation(Process):
    """Models periodic processes (e.g., Lotka-Volterra cycles, pendulum)."""

    def __init__(self, period: float, cycles: int = 1):
        self.period = period
        self.cycles = cycles

    @property
    def duration(self) -> float:
        return self.period * self.cycles

    def __matmul__(self, other: "Process"):
        # Concatenating oscillatory cycles
        if isinstance(other, Oscillation) and other.period == self.period:
            return Oscillation(self.period, self.cycles + other.cycles)
        return Sequence([self, other])

    def __or__(self, other: "Process"):
        return Concurrent([self, other])

    def equiv(self, other, tolerance=1e-9) -> bool:
        return (
            isinstance(other, Oscillation)
            and abs(self.period - other.period) < tolerance
            and self.cycles == other.cycles
        )

These classes allow modeling of concrete systems:
- A NBodyOrbit class (not shown here) composes body-body pairwise processes.
- DampedOscillator for irreversible transients (energy dissipation).
- StableEquilibrium for fixed points.
- Reversal for stable cycles, e.g., as Oscillation(period=T, cycles=1).

The abstraction boundary: any process that satisfies is_stable() according to the abstract axioms above will behave as a "Tlön object": its composition is idempotent (up to tolerance), making mathematical stability operational in code.

Processes are classified in a purity spectrum:

From Primitives to Simulations

The abstract Process class needs to model actual dynamical systems to be useful. The DynamicalProcess base class bridges Tlon's axioms to numerical simulation:

class DynamicalProcess(Process):
    """
    Base class for dynamical system processes.

    A trajectory segment (x₀ → x₁ over dt) is a process.

    Mapping to Tlon axioms:
        - Duration = time interval of evolution
        - Sequential (;) = concatenate trajectories
        - Concurrent (∥) = parallel systems (with coupling)
        - Stability = fixed points / limit cycles
        - Interference = coupling between systems
    """

    def _seq_compose(self, other: 'DynamicalProcess') -> 'DynamicalProcess':
        """Sequential: evolve self, then continue from self's final state."""
        continued = other._evolve(self.final_state, other.duration)
        return self._make_process(TrajectorySegment(
            initial_state=self.initial_state,
            final_state=continued.final_state,
            dt=self.duration + other.duration,  # Axiom B4
        ))

    def _conc_compose(self, other: 'DynamicalProcess') -> 'DynamicalProcess':
        """Concurrent: run both systems in parallel."""
        max_dt = max(self.duration, other.duration)  # Axiom C5
        # Create product system...

This pattern allows any ODE system to be wrapped as a Tlon process. The framework handles composition, stability checking, and equivalence. The subclass provides the vector field.

Simulations: Seeing Tlön in Action

Implementing dynamical systems and watching them demonstrate Tlön concepts has been where the framework proves itself. The Tlon framework provides three things that standard numerical integration does not: a consistent API for composition, automatic stability checking, and a vocabulary for classifying dynamical behavior.

The Double Pendulum: Transient Processes

The double pendulum is the canonical example of chaos in classical mechanics (Strogatz, 2015). In Tlön terms, it demonstrates transient processes - trajectories that never return to themselves.

Two double pendulums started with nearly identical initial conditions ($10^{-6}$ difference). Within seconds, they diverge completely. This is the signature of a transient process.

The implementation wraps the Lagrangian equations of motion in the Tlon Process interface:

class DoublePendulumProcess(Process):
    """
    A double pendulum trajectory as a Tlon Process.

    Tlon interpretation:
        - Process = trajectory segment (x₀ → x₁ over dt)
        - Duration = time elapsed
        - Sequential (;) = concatenate trajectories
        - Stability = fixed points (equilibria)
        - Equivalence = same final state (behavioral)
    """

    def _derivatives(self, state: NDArray) -> NDArray:
        """Lagrangian equations: d/dt [θ1, θ2, ω1, ω2]"""
        θ1, θ2, ω1, ω2 = state
        Δθ = θ1 - θ2
        denom = 2 * m1 + m2 - m2 * np.cos(2 * Δθ)

        # Angular accelerations from Euler-Lagrange equations
        α1 = (-g * (2*m1 + m2) * np.sin(θ1) - m2*g*np.sin(θ1 - 2*θ2)
              - 2*np.sin(Δθ) * m2 * (ω2**2*L2 + ω1**2*L1*np.cos(Δθ))) / (L1 * denom)
        α2 = (2*np.sin(Δθ) * (ω1**2*L1*(m1+m2) + g*(m1+m2)*np.cos(θ1)
              + ω2**2*L2*m2*np.cos(Δθ))) / (L2 * denom)

        return np.array([ω1, ω2, α1, α2])

    def _seq_compose(self, other: 'DoublePendulumProcess') -> 'DoublePendulumProcess':
        """Sequential: continue evolution from self.final_state."""
        final, traj, times = self._integrate_rk4(self._final, other.duration)
        return DoublePendulumProcess(
            initial_state=self._initial,
            final_state=final,
            dt=self._dt + other.duration,  # Axiom B4: durations add
        )

    def equiv(self, other: 'Process', tolerance: float = 1e-6) -> bool:
        """Behavioral equivalence: same final state."""
        if not isinstance(other, DoublePendulumProcess):
            return False
        return self._final.distance_to(other._final) < tolerance

The Process interface forces you to think about what "composing" trajectories means: sequential composition concatenates them, and the duration axiom (B4) ensures time accounting is correct. The stability methods provide immediate feedback about the system's character:

# Create and evolve from a chaotic initial condition
initial = PendulumState(theta1=np.pi/2, theta2=np.pi/2, omega1=0, omega2=0)
proc = DoublePendulumProcess.evolve(initial, dt=10.0)

# The framework answers: is this stable?
print(proc.is_stable())           # False - never returns to itself
print(proc.purity_class())        # "transient"

# Sequential composition: what happens if we run it twice?
proc_twice = proc @ proc  # π ; π
print(proc_twice.equiv(proc))     # False - chaos means π;π ≉ π

# Perturb and compare: sensitivity to initial conditions
perturbed = PendulumState(theta1=np.pi/2 + 1e-6, theta2=np.pi/2, omega1=0, omega2=0)
proc_perturbed = DoublePendulumProcess.evolve(perturbed, dt=10.0)
print(f"Divergence: {proc._final.distance_to(proc_perturbed._final):.2f}")  # Large!