AI model taxonomy for product and infrastructure teams.

Abstract

Model classification is an operating problem, not just a naming problem.

After cataloging commercial model APIs, the most useful first cut is not the provider brand or the marketing name. It is the shape of the output: text, image, audio, video, embedding vectors or 3D assets. Product teams buy capabilities, but infrastructure teams operate interfaces, pricing units, compatibility rules and failure modes. A usable taxonomy needs to support both perspectives.

This paper adapts an internal KYREN note into a practical classification framework for AI application teams. It separates user-facing modality from model architecture, maps input-output combinations into application directions, and explains why tool use belongs to orchestration rather than model type.

1. Output modality

Start with what the model returns.

From the user perspective, output modality is the most intuitive classification. Four categories return content that people can directly consume: text, image, audio and video. Two categories are different: embeddings are numerical representations for machines, while 3D systems usually return asset files rather than a final human-readable answer.

2. Technical family

LLM is not a synonym for every text-output model.

A common mistake is to call every text-output system an LLM, or to use LLM as a loose synonym for all AI models. In a stricter sense, an LLM is an autoregressive transformer language model: it emits tokens one after another, which is why its commercial interface is usually priced by token usage.

Transformer is not a peer category beside LLM; it is the architectural foundation that made modern LLMs possible. Diffusion systems also increasingly run on transformer-based architectures, and diffusion is no longer limited to still images. Video generation is heavily shaped by diffusion and DiT systems. At the same time, strong image systems are not always best explained as pure diffusion products. The commercial taxonomy therefore needs to record both modality and technical family without collapsing one into the other.

3. Application directions

Applications emerge from input modality combined with output modality.

Once output categories are clear, application mapping becomes easier: ask what is fed into the model and what comes out. The same output modality can support many application directions depending on the input modality. Text output from text input is chat or coding; text output from image input is OCR or visual reasoning; embedding output from text input is retrieval infrastructure; 3D output from image input is reconstruction or asset generation.

Omni-modal systems make this map less rigid, but they do not remove the need for classification. They increase the importance of describing capabilities at the API route level, because a single model name may expose several modes with different cost, latency and reliability profiles.

4. Tool use and agent behavior

Tool use is an orchestration loop around the model.

Tool use, web search and computer use are important capabilities, but they do not create new model categories by themselves. In most implementations, the model emits a textual action: which tool to call, what search query to run, or where to click. External software executes that action, returns the result to the model, and the loop continues until the task is complete.

For tool use and web search, the model-side interface is usually text-to-text: the model emits JSON, arguments or search terms, then reads back textual results. Computer use is different because the model often needs to see the screen. Its input is closer to text plus image, and its output is still an action described in text.

5. Catalog and verification implications

Model operations need records beyond the provider name.

Verification is most mature for LLM-style text outputs because probes are easier to automate and compare. Teams can ask whether an endpoint claiming to be a specific model behaves like that model, whether quality has degraded, or whether hallucination and instruction-following patterns have changed. For image, audio, video and 3D outputs, testing is still possible, but objective scoring is more expensive and less standardized.

For production routing, KYREN treats taxonomy as operational metadata. A model record should distinguish provider, model name, output modality, accepted input modalities, technical family, pricing unit, interface format, aliases, compatibility notes and verification status. Without that separation, routing logic turns into provider folklore rather than infrastructure.