Why did we build no/llm?
Deploying and managing Large Language Models (LLMs) in production can quickly become overwhelming. Each provider—such as OpenAI, Anthropic, or Vertex—has distinct APIs, parameter sets (temperature, max_tokens, etc.), and unique limitations.
At Noxus we’ve been dealing with LLMs daily for the past year; we’ve scoured the internet for the best way to manage using different providers at our own platform.
We’ve tried them all, and I mean, all.
And eventually, when you’re trying to prepare your stack for production, and you can’t depend on hosted services, we always found one scenario, one edge case, something, that just wouldn’t work. Some of the open-source tools we could try to adapt, honestly, are not well written from the beginning, lack extensive testing, or are just not our taste.
As you can see, solving this issue, means one of two things:
- developers must sift through extensive documentation, write repetitive setup code, and constantly adjust to the quirks and constraints of each model. This process isn’t just tedious; it's prone to errors, delays, and unexpected costs.
- use an external gateway or a gimped version of a local gateway, or one of the existing open-source-turned company tools that must keep adding complexity and features for the sake of adding them
This is why we’ve resorted to building no/llm and are now sharing it with the community, it’s not our product, we’re not selling it, we just want to give back to a vibrant community and assist some folks who are on the same boat as us.
no/llm – one library, many models, no issues
We're excited to introduce no/llm, an open-source Python library that standardizes how you manage, access and configure LLMs across providers. no/llm offers a unified interface to simplify interactions with various models including hooks for every stage of the request lifecycle, ensuring consistency, reliability, and ease of use. It has a test coverage of almost 100% and is simple enough to be extended on your end.
Why use no/llm?
Centralized Model Configuration
Instead of maintaining separate configurations for each provider, no/llm allows you to define all models and their settings in one place. It abstracts away the provider-specific details, reducing boilerplate code and making it straightforward to switch or add new models to your workflow. It has hooks for everything, really everything.
Automatic Parameter Validation
Different LLMs support different parameters, and using unsupported values can cause API errors or unpredictable behavior. no/llm comes with pre-defined configurations for popular models, clearly outlining supported parameters, value ranges, and required settings. We’re also hoping to expand our repository with the community’s assistance for other providers. It automatically validates parameters, helping you avoid common pitfalls, handling complex parameter scenarios. Depending on your preference, you can set the library to raise errors, clamp invalid values, or issue warnings.
Easy Access to Essential Model Metadata
Information about LLMs—like context window limits, token pricing, and data privacy policies—is often buried in documentation. no/llm aggregates and provides easy programmatic access to this critical metadata. This means you can quickly retrieve details such as context limits or cost per token, enabling smarter decision-making, pricing, and other critical aspects right within your application.
Stay Updated on Model Performance
Choosing the right LLM is highly task-dependent and requires up-to-date performance data. no/llm integrates with platforms like LM Arena, providing the latest benchmarks on various tasks (e.g., coding, summarization). This helps you select model presets based on real-world performance data, not just vendor claims.
Seamless Integration with Existing Tools
We designed no/llm to fit naturally into your current AI development stack. It integrates smoothly with popular libraries including:
- LiteLLM: Simplify routing configurations between multiple models.
- LangChain: Use validated, standardized configurations in LangChain workflows.
- LlamaIndex: Streamline your data indexing and retrieval configurations.
- Pydantic AI: Leverage reliable, validated configurations for AI-enabled Pydantic models.
This means adopting no/llm enhances your existing workflows without disruptive changes.
Getting Started with no/llm
Here's a quick guide to start using no/llm in your project:
Installation
Install no/llm via pip:
pip install no_llm[pydantic-ai]
Quick Start Example
import os
from no_llm.integrations.pydantic_ai import NoLLMModel
from no_llm.registry import ModelRegistry
from no_llm.settings import ValidationMode, settings
from pydantic_ai import Agent
from pydantic_ai.settings import ModelSettings
settings.validation_mode = ValidationMode.ERROR
# or ValidationMode.WARN, ValidationMode.CLAMP
os.environ["OPENROUTER_API_KEY"] = "..."
# Get model from registry
registry = ModelRegistry()
openrouter_models = list(registry.list_models(provider="openrouter"))
print([m.identity.id for m in openrouter_models])
# > ['claude-3.5-haiku', 'claude-3.5-sonnet-v2', 'claude-3.7-sonnet', 'deepseek-chat', 'deepseek-r1-llama-70b-distilled', 'deepseek-reasoner', ...]
no_llm_model = NoLLMModel(*openrouter_models)
# Use with Pydantic AI
agent = Agent(no_llm_model, model_settings=ModelSettings(temperature=1.2))
result = agent.run_sync("What is the capital of France?")
print(result.data)
# > 2025-04-09 09:50:51.375 | WARNING | no_llm.integrations.pydantic_ai:request:220 - Model deepseek-chat failed, trying next fallback. Error: Invalid value for parameter 'temperature'
# > Current value: 1.2
# > Valid range: (0.0, 1.0)
# > Error: Value 1.2 outside range [0.0, 1.0]
# > 2025-04-09 09:50:51.375 | WARNING | no_llm.integrations.pydantic_ai:request:220 - Model deepseek-r1-llama-70b-distilled failed, trying next fallback. Error: Invalid value for parameter 'temperature'
# > Current value: 1.2
# > Valid range: (0.0, 1.0)
# > Error: Value 1.2 outside range [0.0, 1.0]
# :white_tick: gemini-2.0-flash
# > The capital of France is **Paris**.Core Concepts
- Provider System: Manage multiple LLM providers through a unified interface.
- Parameter Management: Validate parameters with type-safe checks and defined value constraints.
- Configuration System: Define models and providers in Python or YAML, supporting clear, structured configuration.
- Registry System: Centralized management of models and providers, simplifying access and maintenance.
Documentation
Detailed documentation is available covering:
- Introduction and Core Concepts
- Provider System
- Parameter Management
- Configuration System
- Registry System
Community-Driven Open Source
no/llm is fully open source. We welcome contributions from the community—whether it's feedback, bug fixes, feature requests, or entirely new integrations. Join us on GitHub to help shape the future of simpler, more reliable LLM management.
Get Started Today
Ready to simplify your LLM management? Visit the no/llm GitHub repository to get started today. Explore our quickstart guides, try out examples, and see how no/llm can streamline your workflow and improve your productivity.
