When we first launched Noxus, our vision was clear: to create a powerful no-code platform that would revolutionize how enterprises leverage AI in their processes. What we didn’t necessarily plan for was that, to achieve this, we would need to build a highly scalable orchestration engine for AI workloads, and, due to the nature of our system, this engine also had to be packaged into an easily portable set of services.
But we're getting a bit ahead of ourselves. The goal of this sequence of blog posts is to bring some visibility into what we've been building behind closed doors—from the why to the how. By the end, I hope we’ll have shed some light on the technology behind our platform, the decisions we’ve made, and the data that supports those choices. However, we’ll start with the end: why we built the engine and how we’re making it increasingly developer-friendly.
Why?
Every great solution starts with a problem. In our case, very early in our journey, it was the frustration of trying to scale AI workloads using existing orchestration tools.
Of course, you can pair an open-source stack—say LangChain—and integrate it with one of many data-processing orchestration tools. But what happens when you try to scale that to millions of steps in a single execution? And what if you need to debug intermediate results? Or the data needs to be streamed? Or you require a more fluid execution model to support agentic processes?
There's a common theme on the orchestration tools, either you're sticking to a pre-computed DAG or you're dynamically creating the execution graph as you go. The first felt too restrictive, but the second didn't scale with our needs.
We found ourselves jumping from one solution to another, each time encountering limitations that didn’t quite align with the unique challenges we were facing. Even the modern orchestration solutions now marketed for AI felt like the AI aspect was an afterthought—a thin layer on top of their data engineering roots.
We tried Prefect, Flyte, Temporal, Airflow—you name it. While conceptually these workloads seem similar and may even have comparable representations, when it comes to actually running them and handling edge cases, the abstractions start to lack. Furthermore, some of these tools introduced significant infrastructure dependencies, making it difficult to adhere to the core principle of being "portable".
Building Our Engine: A Necessity, Not a Choice
Rather than continue working around these limitations, we made the decision to build our own orchestration engine relatively early on. Starting from the issues we encountered with previous attempts, we focused on creating a system that could meet a very specific set of requirements:
- Handle AI-specific workloads with built-in resilience, streaming capabilities
- Scale horizontally from a single conversation to thousands of agents in lengthy workflows and millions of steps
- Track every operation while balancing scale with visibility and control
- Support the execution of workloads for both workflows, conversations, and agents
- Minimize the number of moving parts and dependencies in the infrastructure
Over time, the engine itself grew alongside our platform, becoming the workhorse that powers workflows, data processing, agents, and everything in between. Due to the nature of our platform, it is often deployed in our customers' cloud environments. Therefore, when we set out to build it, we also had to ensure it was a "portable" engine—easily packageable and deployable in any cloud.
This introduced its own set of challenges, such as minimizing dependencies and avoiding external services with high infrastructural complexity, which we’ll address in more detail in upcoming blog posts.
Unexpected Interest
As we rolled out our no-code platform to customers, something unexpected and exciting happened. Engineering teams began approaching us with a different request: they wanted to understand what was powering our platform.
Was it a LangChain abstraction? Could they pause execution and restart it programmatically? How did the state tracking work? How did it scale in their cloud? Could they integrate it directly with their back-office systems?
These were just a few of the questions we received (and, by the way, we’ll answer them throughout this series). With these inquiries in mind, we naturally began considering how we could open up what we’d been building to our users.
As the potential for using our engine as a foundation for building more complex, customized solutions became clear, we embarked on the journey of transforming it into a more developer-friendly experience. This required making a few changes, of course, but most importantly, it meant starting to build bridges to the outside world. The first version of our API became the first of these bridges—and, in a way, this sequence of blog posts serves as the second.
The Path Forward
This growing developer interest has definitely influenced our roadmap. While remaining committed to revolutionizing how enterprises leverage AI in their day-to-day operations, we’re now exposing more of our engine’s capabilities—initially through our API but, eventually, through a more in-depth SDK to extend the engine’s functionality and by open-sourcing several core components of our stack.
Our vision extends beyond powering our platform; we aim to create a rich AI execution layer that can be reused and extended, enabling other core products across enterprises worldwide. This first step is our way of putting ourselves out there and opening the door for broader collaboration.
The journey is a long one, and it will take time to fully realize our vision. However, we want everyone to be part of it. Moving forward, we’ll continue this series of blog posts to share more insights into what we’ve built and what we’re building. Our next blog post will be the first of many focused on the technical aspects of the engine, particularly the data structures and how the orchestration works, so stay tuned.