We've built a new developer tool that makes it far easier to systematically evaluate, monitor, and improve LLM applications – both during development and in production.
We’re excited to introduce the new product that we’ve been building at Inductor! Having seen too many teams experience (and having experienced ourselves) the pain of going from an LLM-powered demo to production-ready LLM-powered application, we’ve built a developer tool that addresses the most critical problems that teams face when working to build and ship LLM applications. Read on to learn more!
Why we built Inductor
LLMs unlock fantastic new possibilities, and make it easy to create compelling demoware. But, going from an idea or demo to an LLM application that is actually production-ready (i.e., high quality, safe, cost-effective) is painful and time-consuming.
Doing so requires grappling with critical questions like
“How good (i.e., high-quality, safe, cost-effective) is our LLM application?”
“How can we improve it? Are we actually improving?”
“Is our LLM app ready to ship? Does it produce high-quality results sufficiently reliably?”
“How is our LLM app behaving on live traffic?”
In order to build and ship a production-ready LLM application, teams need to answer these questions repeatedly as they iteratively develop, as well as deploy and run live. However, today, answering these questions for LLM apps is painful and time-consuming.
Inductor solves this problem by supplying you with the right workflows and tools to answer these questions rapidly, easily, and systematically. In turn, this enables you to deliver high-quality, safe, and cost-effective LLM apps far more rapidly and easily.
With Inductor, you can
Iterate and ship faster – so that you can increase your productivity and get to market faster.
Inductor makes it easy to continuously test and evaluate as you develop, so that you always know your LLM app’s quality and how it is changing. Inductor also enables you to systematically make changes to improve quality and cost-effectiveness by rapidly testing different app variants and actionably analyzing your LLM app’s behavior.
Reliably deliver high-quality results – so that you can ship confidently and safely.
Inductor enables you to rigorously assess your LLM app’s behavior before you deploy, in order to ensure quality and cost-effectiveness when you’re live. You can then use Inductor to easily monitor your live traffic to detect and resolve issues, analyze actual usage in order to improve, and seamlessly feed back into your development process.
Collaborate easily and efficiently – so that you can loop in your team to help build and ship.
Inductor makes it easy for engineering and other roles to collaborate, for example to get critical human feedback from non-engineering stakeholders (e.g., PM, UX, or subject matter experts) to ensure that your LLM app is user-ready. With Inductor, you no longer need to pass around unwieldy spreadsheets that rapidly become outdated.
Carefully crafted for developers and their teams
Teams need to go through three key workflows in order to build and ship production-ready (i.e., high-quality, safe, cost-effective) LLM applications:
Construct, run, and evolve test suites – in order to systematically assess your LLM app’s quality, safety, and cost-effectiveness.
Iteratively experiment and analyze results– in order to actionably understand your LLM app’s behavior, systematically improve, and rigorously decide when you are ready to ship.
Monitor in production and feed back to dev – in order to understand how your users are actually using your LLM app, address any issues, determine where you need to improve, and feed these insights back into your development process.
We’ve carefully crafted Inductor to make it easy and seamless to do these things in your existing development environment, by using Inductor’s CLI, API, and web UI. Inductor is packed with a powerful, easy-to-use set of capabilities purpose-built for the needs of teams working to build and ship LLM apps:
Test suites
Inductor’s test suites make systematic, continuous evaluation easy – both during iterative development, and prior to deployment (to ensure quality and cost-effectiveness when you’re live). Define test cases and quality measures easily via YAML or our Python API. Easily span the full spectrum of automated to human evaluation by using Inductor’s programmatic, human, or LLM-powered quality measures – with a great workflow for ensuring that your LLM-powered quality measures are rigorously calibrated to your human-defined evaluation criteria. Run test suites via the Inductor CLI or API, and actionably analyze the results via our purpose-built web UI.
Hyperparameters
Massively accelerate your iteration by including hyperparameters in your test suites to automatically test different variants of your app. Inductor enables you to easily define and inject hyperparameters into your LLM app to test anything - for example, prompts, models, RAG strategies, or anything else. Inductor then takes care of running every test case against every combination of your hyperparameter values and making the results actionable. You will very soon also be able to include hyperparameters in your live deployments in order to run and analyze live A/B tests.
Live production monitoring
When your LLM app goes live, just add a single line of code to enable logging and monitoring your live traffic using Inductor. Inductor then automatically logs and enables you to richly analyze your live traffic, in order to detect and resolve any issues, analyze live usage in order to improve, and seamlessly feed back into your development process.
Actionable analytics
Whether you are analyzing the results of a test suite run or examining your live traffic, Inductor makes it easy to rapidly extract actionable insights. Use Inductor’s web UI to view summary statistics for your LLM app’s performance as well as examine individual LLM app executions (whether within a test suite or in live traffic), including the details of any specific execution and evaluations of its quality. Use Inductor’s rich filtering capabilities to identify and diagnose areas for improvement. Easily compare the performance of different versions of your LLM app or different app variants parameterized by hyperparameters in order to systematically improve.
Easy, secure collaboration
Collaboration is often critical to building and shipping production-ready LLM apps. For example, it is often essential to get feedback from non-engineering stakeholders (e.g., PM, UX, or subject matter experts), or to collaboratively analyze test suite results or live traffic. Inductor’s secure sharing and permissioning capabilities enable you to easily share, get feedback, and work together using Inductor – with both technical and non-technical team members.
Automatic versioning and auto-logging
Inductor is powered by a data model and platform that is purpose-built for the workflows required to build and ship production-ready LLM applications. Inductor automatically tracks and versions all of your work (e.g., including all test cases, quality measures, and hyperparameters in every test suite), and provides a variety of auto-logging capabilities, so that you can iterate at the speed of thought while staying organized by default. Because Inductor enables you to define your test suites alongside your code (e.g., via YAML or our API), you can also version your test suites within any existing version control system that you use (e.g., via git).
Get started quickly
We’ve designed Inductor to make it easy to get started fast. As soon as you have an account, it takes just a few minutes to create your first test suite and run it on your LLM app – with zero code modifications. To then also use Inductor to log and monitor your live traffic, you only need to add one line of code to your app.
Inductor is built to work seamlessly in your existing development environment by using our CLI and Python API – whether you work in an IDE such as VS Code, use a text editor and terminal, or work in a notebook environment such as Jupyter or Google Colab. Inductor is also designed to work well with your existing tech stack, including any model (whether you are using an LLM API, an open source model, or your own model) and any way of writing LLM apps (from straight-up Python to LangChain and beyond). The Inductor service is architected from the ground up to be runnable in our cloud account or yours; we offer the option to self-host if desired.
Finally, we’ve also designed our pricing to make it easy to get started – we offer straightforward, developer-friendly, per-user-per-month pricing.
Interested in using Inductor?
Request access here, and follow us as we release new features and content!
Top comments (0)