Open-source, Git-based data version control that tracks datasets, models and experiments next to your code in ML projects.
Open source
Vals AI
Evaluation platform for testing LLM applications with test suites, plus custom benchmarks built from your GitHub repo.
What is Vals AI?
Vals AI is an evaluation platform for teams that build on large language models. You create test suites made of tests, each with one input and a set of checks such as includes, is_concise or grammar, then run the suite against a chosen model and read pass rates, confidence scores and per-check feedback. Suites can be built in the web app, imported from a library, or driven from a Python SDK and CLI, and a GitHub Actions workflow can run them on every pull request. A second product, Vals Smith, reads a public or private GitHub repository, converts merged pull requests into tasks with hidden tests, and scores the models and coding agents you select. The company also publishes public model comparisons on its site. Pricing is not published in its documentation, so contact Vals AI for plan details.
Use cases
- Test LLM apps with test suites
- Compare models on your own prompts
- Run evaluations in CI on every pull request
- Benchmark coding agents on a GitHub repo
- Grade outputs you already collected from a CSV
- Measure refusal rates and answer confidence
Who it's for
Why use Vals AI
- Gives LLM apps a repeatable test process instead of ad hoc prompt checks.
- Compares several models against the same suite so model choice rests on your own data.
- CI support lets teams catch regressions on every pull request.
- Vals Smith turns a repository's merged pull requests into private benchmark tasks.
- Both a web app and a Python SDK are available, so non-coders and engineers can share suites.
Vals AI features
- Test suites with inputs and checks
- Check operators such as includes, is_concise and grammar
- Runs against selectable models with temperature, max tokens and system prompt settings
- CSV upload to grade outputs you already have
- Confidence evaluation and refusal detection
- Heavyweight factor to repeat checks and reduce variance
- Python SDK and CLI for programmatic runs
- GitHub Actions integration for CI
- Vals Smith custom benchmarks from public or private repositories
Vals AI integrations
How to get started with Vals AI
- Create an account and open the Suites page, then start a new suite or import one from the library.
- Add tests with an input and one or more checks that describe what a good output looks like.
- Start a run, choose the model and parameters, and wait for results.
- Review pass rates, confidence scores and per-check feedback, or script the same flow with the SDK.
Vals AI limitations
- Only a subset of models is available by default; others, including those on Bedrock and Azure, need a request to the Vals team
- Pricing is not published on the documentation pages
Vals AI pricing
We couldn't confirm pricing on the official site. Check vals.ai for current prices.
Prices can change and may vary by country or exclude tax. Check the official pricing page.
Vals AI alternatives
The main Vals AI alternatives listed on Debuly are Braintrust, Promptfoo, Arize Phoenix, Alibaba Cloud Model Studio, IBM watsonx.ai and LangTest. The table compares pricing and platforms; order is A–Z.
| Tool | Pricing | Starting price | Platforms |
|---|---|---|---|
| Vals AI | Pricing unknown | — | Web, Python SDK, CLI |
| Alibaba Cloud Model Studio | Paid | $6/mo | Web, API |
| Arize Phoenix | Open source | — | Web, Self-hosted, Docker, Kubernetes |
| Braintrust | Freemium | $249/mo | Web, API, Self-hosted |
| IBM watsonx.ai | Freemium | — | Web, API, SDK |
| LangTest | Open source | — | Python, Self-hosted |
| Promptfoo | Open source | — | CLI, Web, API, Self-hosted |
Alibaba Cloud Model Studio
Alibaba Cloud platform giving developers API access to Qwen and partner models for text, image, video and speech, billed per token or by subscription.
Arize Phoenix
Open-source tool for tracing, evaluating and experimenting on LLM apps and agents, free to self-host or run locally.
Open sourceVisit Arize Phoenix (official site)
Braintrust
AI observability and evaluation platform for tracing LLM apps and agents, scoring outputs and catching regressions before release.
FreemiumVisit Braintrust (official site)
IBM watsonx.ai
IBM's AI development studio for building generative AI apps and agents with foundation models, prompt tools, and machine learning, billed by usage.
LangTest
Open-source Python library for testing NLP models and LLMs on robustness, bias, fairness, accuracy and representation.
Open sourceVisit LangTest (official site)
Promptfoo
Open-source LLM evaluation and red-teaming tool that finds prompt injections, jailbreaks and data leaks in AI apps and agents.
Open sourceVisit Promptfoo (official site)
Vals AI FAQ
How much does Vals AI cost?
Vals AI does not publish prices in its documentation or product pages. Contact the Vals team through vals.ai for plan and pricing details.
Does Vals AI have an API or SDK?
Yes. There is a Python SDK, installed with pip install valsai, and a CLI. You authenticate with an API key created on the platform's Admin page.
Can Vals AI run in CI?
Yes. The documentation describes a CI/CD integration that runs your suites automatically on every pull request using GitHub Actions.
What is Vals Smith?
Vals Smith builds a custom benchmark from a GitHub repository. It turns merged pull requests into tasks with hidden tests, runs the models and coding agents you choose, and reports pass rates.
Who is Vals AI for?
It suits teams building products on LLMs and teams deciding which model or coding agent to adopt for their own code.
What are alternatives to Vals AI?
Other LLM evaluation tools include Braintrust, LangSmith, Promptfoo, Arize Phoenix and Weights & Biases Weave.
Sources & verification
Facts on this page come from the official sources below, each with the date we read it. Written by Debuly; the vendor did not review this page.
| Source | Type | Accessed |
|---|---|---|
| vals.ai/ | official | |
| vals.ai/comparison | official | |
| docs.vals.ai/get_started/quickstart | docs | |
| docs.vals.ai/get_started/introduction | docs | |
| vals.ai/vals-smith | official | |
| docs.vals.ai/web_app/run | docs |
Image credits
- Logo: from vals.ai · fetched · Vendor trademark, used only to identify the product; removed on request
- Cover: none collected yet, so a generated tile is shown.
Spot an error? Report an issue with this listing. We also remove a logo or image on the owner's request.
More in Developer platforms 4
All developer platforms toolsOpen-source Python library from NVIDIA that adds programmable safety, topic and security rails between your app and an LLM.
Open sourceAsk questions of your data in plain language and get charts and answers, as an open source Python library and a hosted analyst called Annie.
FreemiumA type-safe, model-agnostic Python framework for building agents and LLM applications, from the team behind Pydantic.
Open source
