How to evaluate LLM outputs before you ship

By the end of this guide you will have a repeatable way to check LLM outputs before you ship: a test set of real prompts, a scoring method for each kind of failure, a baseline score, and a check that runs whenever a prompt or model changes. The job is done by an LLM evaluation tool, often with a second model acting as a judge and a person checking the hard cases.

If you want to run LLM evaluation properly, the main decision is how you score outputs and where the tests run: a command-line tool in your repository, or a hosted platform with dashboards and review queues. This guide covers both and compares six tools from our Developer platforms category.

Key takeaways

• Build a test set from real prompts, write down what a good answer looks like, score outputs with a mix of rules, an LLM judge and human review, and record a baseline before you change anything.
• The main decision is where evals run: in a CLI in your own repository and CI (Promptfoo), or in a hosted platform with dashboards, datasets or prompt pipelines (Braintrust, Langfuse, Arize AX, Openlayer, PromptHub).
• An LLM judge is itself a model that can be wrong. Check its scores against a sample of human ratings.
• Free plans cap traces, scores or projects, so check the limits before you route production traffic through one.

What you need before you start

• A defined task. One sentence on what the application should do and what counts as a failure.
• Real example inputs. Prompts from users, support logs or realistic made-up cases, with edge cases and a few adversarial ones.
• Expected answers or criteria. Reference answers where they exist, or a short checklist where they do not.
• API keys for your models. Many eval tools run your prompts against models you supply (PromptHub, for example, has you bring your own LLM API keys), and an LLM judge needs a model too.
• Time. A first working eval takes a few hours. Growing the test set is ongoing work.
• Difficulty. Moderate. You need to be comfortable with YAML or an SDK and with reading model outputs critically.

Which kind of AI tool fits this job

Pick by where your team wants to run tests and review results.

A command-line tool that lives in your repository. Promptfoo is an open-source CLI where test cases are defined in YAML and results are viewed locally. It also does red teaming for prompt injection and data leaks, and plugs into CI/CD pipelines. The free Community edition is described as free forever.

A hosted evaluation and tracing platform. Braintrust records traces, scores outputs with LLM-as-a-judge scorers, autoevals, custom code or human reviewers, and compares experiments in playgrounds. Langfuse is open source, can be self-hosted under the MIT licence or used as Langfuse Cloud, and lists datasets, experiments, LLM-as-judge evaluators and human annotation queues. Arize AX combines OpenTelemetry-compatible tracing with online and offline evals, agent and LLM judges, playgrounds and experiments.

A testing and governance platform for larger teams. Openlayer runs evaluations when models, prompts, data or agent behaviour change, can run tests in CI and block a risky deployment, and adds guardrails and an AI registry for larger organisations, with compliance reports on its Enterprise plan.

A prompt-management tool that includes evals. PromptHub stores prompts with Git-style branches and commits, compares outputs from several providers side by side, and, on its Team and Enterprise plans, runs evaluations with string rules or an LLM as judge in pipelines on commits or merge requests.

Step 1: Define what good output looks like

You cannot score what you have not described.

1. Write the task and the user it serves in two sentences.
2. List the failure types that matter: wrong facts, wrong format, unsafe content, off-topic replies, missing citations, tone.
3. For each failure type, decide whether a rule can catch it (valid JSON, contains a citation), or whether it needs a judge or a person.
4. Rank the failures by cost, so you know which ones must block a release.

Check it worked by giving a colleague two sample outputs and having them label each good or bad using your criteria, and agreeing.

Step 2: Build a golden test set

The test set is the most valuable asset in the whole process.

1. Collect real inputs from logs, support tickets or testers, removing personal data first.
2. Add edge cases: empty input, very long input, other languages, ambiguous requests and attempts to misuse the system.
3. Attach the expected answer or the checklist to each case.
4. Keep the set in version control or in a platform dataset. Braintrust, Langfuse and Arize AX all list datasets, and Promptfoo keeps tests in YAML.

Check it worked by running ten cases by hand and confirming the expected answers are actually right.

Step 3: Choose an evaluation tool

You end this step with one tool, picked by workflow and budget.

1. Decide whether tests should run from the command line and CI, or in a shared hosted dashboard.
2. Check free-plan limits. Braintrust's Starter plan includes 1 GB of processed data and 10k scores a month. Langfuse's Hobby plan includes 50k units a month and 2 users. Arize AX Free allows 25k trace spans a month. Openlayer's Basic plan allows 5 projects.
3. Check whether you need self-hosting. Langfuse can be self-hosted, while Arize lists self-hosting as an Enterprise feature.
4. Use the comparison table below for pricing and stated limits side by side.

Check it worked by confirming your expected test volume fits within the plan.

Step 4: Write scorers: rules, LLM judges and human review

Use the cheapest scorer that can catch each failure.

1. Start with deterministic checks: valid JSON, required fields, length, banned words, presence of a citation.
2. Add an LLM-as-a-judge scorer for fuzzy qualities such as helpfulness or faithfulness to sources. Braintrust, Langfuse and Arize AX list this, and PromptHub does on its Team and Enterprise plans.
3. Write the judge prompt with a clear rubric and ask for a short reason with each score.
4. Set up human review for the cases that matter most. Langfuse lists annotation queues, and Arize AX lists labelling queues.
5. Compare the judge with human ratings on a sample, and fix the rubric until they mostly agree.

Check it worked by finding that the judge's scores match a person's on most of a hand-labelled sample.

Step 5: Run a baseline

Measure the current system before you improve it.

1. Run the full test set against your current prompt and model.
2. Record the scores per failure type, not just one average.
3. Read the worst outputs yourself. Scores point you to problems; reading explains them.
4. Save the run so later runs can be compared against it. Braintrust and Arize AX list experiments for this.

Check it worked by being able to name your three most common failure types and show an example of each.

Step 6: Compare changes and fix failures

Change one thing at a time and re-run.

1. Change a prompt, model, retrieval setting or temperature, one at a time.
2. Re-run the same test set and compare with the baseline.
3. Watch for regressions in cases that used to pass, not just improvements overall.
4. PromptHub's pipelines (Team and Enterprise plans) run evaluators on commits or merge requests so a version that fails does not ship, and Promptfoo findings can appear in pull requests.

Check it worked by seeing a change that improves one failure type without lowering another.

Step 7: Put evals in CI and watch production

Evals only protect you if they run every time.

1. Run the test set in CI on every prompt or model change. Promptfoo plugs into GitHub, GitLab and Jenkins pipelines, and Openlayer lists GitHub Actions, Jenkins and CircleCI.
2. Set thresholds that block a release, based on the failures you ranked in Step 1.
3. Trace production traffic and score a sample. Langfuse, Arize AX and Braintrust all trace production runs; see how to monitor and trace LLM apps in production.
4. Add real failures back into the test set so it grows with your product.

Check it worked by making a deliberately bad prompt change and seeing the pipeline fail.

Tools for this job compared

Tool	Good fit for	Pricing	Stated limits
Promptfoo	YAML evals and red teaming from a CLI, in CI	Open source	Community edition caps red teaming at 10k probes a month; monitoring, SSO and API access are Enterprise
Braintrust	Tracing, scorers and experiments in a hosted platform	Freemium, paid from $249/mo (checked 24 Oct 2026)	Starter: 14 days of data, 1 GB processed data, 10k scores a month; dashboards need Pro
Langfuse	Open-source tracing, datasets and judges, cloud or self-hosted	Freemium, paid from $29/mo (checked 24 Oct 2026)	Hobby: 50k units a month, 30 days of data, 2 users; units from judge runs also count
Arize AX	Tracing plus online and offline evals for agents	Freemium, paid from $50/mo (checked 24 Oct 2026)	Free: 25k spans a month, 15-day retention, 1 GB ingestion; self-hosting and SSO are Enterprise
Openlayer	CI tests, production tracing and governance	Freemium	Basic: 1 member, 5 projects, 20,000 inference logs a month, 3 months retention
PromptHub	Versioned prompts with evals and pipelines	Freemium, paid from $9/mo (checked 24 Oct 2026)	Free plan prompts are all public; evaluations and pipelines are on Team and Enterprise

Prices and plan limits change, so confirm the current terms on each vendor's site before you commit.

Common mistakes to avoid

• Testing only easy cases. A set of friendly prompts hides real failures. Fix it by adding edge cases, other languages and attempts to misuse the system.
• Trusting an LLM judge without checking it. A judge can favour long answers or miss subtle errors. Fix it by comparing its scores with human labels on a sample and tightening the rubric.
• Looking at one average score. An average hides a failure type that matters. Fix it by scoring each failure separately and setting thresholds for the serious ones.
• Changing several things at once. You cannot tell what helped. Fix it by changing one variable per run and keeping the baseline.
• Skipping human review on high-stakes outputs. Medical, legal, financial or safety-related answers need a qualified person. Fix it by routing those cases to review before they ship.

How we put this together

This is an evidence-based guide built from each tool's listing in our directory, which is compiled from the vendor's own site and pricing page. It is not a review based on using the tools. We don't rank tools, and nobody paid to be included. Some outbound links may be affiliate links; see our disclosure.

Frequently asked questions

Is there a free way to evaluate LLM outputs?

Yes. Promptfoo's Community edition is free forever and Langfuse can be self-hosted under the MIT licence. Braintrust's Starter plan, Arize AX Free, Langfuse Hobby and Openlayer Basic are free tiers with caps. You still pay your model provider for the calls the tests make.

What is LLM-as-a-judge?

It means using a language model to score another model's output against a rubric. It scales well but is not perfect, so check it against human ratings on a sample. Braintrust, Langfuse and Arize AX each list it as a scorer type, and PromptHub offers it on its Team and Enterprise plans.

How many test cases do I need?

The tools do not publish a required number. Start with a small set that covers your main tasks and known failure types, then add every real failure you find. A smaller, well-chosen set beats a large, careless one.

Can I run evals without sending data to a vendor?

Partly. Promptfoo's CLI runs locally with results viewed on your machine, and Langfuse can be self-hosted. Your prompts still go to whichever model providers you call. Read each vendor's data terms before using customer data.

How often should I run evals?

Run them on every change to a prompt, model or retrieval setup, ideally in CI, and sample production traffic on a regular schedule.

Compare more options in our Developer platforms category.