LangFair

What is LangFair?

LangFair is a Python library for evaluating bias and fairness across LLM use cases. Instead of relying on static benchmarks, it follows a Bring Your Own Prompts approach: you supply prompts from your own application, generate responses with any LangChain LLM class, and compute output-based metrics that do not need access to model internals. Supported metric families cover toxicity, stereotypes, counterfactual fairness for text generation and summarization, plus fairness checks for classification and recommendation use cases. The AutoEval class runs the full text-generation workflow in a couple of lines, SelfEval accepts responses you generated yourself, and red-teaming evaluations test worst-case toxicity and counterfactual behavior. The package installs with pip install langfair, comes with a documentation site, demo notebooks and a framework for choosing metrics, and is published by CVS Health. It is free open-source software with no paid plans.

Use cases

• Measure toxicity in LLM outputs
• Test an LLM for stereotype bias
• Run counterfactual fairness checks on prompts
• Audit LLM classification fairness
• Evaluate recommendation fairness
• Red-team worst-case LLM behavior

Who it's for

• ML engineers
• Responsible AI teams
• Data scientists
• AI governance teams

Why use LangFair

• Metrics are computed on prompts from your real application, so results reflect your use case rather than a generic benchmark.
• Output-based metrics work without access to internal model states, which suits governance audits of hosted models.
• A published framework helps pick a relevant subset of metrics instead of computing everything.
• AutoEval reduces a multi-step assessment to a few lines of code.
• Free open-source library with documentation, tutorial and demo notebooks.

LangFair features

• Bring Your Own Prompts evaluation of LLM use cases
• ResponseGenerator for creating multiple responses per prompt with LangChain LLMs
• ToxicityMetrics such as Expected Maximum Toxicity and Toxic Fraction
• StereotypeMetrics including Co-occurrence Bias Score
• Counterfactual fairness metrics with sentiment parity, BLEU and ROUGE-L
• Recommendation and classification fairness metrics
• AutoEval for semi-automated text generation assessments
• SelfEval for bring-your-own-responses evaluation
• Red-teaming evaluations for adversarial toxicity and counterfactuals
• Decision framework for choosing metrics for a use case

LangFair integrations

• LangChain
• PyPI
• Vertex AI

How to get started with LangFair

1. Install the package with pip install langfair, ideally in a virtual environment.
2. Collect a list of prompts from your LLM use case and generate responses with ResponseGenerator or your own pipeline.
3. Compute toxicity, stereotype and counterfactual metrics, or run AutoEval or SelfEval for the full workflow.
4. Review the scores, and optionally run red-teaming evaluations to look at worst-case behavior.

LangFair limitations

• It is a code library, so using it requires writing Python
• You need prompts from your own use case to run evaluations
• Response generation needs access to an LLM through LangChain unless you use SelfEval
• A GPU is optional but speeds up toxicity computation

LangFair pricing

LangFair is open source, so you can use and self-host it for free.

Prices can change and may vary by country or exclude tax. Check the official pricing page.

LangFair alternatives

The main LangFair alternatives listed on Debuly are LangTest, Kiln, Vals AI, Agent Development Kit (ADK), Agentic Security and CloudByte AI. The table compares pricing and platforms; order is A–Z.

1. 

   Agent Development Kit (ADK)

   Google's open-source, code-first Python framework for building, testing and deploying AI agents and multi-agent workflows.

   Developer platformsAutomation and agentsCoding
   Open sourceVisit Agent Development Kit (ADK) (official site)
2. 

   Agentic Security

   Open-source vulnerability scanner that fuzzes and attacks LLM endpoints and agent workflows, from a pip install or CI check.

   Developer platformsCoding
   Open sourceVisit Agentic Security (official site)
3. 

   CloudByte AI

   Observability dashboard for teams using AI coding tools, tracking sessions, token spend, commits, security and DORA metrics.

   Developer platformsCodingProductivity
   FreemiumVisit CloudByte AI (official site)
4. 

   Kiln

   Free desktop app and MIT-licensed Python library for evals, prompt optimization, fine-tuning and synthetic data, with datasets synced through Git.

   Developer platformsCodingAutomation and agents
   FreemiumVisit Kiln (official site)
5. 

   LangTest

   Open-source Python library for testing NLP models and LLMs on robustness, bias, fairness, accuracy and representation.

   Developer platformsCoding
   Open sourceVisit LangTest (official site)
6. 

   Vals AI

   Evaluation platform for testing LLM applications with test suites, plus custom benchmarks built from your GitHub repo.

   Developer platformsCoding
   Visit Vals AI (official site)

LangFair FAQ

Sources & verification

Facts on this page come from the official sources below, each with the date we read it. Written by Debuly; the vendor did not review this page.

Image credits

• Logo: from avatars.githubusercontent.com · fetched 24 Oct 2026 · Vendor trademark, used only to identify the product; removed on request
• Cover: none collected yet, so a generated tile is shown.

Spot an error? Report an issue with this listing. We also remove a logo or image on the owner's request.