How to monitor and trace LLM apps in production

By the end of this guide you will have an LLM application that records every request as a trace, shows cost, latency and errors on a dashboard, alerts you when something breaks, and lets you sample real conversations for quality. The job is done by an LLM observability platform: you add a small SDK or an OpenTelemetry exporter to your code, and the platform stores and displays what your app and its model calls did.

If you need LLM observability, the main decision is whether to host the tool yourself or use a managed cloud, and how much evaluation you want built in. This guide walks through the setup and compares seven tools from our Developer platforms category.

Key takeaways

• Instrument your app so each request becomes a trace with its model calls, retrieval, tool calls, tokens, cost and latency, then add session and user identifiers so you can follow a conversation.
• The main decision is hosting: open-source tools you run yourself (Langfuse, Arize Phoenix, Opik, OpenLIT) versus managed platforms (Arize AX and Braintrust, which have free tiers, and Lunary, a paid service).
• Free plans cap volume and retention, for example 25k spans a month on Arize AX Free and Opik Free Cloud, or 50k units a month on Langfuse Hobby.
• Traces contain prompts and answers, which often include personal data. Decide what to mask, who can see it and how long to keep it before you turn on production logging.

What you need before you start

• A running LLM application. Ideally with a staging environment where you can try the setup first.
• Access to the code. You will add an SDK or exporter, so you need to be able to deploy a change.
• A list of what you care about. Cost per user, slow responses, errors, wrong answers, unsafe outputs.
• A privacy decision. Whether prompts can leave your network, and whether personal data must be masked.
• Time. A first trace usually appears within an hour. Dashboards, alerts and evaluations take a few days to tune.
• Difficulty. Moderate. You need to deploy code and, for self-hosting, run a database.

Which kind of AI tool fits this job

Choose by hosting, volume and how much you want beyond tracing.

Open source you can self-host. Langfuse records traces, sessions, users, token usage and cost, and adds prompt management, datasets and evaluators. You can use Langfuse Cloud or self-host under the MIT licence. Arize Phoenix is free to self-host or run locally, takes traces over OpenTelemetry, and has built-in instrumentation for frameworks such as LlamaIndex, LangChain and the Vercel AI SDK. Opik from Comet logs every step an agent takes and can be self-hosted or used in the cloud. OpenLIT is OpenTelemetry-native, covers more than 90 LLMs and agent frameworks, and also lists GPU monitoring.

A managed platform. Arize AX is the managed sibling of Phoenix, with tracing, issue detection that groups repeat failures, and online and offline evals. Braintrust records traces of prompts, responses and tool calls and scores them with judges, code or human reviewers. Lunary logs LLM calls with traces and error stack traces, and adds analytics on usage, cost, topics and user satisfaction. Lunary has no free plan: its Team plan is $20 per user a month, and payment is required before workspace access.

When tracing is not enough, add evaluation. See how to evaluate LLM outputs before you ship for scoring methods, and how to reduce your LLM API costs once the cost data starts coming in.

Step 1: Decide what to observe

Good monitoring starts with questions, not dashboards.

1. List the questions you want to answer: which requests are slow, which fail, which cost most, which users complain.
2. Map your app to a trace: a user request, the retrieval step, each model call, each tool call and the final answer.
3. Choose identifiers to attach: user ID, session ID, prompt version, model name and environment.
4. Decide what must be masked. Lunary lists PII masking as an Enterprise feature, so check what your chosen tool offers.

Check it worked by sketching one request on paper with every step you expect to see in its trace.

Step 2: Choose a tool and hosting model

You end this step with one tool and a place to run it.

1. Decide between self-hosting and managed cloud. Self-hosted Langfuse, Phoenix, Opik and OpenLIT keep traces in your own infrastructure but you run the database. OpenLIT needs a ClickHouse database that you operate.
2. Check free-plan limits. Arize AX Free allows 25k spans a month with 15-day retention. Opik Free Cloud allows 25k spans a month with 60-day retention. Langfuse Hobby includes 50k units a month, 30 days of data access and 2 users. Braintrust Starter keeps data for 14 days.
3. Check how each tool counts usage: spans, units, events or processed data. Langfuse counts traces, observations and scores, and units created by LLM-as-a-Judge also count.
4. Use the comparison table below to see pricing and stated limits side by side.

Check it worked by estimating your monthly volume and confirming it fits the plan or the server you chose.

Step 3: Instrument your application

The goal is a first trace on screen.

1. Install the SDK or exporter in a staging copy of your app. Langfuse offers Python and JavaScript SDKs, OpenTelemetry and a LiteLLM proxy. Lunary offers a lightweight SDK with a one-line monitor call for OpenAI. Phoenix and Arize AX accept OpenTelemetry.
2. Use auto-instrumentation if your framework is supported, so you do not wrap every call by hand.
3. Add session and user identifiers, and the prompt version, to each trace.
4. Send a handful of test requests, including one that fails on purpose.

Check it worked by finding your test requests in the dashboard, each showing prompts, answers, tokens and latency.

Step 4: Track cost, latency and errors

Once traces flow, turn them into numbers you can act on.

1. Build a dashboard of cost per day, cost per user or feature, latency percentiles and error rate. Langfuse and Lunary list custom dashboards, Braintrust offers them on Pro, and Opik and OpenLIT track token usage and cost.
2. Look at the most expensive and slowest traces first. They often come from one prompt, one tool or one user.
3. Compare models and prompt versions on the same dashboard to see the trade-off between quality, speed and price.
4. Review error stack traces and group similar failures. Arize AX's Signal feature clusters repeat failures into a ranked list.

Check it worked by identifying the single largest source of cost and the single most common error.

Step 5: Set alerts

Alerts turn a dashboard into a safety net.

1. Pick a few alerts that matter: error rate, latency, cost per hour or day, and a drop in quality score.
2. Send them where people will see them. Lunary, Langfuse and Opik list alerts, and check the vendor's site for supported channels.
3. Set thresholds from real baseline data rather than guesses, and expect to adjust them.
4. Write down who responds and what they do first.

Check it worked by triggering a test alert and confirming the right person receives it.

Step 6: Score live traffic and collect feedback

Tracing shows what happened; scoring shows whether it was good.

1. Sample a share of production traces for automatic scoring with an LLM judge, code checks or both. Arize AX, Braintrust, Langfuse, Opik and OpenLIT list judge-style evaluations.
2. Add human review for a smaller sample. Langfuse, Arize AX and Opik list annotation queues.
3. Capture user feedback such as thumbs up or down and attach it to the trace.
4. Turn bad traces into test cases. Arize Phoenix lets you build datasets from traces and rerun them through new prompt versions.

Check it worked by finding at least one real failure through scoring or review that your error alerts did not catch.

Step 7: Protect data and keep the system healthy

Production traces are sensitive, so put controls around them.

1. Limit who can see traces. Phoenix lists OAuth2, LDAP and role-based access controls, while SSO and audit logs are often paid tiers, as with Arize AX Enterprise.
2. Mask or drop personal data before it reaches the platform, in your code if the tool cannot do it for you.
3. Set retention to match your policy and the plan, since retention differs between tools.
4. Review the setup each month: unused dashboards, noisy alerts and volume against plan limits.

Check it worked by searching traces for a test email address or phone number and confirming it is masked or absent.

Tools for this job compared

Tool	Good fit for	Pricing	Stated limits
Langfuse	Open-source tracing with prompts, datasets and evaluators, cloud or self-hosted	Freemium, paid from $29/mo (checked 24 Oct 2026)	Hobby: 50k units a month, 30 days of data, 2 users; judge-created units also count
Arize Phoenix	Local or self-hosted tracing and evals over OpenTelemetry	Open source	You provision and maintain it; community support; ELv2 licence rather than a permissive one
Arize AX	Managed tracing, issue detection and evals	Freemium, paid from $50/mo (checked 24 Oct 2026)	Free: 25k spans a month, 15-day retention, 1 GB ingestion; self-hosting, SSO and audit logs are Enterprise
Opik	Open-source tracing and evaluation from Comet	Freemium, paid from $19/mo (checked 24 Oct 2026)	Free Cloud: 25k spans a month, 60-day retention; SSO and service accounts are Enterprise
Lunary	Logging, analytics and prompt management for chatbots and agents	Paid, from $20/mo (checked 24 Oct 2026)	Team: 50k events a month, seats capped at 10; self-hosting, SSO and PII masking are Enterprise
OpenLIT	OpenTelemetry-native tracing with cost and GPU monitoring, self-hosted	Open source	You run the dashboard and a ClickHouse database; custom RBAC and audit logs need the Enterprise build
Braintrust	Tracing plus scorers and experiments	Freemium, paid from $249/mo (checked 24 Oct 2026)	Starter: 14 days of data, 1 GB processed data, 10k scores a month; custom dashboards need Pro

Prices and plan limits change, so confirm the current terms on each vendor's site before you commit.

Common mistakes to avoid

• Logging everything with no privacy plan. Prompts and answers can hold personal or confidential data. Fix it by deciding what to mask, who can view traces and how long to retain them before production logging goes live.
• Tracing only the model call. Many failures sit in retrieval or tool steps. Fix it by tracing the whole request, with a span for each step.
• Forgetting identifiers. Without session, user and prompt version, you cannot follow a conversation or link a regression to a change. Fix it by attaching them from the first trace.
• Ignoring plan limits. Free tiers cap spans, units or retention, and data past the limit may be dropped. Fix it by estimating volume, sampling if needed, and watching usage.
• Relying on cost and latency alone. A fast, cheap answer can still be wrong. Fix it by scoring a sample of live traffic and reading real conversations.

How we put this together

This is an evidence-based guide built from each tool's listing in our directory, which is compiled from the vendor's own site and pricing page. It is not a review based on using the tools. We don't rank tools, and nobody paid to be included. Some outbound links may be affiliate links; see our disclosure.

Frequently asked questions

Is there a free way to monitor LLM apps?

Yes. Arize Phoenix, OpenLIT, Opik and Langfuse are open source and can be self-hosted, and Arize AX, Opik, Langfuse and Braintrust all have free cloud tiers with limits. Self-hosting means you run and maintain the infrastructure.

What is the difference between tracing and logging?

Logging records individual events, while tracing links the steps of one request into a single timeline, so you can see the retrieval, model calls and tool calls behind one answer. LLM observability tools focus on traces, and add token and cost data.

Can I keep prompts and answers inside my own network?

Yes, with a self-hosted option. Phoenix states that traces can stay inside your own network, Langfuse can be self-hosted under the MIT licence, and OpenLIT can be self-hosted under Apache 2.0. Managed clouds receive your traces, so read the vendor's data terms first.

Does monitoring slow my app down?

The listings for these tools do not publish an overhead figure, so there is no number to quote. Test in staging, compare request times with and without the SDK, and only then roll out to production.

How long does setup take?

Start on a staging copy, where a first trace is usually the quickest part. Dashboards, alerts, scoring and privacy controls take longer to settle, because thresholds need real baseline data.

Compare more options in our Developer platforms category.