LangSmith vs promptfoo

Overview

LangSmith Observability is LangChain's platform for seeing what AI agents and LLM applications actually do in production. It captures traces of every model call, tool call, retrieval step and agent decision, then lets you inspect, compare, evaluate and alert on them.

It is framework-agnostic: you can trace LangGraph and LangChain applications, or instrument any stack with the Python, TypeScript, Go or Java SDKs.

Core capabilities

• Tracing — hierarchical traces show each run, its inputs/outputs, latency, token usage and errors, including nested tool and retrieval calls.
• Agent visibility — follow an agent's reasoning and tool use across steps to find where it went wrong.
• Evaluation — build datasets from real traces, run offline and online evaluators, and compare prompt/model versions.
• Prompt management — version, test and deploy prompts from a central registry.
• Monitoring and alerts — dashboards for latency, error rate, cost and quality; alert when behavior drifts.
• Framework and SDK coverage — native integrations for LangGraph/LangChain plus OpenTelemetry-style instrumentation for other stacks.
• Self-hosting and BYOC — Enterprise can run LangSmith in their own Kubernetes cluster on AWS, GCP or Azure so data never leaves their environment.
• Low overhead — the SDK uses an async callback handler and a distributed collector, so tracing does not block application requests; if LangSmith has an incident, the agent keeps running.

Data and privacy

LangSmith cloud data is stored in GCP us-central-1. LangChain states it does not train on customer data, and customers retain rights to their data. Enterprise self-hosted/BYOC options exist for compliance-sensitive teams.

Getting started

1. Create a LangSmith account at smith.langchain.com.
2. Install the SDK for your language and set the API key.
3. Wrap your app or agent; traces appear in the project view.
4. Build a dataset from interesting traces and add evaluators.
5. Set up monitoring dashboards and alerts for production.
6. For self-hosting, follow the Kubernetes deployment guide.

Pricing

LangSmith has a free tier for development and small-scale production; paid plans scale with trace volume, and Enterprise adds self-hosting/BYOC and support. See langchain.com/pricing for current tiers.

Links

• Product: langchain.com/langsmith/observability
• App: smith.langchain.com
• Docs: docs.smith.langchain.com
• Self-hosting: docs.smith.langchain.com/self_hosting/kubernetes

Promptfoo: LLM evals & red teaming

promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps. Stop using trial-and-error... start shipping secure, reliable agents

Website · Getting Started · Red Teaming · Documentation · Discord

Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.

Quick Start

npm install -g promptfoo
promptfoo init --example getting-started

Also available via brew install promptfoo and pip install promptfoo. Or use npx promptfoo@latest to run any command without installing.

Most LLM providers require an API key. Set yours as an environment variable:

export OPENAI_API_KEY=sk-abc123

Run an eval and view results:

cd getting-started
promptfoo eval
promptfoo view

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more!

What can you do with Promptfoo?

• Test your prompts and models with automated evaluations
• Secure your LLM apps with red teaming and vulnerability scanning
• Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
• Automate checks in CI/CD
• Review pull requests for LLM-related security and compliance issues with code scanning
• Share results with your team

Here's what it looks like in action:

It works on the command line too:

It also can generate security vulnerability reports:

Why Promptfoo?

• Developer-first: Fast, with features like live reload and caching
• Private: LLM evals run 100% locally - your prompts never leave your machine
• Flexible: Works with any LLM API or programming language
• Battle-tested: Powers LLM apps serving 10M+ users in production
• Data-driven: Make decisions based on metrics, not gut feel
• Open source: MIT licensed, with an active community

Learn More

• Getting Started
• Full Documentation
• Red Teaming Guide
• CLI Usage
• Node.js Package
• Supported Models
• Code Scanning Guide

Contributing

We welcome contributions! Check out our contributing guide to get started.

Join our Discord community for help and discussion.