DeepEval vs Opik

The LLM Evaluation Framework

Documentation | Metrics and Features | Getting Started | Integrations | Confident AI

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=de">Deutsch</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=es">Español</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=fr">français</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ja">日本語</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ko">한국어</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=pt">Português</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ru">Русский</a> |
<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=zh">中文</a>

DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which use LLM-as-a-judge and other NLP models that run locally on your machine.

Whether you're building AI agents, voice agents, multi-turn chatbots, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:

• LLM apps end-to-end as black boxes
• Complete AI agent trajectories across every decision and action
• Individual AI agent steps such as LLM calls, tool use, retrieval, and sub-agent handoffs

Use these evaluations to determine the optimal models, prompts, and architecture to improve your AI quality, prevent prompt drifting, or even transition from OpenAI to Claude with confidence.

Jev is now available in DeepEval, both in existing metrics and the latest JevEval.

[!IMPORTANT]
If you're already in production, Confident AI surfaces failing traces before anyone has written an eval for them, clusters them into failure modes, and turns them into your next eval.

🔥 Metrics and Features

• 📐 Large variety of ready-to-use LLM eval metrics (all with explanations) powered by ANY LLM of your choice, statistical methods, or NLP models that run locally on your machine covering all use cases:

  • Custom, All-Purpose Metrics:

    • JevEval — a custom metric scored by Jev, a System One model that answers your questions with calibrated probabilities instead of generated text
    • G-Eval — a research-backed LLM-as-a-judge metric for evaluating on any custom criteria with human-like accuracy
    • DAG — DeepEval's graph-based deterministic LLM-as-a-judge metric builder
  • AI Agent Metrics

    • Task Completion — evaluate whether an AI agent accomplished its goal
    • Tool Correctness — check if the right tools were called with the right arguments
    • Goal Accuracy — measure how accurately the agent achieved the intended goal
    • Step Efficiency — evaluate whether the agent took unnecessary steps
    • Plan Adherence — check if the agent followed the expected plan
    • Plan Quality — evaluate the quality of the agent's plan
    • Tool Use — measure quality of tool usage
    • Argument Correctness — validate tool call arguments
  • Voice Agent Metrics

    • Agent Responsiveness — check whether the voice agent answered when it was spoken to
    • Turn-Taking Naturalness — measure response latency and interruptions across the call
    • Voice Naturalness — evaluate how natural the voice agent sounds
    • Transcription Accuracy — check whether the voice agent's speech-to-text heard the caller correctly
    • Audio Glitch — detect audio artifacts in the voice agent's speech
  • Multi-Turn Metrics

    • Knowledge Retention — evaluate whether the chatbot retains factual information throughout a conversation
    • Conversation Completeness — measure whether the chatbot satisfies user needs throughout a conversation
    • Turn Relevancy — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
    • Turn Faithfulness — check if the chatbot's responses are factually grounded in retrieval context across turns
    • Role Adherence — evaluate whether the chatbot adheres to its assigned role throughout a conversation
  • RAG Metrics

    • Answer Relevancy — measure how relevant the output is to the input
    • Faithfulness — evaluate whether the RAG pipeline's output factually aligns with the retrieval context
    • Contextual Recall — measure how well the RAG pipeline's retrieval context aligns with the expected output
    • Contextual Precision — evaluate whether relevant nodes in the RAG pipeline's retrieval context are ranked higher
    • Contextual Relevancy — measure the overall relevance of the RAG pipeline's retrieval context to the input
    • RAGAS — average of answer relevancy, faithfulness, contextual precision, and contextual recall
  • MCP Metrics

    • MCP Task Completion — evaluate how effectively an MCP-based agent accomplishes a task
    • MCP Use — measure how effectively an agent uses its available MCP servers
    • Multi-Turn MCP Use — evaluate MCP server usage across conversation turns
  • Multimodal Metrics

    • Text to Image — evaluate image generation quality based on semantic consistency and perceptual quality
    • Image Editing — evaluate image editing quality based on semantic consistency and perceptual quality
    • Image Coherence — measure how well images align with their accompanying text
    • Image Helpfulness — evaluate how effectively images contribute to user comprehension of the text
    • Image Reference — evaluate how accurately images are referred to or explained by accompanying text
  • Other Metrics

    • Hallucination — check whether the LLM generates factually correct information against provided context
    • Summarization — evaluate whether summaries are factually correct and include necessary details
    • Bias — detect gender, racial, or political bias in LLM outputs
    • Toxicity — evaluate toxicity in LLM outputs
    • JSON Correctness — check whether the output matches an expected JSON schema
    • Prompt Alignment — measure whether the output aligns with instructions in the prompt template

• 🎯 Supports both end-to-end and component-level LLM evaluation.

• 🧩 Build your own custom metrics that are automatically integrated with DeepEval's ecosystem.

• 🔮 Generate single and multi-turn synthetic datasets, and simulate live voice calls against your voice agent.

• 🔗 Integrates seamlessly with ANY CI/CD environment.

• 🧬 Optimize prompts automatically based on evaluation results.

• 🏆 Easily benchmark ANY LLM on popular LLM benchmarks in under 10 lines of code., including MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K.

🔌 Integrations

DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more — and into voice stacks like Vapi, ElevenLabs, LiveKit, and Pipecat. The same traces can go to Confident AI once you're live, where evals run on every production trace.

Frameworks

• LangChain — evaluate LangChain applications with a callback handler
• LangGraph — evaluate LangGraph agents with a callback handler
• Pydantic AI — evaluate Pydantic AI agents with type-safe validation
• CrewAI — evaluate CrewAI multi-agent systems
• Anthropic — evaluate and trace Claude applications via a client wrapper
• AWS AgentCore — evaluate agents deployed on Amazon AgentCore
• Google ADK — evaluate Google ADK agents and multi-agent workflows
• AI SDK — evaluate AI SDK generations and tool-loop trajectories
• Mastra — evaluate Mastra agents and workflows with native tracing
• OpenAI — evaluate and trace OpenAI applications via a client wrapper
• OpenAI Agents — evaluate OpenAI Agents end-to-end in under a minute
• LlamaIndex — evaluate AI agents and RAG applications built with LlamaIndex

Voice Agent Connectors

• Vapi — simulate calls to a Vapi assistant over its WebSocket transport
• ElevenLabs — simulate calls to an ElevenLabs conversational agent
• LiveKit — join your agent's LiveKit room and trade audio over WebRTC
• Pipecat — connect to your Pipecat pipeline over its WebSocket transport

☁️ Platform + Ecosystem

Confident AI is AI monitoring for applications in production. DeepEval tests what you know to test before you ship; Confident AI finds what you didn't, and it's model- and framework-agnostic.

• Failing traces surface before anyone has written an eval for them, clustered into failure modes, with user sentiment and behavior across traces.
• Failing traces route to the right annotators and become goldens, so production issues turn into your next eval.
• For enterprise teams, one system for AI quality: evals, monitoring, human annotation, and voice-native observability, with governance as release gates.
• Prefer to stay in your editor? Inspect failing traces and pull the datasets built from them in Claude Code or Cursor through Confident AI's MCP server.

🤖 Vibe-Coder QuickStart

Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your AI agent, voice agent, multi-turn chatbot, RAG pipeline, or any other LLM app, and ask it to generate a dataset, write the eval suite, run deepeval test run, and iterate on the failing metrics.

Start with the 5-minute vibe-coder guide.

🚀 Human QuickStart

Let's pretend your LLM application is a customer support AI agent; here's how DeepEval can help test what you've built.

Installation

Deepeval works with Python>=3.10+.

pip install -U deepeval

Create an account (highly recommended)

Using the deepeval platform will allow you to generate sharable testing reports on the cloud. It is free, takes no additional code to setup, and we highly recommend giving it a try.

To login, run:

deepeval login

Follow the instructions in the CLI to create an account, copy your API key, and paste it into the CLI. All test cases will automatically be logged (find more information on data privacy here).

Write your first test case

Create a test file:

touch test_ai_agent.py

Open test_ai_agent.py and write your first test case to run an end-to-end evaluation using DeepEval, which treats your LLM app as a black-box:

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        # Replace this with the actual output from your LLM application
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
    )
    assert_test(test_case, [correctness_metric])

Set your OPENAI_API_KEY as an environment variable (you can also evaluate using your own custom model, for more details visit this part of our docs):

export OPENAI_API_KEY="..."

And finally, run test_ai_agent.py in the CLI:

deepeval test run test_ai_agent.py

Congratulations! Your test case should have passed ✅ Let's break down what happened.

• The variable input mimics a user input, and actual_output is a placeholder for what your application's supposed to output based on this input.
• The variable expected_output represents the ideal answer for a given input, and GEval is a research-backed metric provided by deepeval for you to evaluate your LLM outputs on any custom criteria with human-like accuracy.
• In this example, the metric criteria is correctness of the actual_output based on the provided expected_output.
• All metric scores range from 0 - 1, which the threshold=0.5 ultimately determines if your test has passed or not.

Read our documentation for more information!

Evals With Full Traceability

Use evals_iterator() to run the same dataset through your app, whether you instrument it manually or through one of DeepEval's framework integrations. Because tracing captures the ordered sequence of model decisions, tool calls, and intermediate steps, you can run trajectory-based evaluations over the complete path your AI agent takes.

Here's an example of manual instrumentation:

from deepeval.tracing import observe, update_current_span
from deepeval.test_case import LLMTestCase
from deepeval.metrics import TaskCompletionMetric

@observe()
def inner_component(input: str):
    output = "result"
    update_current_span(test_case=LLMTestCase(input=input, actual_output=output))
    return output

@observe()
def app(input: str):
    return inner_component(input)

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    app(golden.input)

from deepeval.openai import OpenAI
from deepeval.tracing import trace
from deepeval.metrics import TaskCompletionMetric

client = OpenAI()

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator():
    with trace(metrics=[TaskCompletionMetric()]):
        client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": golden.input}],
        )

from agents import Runner
from deepeval.metrics import TaskCompletionMetric

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    Runner.run_sync(agent, golden.input)

import { generateText } from "ai";
import { configureAiSdkTracing } from "deepeval/integrations/ai-sdk";
import { TaskCompletionMetric } from "deepeval/metrics";

const tracer = configureAiSdkTracing({ name: "my-agent" });

const ask = (input: string) =>
  generateText({
    model,
    prompt: input,
    experimental_telemetry: { isEnabled: true, tracer },
  });

// This metric evaluates the complete trajectory captured for this run.
for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await ask(golden.input);
}

import { Mastra } from "@mastra/core/mastra";
import { Observability } from "@mastra/observability";
import { DeepEvalExporter } from "deepeval/integrations/mastra";
import { TaskCompletionMetric } from "deepeval/metrics";

const mastra = new Mastra({
  agents: { agent },
  observability: new Observability({
    configs: {
      deepeval: { exporters: [new DeepEvalExporter()] },
    },
  }),
});

// This metric evaluates the complete trajectory captured for this run.
for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await mastra.getAgent("agent").generate(golden.input);
}

from deepeval.anthropic import Anthropic
from deepeval.tracing import trace
from deepeval.metrics import TaskCompletionMetric

client = Anthropic()

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator():
    with trace(metrics=[TaskCompletionMetric()]):
        client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": golden.input}],
        )

from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator():
    llm.invoke(
        golden.input,
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )

from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator():
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )

from deepeval.metrics import TaskCompletionMetric

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    agent.run_sync(golden.input)

from deepeval.integrations.crewai import instrument_crewai
from deepeval.metrics import TaskCompletionMetric

instrument_crewai()

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    crew.kickoff({"input": golden.input})

from deepeval.integrations.agentcore import instrument_agentcore
from deepeval.metrics import TaskCompletionMetric

instrument_agentcore()

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    invoke({"prompt": golden.input})

import asyncio
from deepeval.evaluate.configs import AsyncConfig
from deepeval.metrics import TaskCompletionMetric

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(
    async_config=AsyncConfig(run_async=True),
    metrics=[TaskCompletionMetric()],
):
    task = asyncio.create_task(agent.run(golden.input))
    dataset.evaluate(task)

import asyncio
from deepeval.evaluate.configs import AsyncConfig
from deepeval.integrations.google_adk import instrument_google_adk
from deepeval.metrics import TaskCompletionMetric

instrument_google_adk()

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(
    async_config=AsyncConfig(run_async=True),
    metrics=[TaskCompletionMetric()],
):
    task = asyncio.create_task(run_agent(golden.input))
    dataset.evaluate(task)

from deepeval.integrations.strands import instrument_strands
from deepeval.metrics import TaskCompletionMetric

instrument_strands()

## This metric evaluates the complete trajectory captured for this run.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    agent(golden.input)

Learn more about component-level evaluations here.

Evaluate Without Pytest Integration

Alternatively, you can evaluate without Pytest, which is more suited for a notebook environment.

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the actual output from your LLM application
    actual_output="We offer a 30-day full refund at no extra costs.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)
evaluate([test_case], [answer_relevancy_metric])

Using Standalone Metrics

DeepEval is extremely modular, making it easy for anyone to use any of our metrics. Continuing from the previous example:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the actual output from your LLM application
    actual_output="We offer a 30-day full refund at no extra costs.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)

answer_relevancy_metric.measure(test_case)
print(answer_relevancy_metric.score)
## All metrics also offer an explanation
print(answer_relevancy_metric.reason)

Note that some metrics are built for specific use cases, such as AI agents, voice agents, multi-turn chatbots, RAG, and more. Make sure to use our docs to pick the right one for your use case.

A Note on Env Variables (.env / .env.local)

DeepEval auto-loads .env.local then .env from the current working directory at import time.
Precedence: process env -> .env.local -> .env.
Opt out with DEEPEVAL_DISABLE_DOTENV=1.

cp .env.example .env.local
## then edit .env.local (ignored by git)

DeepEval With Confident AI

Confident AI monitors AI applications in production. Evals run on every production trace, failing traces are clustered into failure modes, and each one can be routed to the right annotators — engineers, QA, domain experts or PMs — and turned into a golden for your next eval. The traces your app already produces with DeepEval stream to it with no code changes.

Log in from the CLI to connect:

deepeval login

Test runs also appear on the platform once you're logged in.

Prefer to stay in your IDE? Inspect failing traces and pull the datasets built from them through Confident AI's MCP server without leaving your editor.

Learn more in the Confident AI documentation.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Roadmap

Features:

• Integration with Confident AI
• Implement G-Eval
• Implement RAG metrics
• Implement Conversational metrics
• Evaluation Dataset Creation
• Red-Teaming
• DAG custom metrics
• Guardrails

Authors

Built by the founders of Confident AI. Contact jeffreyip@confident-ai.com for all enquiries.

License

DeepEval is licensed under Apache 2.0 - see the LICENSE.md file for details.

Opik: Open-Source LLM Observability, Evaluation & AI Agent Tracing

Opik is the open-source LLM observability and evaluation platform for AI agent tracing, LLM evaluation, prompt management, and production monitoring. Built by Comet. Apache-2.0 licensed, free to self-host the full platform, with 20,000+ GitHub stars.

Website • Slack Community • Twitter • Changelog • Documentation

_{Last updated: 2026-07-17}

🚀 What is Opik?

Opik covers the full LLM application lifecycle, from the first trace in development to production monitoring, for teams building LLM apps and AI agents. Key offerings include:

• AI Agent Tracing & Observability: Deep tracing of LLM calls, conversation logging, and agent activity, with full trace trees for multi-step agents and tool calls.
• LLM Evaluation: Datasets, experiments, and LLM-as-a-judge metrics for hallucination detection, moderation, and RAG assessment.
• Prompt & Agent Optimization: The Opik Agent Optimizer SDK to improve prompts and agents.
• Production-Ready Monitoring: Scalable dashboards and online evaluation rules.
• Opik Guardrails: Features to help you implement safe and responsible AI practices.
• CI/CD Evaluation: A PyTest integration to test LLM pipelines on every commit.

Key capabilities include:

• Development & Tracing:

  • Track all LLM calls and traces with detailed context during development and in production (Quickstart).
  • Extensive 3rd-party integrations for easy observability: Seamlessly integrate with a growing list of frameworks, supporting many of the largest and most popular ones natively (including recent additions like Google ADK, Autogen, and Flowise AI). (Integrations)
  • Annotate traces and spans with feedback scores via the Python SDK or the UI.
  • Experiment with prompts and models in the Prompt Playground.

• Evaluation & Testing:

  • Automate your LLM application evaluation with Datasets and Experiments.
  • Leverage powerful LLM-as-a-judge metrics for complex tasks like hallucination detection, moderation, and RAG assessment (Answer Relevance, Context Precision).
  • Integrate evaluations into your CI/CD pipeline with our PyTest integration.

• Production Monitoring & Optimization:

  • Log high volumes of production traces: Opik is designed for scale (40M+ traces/day).
  • Monitor feedback scores, trace counts, and token usage over time in the Opik Dashboard.
  • Utilize Online Evaluation Rules with LLM-as-a-Judge metrics to identify production issues.
  • Leverage Opik Agent Optimizer and Opik Guardrails to continuously improve and secure your LLM applications in production.

Who it's for: ML engineers building LLM-powered agents, AI teams moving from prototype to production, and engineering teams that need open-source, self-hostable observability they can run in their own environment.

Why open source matters here: Opik is Apache-2.0 licensed and free to self-host: the full platform, backend included, not just a client SDK. The repository includes the server backend, web application, tracing, datasets, experiments, evaluations, prompt management, online evaluation, and agent optimization components, all under Apache-2.0. You can run LLM observability inside your own infrastructure with no data leaving your environment and no Enterprise sales conversation required.

[!TIP]
If you are looking for features that Opik doesn't have today, please raise a new Feature request 🚀

⚡ Quick Start

Install the Python SDK and configure it:

pip install opik
opik configure

Wrap any function with the @track decorator to start logging traces:

from opik import track

@track
def my_function(input: str) -> str:
    return input

Every call to my_function is now logged to Opik, including nested calls, so this works for full agent and pipeline traces, not just single LLM calls. See the Quickstart guide for the TypeScript SDK and other setup options.

Connect your coding agent

Let Claude Code, Cursor, VS Code Copilot, Codex or opencode read your traces, score outputs and run evaluations from chat. One command sets it up. It needs uv and no SDK:

uvx opik mcp configure

The badges and the add-mcp fallback below target Opik Cloud; the command above also handles self-hosted deployments. Other MCP clients on Opik Cloud: npx add-mcp https://www.comet.com/opik/api/v1/mcp --name opik-mcp. Details, troubleshooting and FAQ are in the MCP server guide.

📊 How Does Opik Compare?

Opik competes in the LLM observability / AI agent evaluation category alongside LangSmith, Arize (Phoenix and Arize AX), Weights & Biases (Weave), Langfuse, and Braintrust.

When teams choose Opik: Opik's full observability, evaluation, and optimization platform is Apache-2.0 licensed and free to self-host. Unlike closed platforms whose self-hosted deployment requires an Enterprise plan, Opik can be deployed without a commercial license, and it's framework-agnostic so it won't lock you into a single agent ecosystem. See the table above for where self-hosting and licensing differ across alternatives.

❓ Frequently Asked Questions

Is Opik open source?

Opik is licensed under Apache 2.0. Its server, web application, and core observability and evaluation capabilities can be self-hosted without a commercial license.

Can I self-host Opik?

Yes. Opik can be deployed locally or in your own infrastructure using the documented self-hosting options.

Does Opik support AI agent tracing?

Yes. Opik captures multi-step traces containing LLM calls, tool executions, retrieval steps, and other agent activity.

Does Opik support LLM evaluation?

Yes. Opik supports datasets, experiments, code-based metrics, LLM-as-a-judge evaluation, and online evaluation.

Is Opik tied to a specific agent framework?

No. Opik is framework-agnostic and supports its SDK, OpenTelemetry, and framework-specific integrations.

🛠️ Opik Server Installation

Get your Opik server running in minutes. Choose the option that best suits your needs:

Option 1: Comet.com Cloud (Easiest & Recommended)

Access Opik instantly without any setup. Ideal for quick starts and hassle-free maintenance.

👉 Create your free Comet account

Option 2: Self-Host Opik for Full Control

Deploy Opik in your own environment. Choose between Docker for local setups or Kubernetes for scalability.

Self-Hosting with Docker Compose (for Local Development & Testing)

This is the simplest way to get a local Opik instance running. Note the new ./opik.sh installation script:

On Linux or Mac Environment:

## Clone the Opik repository
git clone https://github.com/comet-ml/opik.git

## Navigate to the repository
cd opik

## Start the Opik platform
./opik.sh

On Windows Environment:

## Clone the Opik repository
git clone https://github.com/comet-ml/opik.git

## Navigate to the repository
cd opik

## Start the Opik platform
powershell -ExecutionPolicy ByPass -c ".\\opik.ps1"

Installation Script Options

The opik.sh and opik.ps1 scripts support the following options:

## Start full Opik suite (default behavior)
./opik.sh

## Start only infrastructure services (databases, caches etc.)
./opik.sh --infra

## Start infrastructure + backend services
./opik.sh --backend

## Enable guardrails with any profile
./opik.sh --guardrails # Guardrails with full Opik suite
./opik.sh --backend --guardrails # Guardrails with infrastructure + backend

## Build the containers from source before starting
./opik.sh --build

## Check that all containers are healthy
./opik.sh --verify

## Stop all containers
./opik.sh --stop

## Stop all containers and remove all Opik data volumes
## WARNING: ALL OPIK DATA WILL BE LOST
./opik.sh --clean

## Show all available options
./opik.sh --help

Use the --help or --info options to troubleshoot issues. Dockerfiles now ensure containers run as non-root users for enhanced security. Once all is up and running, you can now visit localhost:5173 on your browser! For detailed instructions, see the Local Deployment Guide.

Self-Hosting with Kubernetes & Helm (for Scalable Deployments)

For production or larger-scale self-hosted deployments, Opik can be installed on a Kubernetes cluster using our Helm chart. Click the badge for the full Kubernetes Installation Guide using Helm.

💻 Opik Client SDK

Opik provides a suite of client libraries and a REST API to interact with the Opik server. This includes SDKs for Python and TypeScript, plus first-party OpenTelemetry support: any language with an OpenTelemetry SDK — including Java, Ruby, and .NET — can send traces to Opik. For detailed API and SDK references, see the Opik Client Reference Documentation.

Python SDK Quick Start

To get started with the Python SDK:

Install the package:

## install using pip
pip install opik

## or install with uv
uv pip install opik

Configure the python SDK by running the opik configure command, which will prompt you for your Opik server address (for self-hosted instances) or your API key and workspace (for Comet.com):

opik configure

[!TIP]
You can also call opik.configure(use_local=True) from your Python code to configure the SDK to run on a local self-hosted installation, or provide API key and workspace details directly for Comet.com. Refer to the Python SDK documentation for more configuration options.

You are now ready to start logging traces using the Python SDK.

📝 Logging Traces with Integrations

The easiest way to log traces is to use one of our direct integrations. Opik supports a wide array of frameworks, including recent additions like Google ADK, Autogen, AG2, and Flowise AI:

[!TIP]
If the framework you are using is not listed above, feel free to open an issue or submit a PR with the integration.

If you are not using any of the frameworks above, you can also use the track function decorator to log traces:

import opik

opik.configure(use_local=True) # Run locally

@opik.track
def my_llm_function(user_question: str) -> str:
    # Your LLM code here

    return "Hello"

[!TIP]
The track decorator can be used in conjunction with any of our integrations and can also be used to track nested function calls.

🧑‍⚖️ LLM as a Judge metrics

The Python Opik SDK includes a number of LLM as a judge metrics to help you evaluate your LLM application. Learn more about it in the metrics documentation.

To use them, simply import the relevant metric and use the score function:

from opik.evaluation.metrics import Hallucination

metric = Hallucination()
score = metric.score(
    input="What is the capital of France?",
    output="Paris",
    context=["France is a country in Europe."]
)
print(score)

Opik also includes a number of pre-built heuristic metrics as well as the ability to create your own. Learn more about it in the metrics documentation.

🔍 Evaluating your LLM Applications

Opik allows you to evaluate your LLM application during development through Datasets and Experiments. The Opik Dashboard offers enhanced charts for experiments and better handling of large traces. You can also run evaluations as part of your CI/CD pipeline using our PyTest integration.

⭐ Star Us on GitHub

If you find Opik useful, please consider giving us a star! Your support helps us grow our community and continue improving the product.

🤝 Contributing

There are many ways to contribute to Opik:

• Submit bug reports and feature requests
• Review the documentation and submit Pull Requests to improve it
• Speaking or writing about Opik and letting us know
• Upvoting popular feature requests to show your support

To learn more about how to contribute to Opik, please see our contributing guidelines.