DeepEval

An open-source evaluation framework styled as Pytest for LLM applications, with 50+ research-backed metrics including LLM-as-a-judge, G-Eval and hallucination detection, plus synthetic data generation.

Website

https://www.deepeval.com/

Who it's for

Python and TypeScript developers who want local, framework-agnostic eval tests in CI before shipping.

Pricing

Framework free and open source (Apache 2.0). The companion Confident AI monitoring platform is a separate paid service with prices not published.

Category

Agent operations

Last verified

2026-10-22 (current)

Confidence

high

Key facts

• Apache 2.0 licence, maintained by Confident AI
• Evaluates agents, multi-turn chatbots, voice agents, RAG pipelines and MCP systems
• 50+ ready metrics including faithfulness, relevancy, hallucination, bias and toxicity
• Component-level and end-to-end trajectory evaluation
• Local-first; optional login connects to Confident AI for production trace evals

Sources

Every claim above is drawn from these; we do not publish unsourced specifications.

• https://www.deepeval.com/docs/introduction
• https://github.com/confident-ai/deepeval

Notes & uncertainties

Confident AI cloud pricing is not published.

Something changed or out of date? Tell us — corrections are how this stays worth reading.