ZeroEval

What do they actually do

ZeroEval builds tooling to measure and improve AI agents in production. Teams instrument their agents with ZeroEval’s SDK to capture traces, label outcomes, and run targeted evaluations that reflect real user tasks rather than synthetic tests ZeroEval homepage SDK/setup docs.

The product includes instant tracing for multi‑step agent runs and tool calls, so engineers can see exactly what happened during an interaction and query those traces later Instant Tracing Tracing quickstart. It also provides Autotune to compare many model+prompt variants and make it easy to promote the best‑performing option to production with evidence from live or replayed traffic Autotune docs.

For evaluation quality, ZeroEval offers calibrated judges that learn from labeled mistakes to better match human judgment, including for multimodal outputs and longer, multi‑turn behaviors Calibrated judges intro YC launch.

Who are their target customer(s)

• Product managers running AI agents in production: They need to know if changes actually improve user experience; manual QA and offline tests are slow and brittle, making every release risky and time‑consuming to validate YC launch ZeroEval homepage.
• ML / ML‑Ops engineers choosing models and prompts: They spend time A/B testing models and hand‑tuning prompts and want a repeatable way to compare many model+prompt combinations and roll the best to production Autotune docs ZeroEval homepage.
• Observability / platform engineers tracing multi‑step agents: They face noisy, unstructured logs and lack an easy way to capture, label, and query traces from complex, multi‑turn agents Instant Tracing Tracing quickstart.
• QA or research teams evaluating multimodal outputs: Generic automated judges miss task nuances and require lots of manual correction; they need evaluators that learn from labeled mistakes to align with human judgment YC launch Calibrated judges intro.
• Small engineering teams / early AI startups with limited headcount: They can’t afford to build a custom evaluation pipeline and want quick instrumentation and an SDK that starts capturing production traces and feedback with minimal setup SDK/setup docs ZeroEval homepage.

How would they acquire their first 10, 50, and 100 customers

• First 10: Run a tightly supported pilot for YC startups and early AI teams, instrument their agents with the SDK, and deliver a one‑month trial with clear before/after metrics to help the product/engineering lead sign off YC launch SDK/setup docs.
• First 50: Convert pilots into self‑serve by shipping step‑by‑step onboarding templates, one‑click examples for comparisons and tracing, plus community support and weekly office hours to remove friction Autotune setup Tracing quickstart.
• First 100: Scale outbound to mid‑market ML/Platform teams using early case studies, run co‑marketing/integrations with observability and LLM tool vendors, add a low‑touch pricing tier, and productize calibrated evaluators with measurable ROI examples Calibrated judges intro ZeroEval homepage.

What is the rough total addressable market

Top-down context:

The core “AI model/agent evaluation platforms” market is estimated around $1.3–$1.5B in 2024, which maps directly to ZeroEval’s category DataIntelo MarketIntelo. Adjacent budgets in data/observability and A/B testing add several more billions today and are growing quickly Grand View Research FMI Cognitive Market Research.

Bottom-up calculation:

Assume ~40,000 teams globally building and operating LLM features/agents, with 40% buying dedicated eval/observability tools at an average $8k–$12k ARR for evaluation/autotune modules. That implies a bottom‑up TAM of roughly $1.3B–$1.9B, consistent with top‑down reports.

Assumptions:

• ~40k active teams building/operating LLM apps or agents
• ~40% near‑term adoption of dedicated eval/observability for LLMs
• Average annual spend of $8k–$12k per team on evaluation/autotune functionality

Who are some of their notable competitors

• LangSmith (LangChain): Tracing, dataset management, and evaluation tools for LLM apps; widely adopted by teams using LangChain for agentic systems.
• Langfuse: Open‑source LLM engineering platform for tracing, evaluations, and analytics; popular with teams that prefer self‑hosting.
• Weights & Biases (Prompts/Evals): ML experiment tracking vendor with prompt/evaluation tooling for LLM applications, integrated into broader MLOps workflows.
• Arize Phoenix: Open‑source observability and evaluation for LLMs and agents, including tracing, metrics, and debugging workflows.
• Humanloop: Prompt management and evaluation platform aimed at shipping LLM features faster with experiment tracking and feedback loops.