General Analysis logo

General Analysis

Automated AI Safety and Red Teaming Tools

Summer 2024active2024Website
Artificial IntelligenceSaaSTrust & Safety
Sponsored
Documenso logo

Documenso

Open source e-signing

The open source DocuSign alternative. Beautiful, modern, and built for developers.

Learn more →
?

Your Company Here

Sponsor slot available

Want to be listed as a sponsor? Reach thousands of founders and developers.

Report from about 2 months ago

What do they actually do

General Analysis builds tooling to find and block safety failures in large language models and agentic workflows. Today they offer automated red‑teaming that runs many adversarial attacks to surface jailbreaks and prompt injections, then turn those traces into reusable safety classifiers. They also provide a runtime layer (SDK/API/proxy) that screens inputs/outputs and tool calls, with claimed long‑context support (up to 256k tokens) and low‑latency (~10 ms) enforcement, plus dashboards/logs for incident review and tuning (product page, docs).

They’ve shipped a proxy firewall for Model Context Protocol (MCP Guard) to moderate tool calls, an asset‑management scanner that inventories models, vector stores, knowledge bases and agent pipelines, and open‑source guard checkpoints/benchmarks used for evaluation and community validation (MCP Guard, asset management, guard series/benchmarks). The products are positioned for teams deploying LLMs/agents in production and needing continuous red‑team coverage plus runtime protection and observability (overview).

Who are their target customer(s)

  • Security / InfoSec teams at companies running LLMs: They worry about prompt injection, jailbreaks, and data leakage, and need controls that block or at least log unsafe behavior in real time for incident response and audit (runtime security).
  • SREs / platform engineers operating model-serving and agents: They need guardrails that don’t degrade reliability or latency and can filter long contexts and external tool calls that agents rely on; low‑overhead deployment paths (SDK/proxy) are important (runtime security, MCP Guard).
  • ML engineers / model owners: They need concrete failure cases and a way to convert them into detectors or training data; automated red‑teaming and ready‑made/customizable guard checkpoints help close the loop (overview, docs).
  • Compliance, legal, and trust‑and‑safety teams: They need inventories of AI assets/integrations and auditable evidence (logs, traces, policy mappings) to show risk mitigation to regulators and auditors (asset management, runtime security).
  • Safety researchers and open‑source auditors: They want reproducible checkpoints, benchmarks, and tooling to validate coverage and compare defenses; open releases support independent evaluation (guard series/benchmarks, GitHub).

How would they acquire their first 10, 50, and 100 customers

  • First 10: Convert existing open‑source users (GitHub/Hugging Face) into tightly scoped, supported pilots that run customers’ agent flows through automated red‑teaming and deploy the runtime guard, delivering incident traces and sample enforcement logs to prove value (GitHub, guard series, docs).
  • First 50: Standardize the pilot: publish reproducible benchmarks, attack playbooks, and integration guides; pair a self‑serve trial with a templatized POC package and targeted outreach to security/SRE/ML teams to convert inbound interest at scale (benchmarks/guards, runtime docs).
  • First 100: Layer in an enterprise sales motion and channels: productize managed guard training and audit reporting, list on marketplaces/partner catalogs, and build referral partnerships with cloud/MLOps vendors and consultancies running agent stacks (asset mgmt, MCP Guard).

What is the rough total addressable market

Top-down context:

Spending specifically on AI governance/risk tools is still early but growing quickly: multiple analysts size the AI governance market at roughly $198M–$228M in 2024, with forecasts of ~$1.4B by 2030 and ~$6.6B by 2034 as adoption and regulation expand (Grand View Research, GM Insights). Broader AI adoption is accelerating—78% of organizations reported using AI in 2024—creating demand for safety, governance, and runtime controls as deployments scale (Stanford AI Index 2025).

Bottom-up calculation:

If 5,000 mid‑market and enterprise organizations run LLM agents/tool‑integrated apps in production and spend an average of $100,000 annually on red‑teaming, runtime guardrails, and governance, that implies an initial TAM of ~$500M; doubling coverage to 10,000 orgs at a $100,000 average yields ~$1B.

Assumptions:

  • Near‑term buyer universe: 5,000–10,000 organizations with production LLM/agent use cases across regulated and software‑forward industries.
  • Average annual contract value for combined red‑teaming + runtime guardrails + governance: ~$75k–$150k per organization.
  • Purchases consolidate across multiple apps/teams per org into one or a few platform vendors.

Who are some of their notable competitors

  • Robust Intelligence (Cisco): Enterprise‑focused automated red‑teaming and runtime protection now integrated into Cisco’s AI security portfolio; notable for scale and fit with broader Cisco security stacks (Cisco/Robust Intelligence).
  • HiddenLayer: Model security platform with adversarial testing, runtime defenses, and ML supply‑chain checks; overlaps on red‑teaming and monitoring but is positioned as a broader ML security platform (HiddenLayer).
  • Fiddler AI: Observability and monitoring platform that added guardrails to score/filter prompts and responses; competes on runtime filtering plus model monitoring (Fiddler Guardrails).
  • Straiker: Real‑time guardrails and automated attack/defense agents for agentic applications; overlaps on runtime protection and on‑the‑fly testing of multi‑tool workflows (Straiker Defend AI).
  • Mindgard: Automated red‑teaming and security testing for LLMs focused on finding prompt injections and jailbreaks; closest on automated attack generation/testing pipelines (Mindgard).