## Article: Introducing Supabase Evals — Open‑Source Benchmarking for AI Coding Agents
Supabase has open‑sourced **Supabase Evals**, a benchmark framework and runtime for testing how effectively AI agents build software using Supabase. The tool runs real coding agents — such as Claude Code, Codex, and OpenCode — against concrete tasks like creating database schemas, debugging Edge Functions, or fixing Row Level Security (RLS) policies, then scores the results. It powers the public leaderboard at supabase.com/evals and an internal regression suite run daily.
### How Supabase Evals Works
Supabase Evals evaluates AI agents across three dimensions:
– **Products**: database, auth, storage, edge functions, realtime, cron, queues, vectors, data API
– **Topics**: RLS, security, migrations, SQL, SDK, observability, self‑hosting, tests, declarative schema
– **Stages**: build, deploy, investigate, resolve
The framework selects minimal scenarios that touch each dimension once, based on real support tickets, bug reports, and GitHub issues. Scenarios are divided into:
– **Benchmark**: Broad, public scenarios that measure general capability
– **Regression**: Narrow, failure‑mode scenarios that run daily without altering published scores
Each scenario executes in a real Supabase‑like environment. The framework boots a containerized hosted‑like stack and a local CLI project, using a `platform‑lite` runtime that exposes a Management API compatible surface backed by `@supabase/lite`. Scoring combines deterministic checks with an LLM‑as‑a‑judge, and agents are allowed one retry before grading.
### Deployment and Requirements
Yes, Supabase Evals is deployable today. The repository `supabase/evals` is public under Apache‑2.0 and runs locally via `pnpm`.
**Requirements include**:
– A Docker daemon for local‑stack runs
– Provider API keys for certain scenarios
– Free ports 54321–54329
It is intended for developer tooling, cloud infrastructure, data platforms, and regulated backends in fintech or healthcare, where an agent writing an incorrect RLS policy would be a security incident. Use cases include regression‑testing documentation and skill edits, gating SDK releases, and comparing agent harnesses head‑to‑head.
### Key Findings
Agents perform strongly on the Build stage:
– Opus 5 and Kimi K3 score 100% unaided
– With skills loaded, Sonnet 5 rises from 78% to 100%, GPT‑5.6 Sol from 89% to 100%, and GPT‑5.4 mini from 78% to 89%
Three weaknesses emerged:
1. Agents tend to hand‑write migrations instead of using declarative schemas
2. Agents verify authentication manually rather than using `@supabase/server`
3. Documentation usage varies: Codex and GPT‑5.6 read about 8 pages per scenario, while Claude Code checks docs in under 40% of scenarios even with skills loaded
### Key Takeaways
– Supabase Evals is Apache‑2.0 open source and scenario‑driven
– Tests run against real containerized Supabase stacks, not mocks
– Scoring mixes deterministic checks with LLM‑as‑a‑judge, with one retry allowed
– Skills matter most for smaller models; top models perform well even without them
—
## FAQ
**What is Supabase Evals?**
Supabase Evals is an open‑source benchmark and framework for testing how well AI agents build software using Supabase. It runs AI coding agents against real tasks and scores their results, powering a public leaderboard and internal regression testing.
**How can I run Supabase Evals?**
You can run it locally via `pnpm`. The repository `supabase/evals` is Apache‑2.0 licensed and requires Docker, provider API keys for some scenarios, and free ports 54321–54329.
**What scenarios does it test?**
It tests a range of Supabase products and topics — such as database schemas, auth, storage, edge functions, RLS, migrations, and SQL — drawn from real support tickets and GitHub issues. Scenarios are split into benchmark (public) and regression (internal) suites.
**How are agents scored?**
Scoring combines deterministic checks (e.g., whether a user can reach certain data) with LLM‑as‑a‑judge evaluation. Agents are allowed one retry if they fail.
**What did the findings reveal?**
Top models like Opus 5 and Kimi K3 achieve 100% on the Build stage unaided. Skills most improve smaller models. Agents often hand‑write migrations, manually verify auth instead of using `@supabase/server`, and vary widely in documentation usage.
—
## Conclusion
Supabase Evals provides a powerful, open‑source way to measure and compare AI coding agents against real Supabase tasks. By running scenarios in containerized, production‑like environments and combining deterministic checks with LLM judging, it offers meaningful insights into agent capabilities and areas for improvement. As the ecosystem evolves, Evals will continue to be a key tool for developers and organizations relying on Supabase and AI‑driven development.
