Ox Alpha Benchmarks

There is no official Ox Alpha benchmark report. The public evidence available by August 23, 2026 consists of early community tests with different task sets and harnesses. This page collects those results without smoothing away their limitations.

The numbers below are community-reported, unverified. “Unverified” does not mean false. It means the anonymous model provider has not published or endorsed the result, and we have not reproduced every run independently. A score belongs to its setup: task selection, prompt, tool access, reasoning effort, temperature, retries, timeout, and grader all matter.

Public result table

Rows with “No public comparable result found” are intentionally incomplete. A blank cell is more useful than a number borrowed from a different harness and presented as a head-to-head comparison.

Result 1: the 10-task DeepSWE subset

Ben Davis posted an early run on X using ten DeepSWE tasks. The post reports Ox Alpha at 80%, Fable at 65%, and gpt-5.6-sol at 52%. It also says the “real score” could vary substantially because the run covers a subset. That warning is essential context, not fine print.

Ten binary tasks create large steps: one additional success moves the displayed rate by ten percentage points. Task selection can also favor a model by chance. Without the complete benchmark, repeated runs, and confidence intervals, the result is a reason to investigate—not a basis for declaring Ox Alpha better than Claude Fable 5 or an OpenAI model.

The result does have value. It is dated, names the comparison models, and discloses the tiny sample. It suggests Ox Alpha can solve at least some software-engineering tasks in the tester’s setup. The correct follow-up is a larger, controlled run.

Result 2: LiveCodeBench v6

The Ox Alpha LiveCodeBench repository reports 49 passing solutions from 175 problems, producing 28.0% pass@1. The breakdown in the published result is 51.2% on easy problems, 30.8% on medium problems, and 13.8% on hard problems. Generation failures are reported as zero.

This run tests raw code generation rather than a long-running coding agent. The model received problem statements, generated Python solutions, and got one attempt. It did not receive a repair loop, terminal tools, repository context, or an agent harness that could respond to failures. That makes the test relatively clear, but it does not measure the sustained agentic work emphasized in the OpenRouter description.

The associated Reddit post triggered an important disagreement. Some commenters treated the result as evidence of weak coding performance; others argued that developers use models inside tool-rich agents, so a raw one-shot test misses the product experience. Both observations can be true. The benchmark measures a narrow capability under a defined setup. It should not be dismissed, and it should not be stretched beyond that setup.

Why these two scores cannot be averaged

The DeepSWE subset and LiveCodeBench run do not sample the same unit of work. One concerns software-engineering tasks in a particular agent evaluation; the other concerns single-attempt Python solutions to coding problems. The task counts are 10 and 175. The harnesses, graders, possible context, and opportunity for iteration differ.

Averaging 80% and 28% would create a precise-looking number with no coherent meaning. The same problem appears when someone places a vendor’s official score beside a community result without matching the run conditions. Our Claude Fable 5 comparison and GPT-5 comparison show specifications side by side, but they do not call unmatched benchmark numbers a race.

What the evidence does support

The early record supports four limited conclusions:

1. Ox Alpha can complete some coding and software-engineering tasks under public community setups.
2. Performance varies sharply with the task and harness; a strong tiny subset and a modest one-shot coding result can coexist.
3. There is not enough same-harness data to rank Ox Alpha reliably against Claude Fable 5 or GPT-5.
4. Free API pricing lowers the financial cost of running a private evaluation, but it does not validate quality, speed, or availability.

Community experience is mixed in the same way. In an OpenCode discussion, some developers praise repository reading and multi-file work. Other threads describe capacity failures, slow responses, repeated mistakes, or ignored instructions. Anecdotes can reveal failure modes worth testing, but they cannot supply a pass rate.

How we qualify a new row

We add a numeric result only when the public source identifies the tested model, task or benchmark, score, and date. We look for the task count, prompt or harness, reasoning setting, retry policy, timeout, grader, and code or output artifacts. Missing details are called out in the limitation column. Screenshots without a method can appear in a discussion section, but not as equivalent evidence to a reproducible repository.

We do not copy a comparison score from a vendor chart into an Ox Alpha community row unless both models were evaluated in the same run. We also avoid speed labels such as “fast” or “slow” in the table until a source publishes workload, tokens per second, and provider conditions. OpenRouter capacity during a viral free preview can distort latency independently of model generation speed.

Build your own decision benchmark

Public benchmarks are a starting point. For a production decision, choose 10–30 tasks from your real workflow that the model could not have memorized as a public leaderboard. Include straightforward work, ambiguous bugs, large-context navigation, tool failures, and cases where the correct action is to ask a question. Freeze the prompt, tools, permissions, timeout, and retry policy across models.

Score task completion, regression rate, human review time, latency, token use, and unsafe or fabricated actions. Keep raw transcripts and final diffs. Repeat enough runs to see variance. The OpenRouter usage guide provides a minimal starting call and prompt templates, while the pricing page records whether the zero-price preview is still available before you launch a large test.

Frequently asked questions

Does Ox Alpha have an official benchmark report?

No. No model card or official benchmark report from the anonymous provider was public as of August 23, 2026.

Can the community results be compared directly with vendor scores?

Usually not. Harnesses, task subsets, reasoning settings, and retry policies differ, so this page keeps methodology notes beside every score.

What should developers test before using Ox Alpha?

Use a private evaluation set that represents your repository, tools, latency limits, and review requirements.

Last updated: 2026-08-23