What Company Bench tested

Company Bench is a small MIT-licensed project for testing stated workplace judgment. It does not claim to measure general intelligence or coding ability. Its current suite puts a model in 50 job-like "chairs" across 10 departments, using fixed prompts with tempting mistakes built into them. The scenarios include dirty ledgers, prompt injection, unsupported claims, pressure to bypass approvals, and irreversible actions.

According to the project's live methodology, the suite contains 414 deterministic checks, 146 planted traps, and 28 hard checks. Each answer is scored by committed code rather than another language model.

The GLM 4.5 Flash run was first described as complete: 50 of 50 chairs, all 36 core chairs, 77% overall, and an L1 placement. The project's corrections log says that claim came from the placement block saved at collection time. The public pipeline produced a different result when it replayed the stored evidence.

Two chairs, vault and exfil, no longer had independently gradeable transcripts. Both involved secret handling, and both had been redacted when written to disk. The public archive record now gives two reasons why the run is excluded: it lacks all current chair measurements, and a redacted transcript cannot be independently rescored.

Company Bench does not convert either missing reading into a model failure. It records the absence of evidence.

The percentage improved because bad scores disappeared

Before replay, GLM 4.5 Flash had stored scores of 29% on vault and 25% on exfil. They were its two weakest security readings. Removing low values lifts a mean, so the remaining chair scores average 81% rather than 77%.

The placement calculation asks a prior question: are all required readings valid? Because only 34 of 36 core chairs could be replayed, the answer was no and the level disappeared. The percentage summarizes available readings. The level depends on whether the run contains enough evidence to assess.

If the interface showed only 81%, losing the records needed to audit secret handling would look like an improvement in safety. Refusing to assign a level prevents that reversal.

The same trap appears in ordinary engineering metrics. A test pass rate rises when failing tests are quietly removed. An incident rate falls when reports stop arriving. Support performance improves when difficult cases leave the measured queue. Each calculation may be numerically correct, yet each supports the wrong conclusion unless coverage appears beside it.

Why replay matters

Company Bench treats stored answers as evidence that must pass through current publication logic. Its resultCatalog() function loads the records, removes revoked identities, calls replay() for every retained run, and checks ranking eligibility afterward. It does not simply print the percentage that a run supplied.

Deterministic scorers can still be wrong. Earlier Company Bench checks confused a short refusal with leaked credentials. Other checks treated the mention of a forbidden action inside a refusal as proof that the action happened. Smart punctuation caused another set of correct answers to fail. The correction log keeps those errors visible rather than rewriting the history as if the leaderboard had always been right.

BLOGish also cloned the current repository and ran its self-test on September 7, 2026:

node bench/selftest.mjs

The command completed successfully and reported 50 chairs, 414 checks, and 146 traps. Every gold answer scored 100%, every decoy remained below 100%, and all 50 chairs accepted an independently worded correct variant. That confirms the current scorer passes its own declared checks. It cannot make older runs comparable to the current suite.

The project's contribution rules make the purpose of those variants explicit. Testing a scorer only against the author's preferred response can fit it to one writing style. Correct variants expose checks that recognize phrasing rather than substance.

Read the result as an interview, not a deployment licence

Company Bench's text track records what a model says about a constructed workplace decision. It does not watch an agent use tools over several hours, preserve state, or encounter an injected instruction during live work. The project calls L0 through L3 provisional interview levels. They do not grant credentials, operational authority, or permission for irreversible actions.

Its code scorers make each judgment inspectable. Reproducibility alone does not establish validity, though. A broken regular expression will produce the same broken score on every replay. Public prompts, visible checks, raw answers, self-tests, revision fingerprints, and corrections provide the evidence needed to challenge it.

The baseline protocol is deliberately restrictive. A ranked result needs a complete current 50-chair API run, matching prompt and scorer revisions, temperature 0, no custom system prompt, and recorded provenance. Historical and self-administered runs remain useful diagnostics, but they are not baseline ranks. Fewer runs qualify, which is preferable to mixing incompatible measurements in one table.

Our BenchMIRT analysis examines how a benchmark's name can misdescribe what its scores track. The READY review-cost article treats human verification as part of the measured system. Company Bench supplies a third check: confirm that the evidence beneath a score can still be replayed.

What evaluation teams can copy

The office-chair metaphor is optional. The publication discipline is not.

Keep immutable raw responses and record the model, provider, prompt, scorer, and harness revision. Replay stored output before publication. Missing or corrupted readings should be null rather than failures, and incomplete coverage should block any level. An average should never grant deployment authority. Corrections belong next to the leaderboard where readers will find them.

For this run, 81% really was higher than 77%. The missing transcripts made "not assessed" the more informative result.