Most conversations about AI hiring bias stop at the headline: passed, or didn’t. That’s the least useful part of the sentence. An audit examines a defined system, a specific candidate population, a fixed time window, and a named metric — a favorable result supports a conclusion about that test, not a blanket claim about every role or future deployment. The 1978 Uniform Guidelines on Employee Selection Procedures gave employment law its bridge to this kind of testing decades before an algorithm was ever part of the sentence, and the method it introduced — the four-fifths rule — is still the backbone of how these audits get scored today.
Bias Isn’t One Thing — Which Type Decides Which Test Applies
Not every disparity is the same problem, and a good audit doesn’t treat them as interchangeable:
A test built for one of these won’t reliably catch another — which is part of why a single headline number rarely tells the whole story.
The Four-Fifths Rule, With The Actual Math
The rule is plain arithmetic. Take each group’s advancement rate at a given hiring stage — regulators call it the “selection rate” — and compare it to whichever group advanced at the highest rate. If any group’s rate falls below 80% of that top rate, the gap gets flagged as potential adverse impact.
Here’s what that looks like with real numbers. If 50% of men in an applicant pool advance to interview and 30% of women do, women’s rate is 60% of men’s — well under the 80% line, which triggers a closer look. The rule doesn’t say why the gap exists. It just says a gap this size doesn’t get to pass quietly. Eightfold’s Matching Model audit runs exactly this comparison across protected categories, and the full methodology sits on that page — the tables and the underlying data, not a headline number standing alone.
Counterfactual Pairs: The Test Aggregate Numbers Can’t Run
The four-fifths rule tells you whether a gap exists. It doesn’t tell you whether the algorithm caused it, versus the shape of who happened to apply that quarter. For that you need a second, different test. Take one resume. Duplicate it. Change exactly one signal that maps to a protected characteristic — a name commonly associated with a particular gender or ethnicity, say — and leave every qualification identical, byte for byte. Run both versions through the model. If the evaluation changes even though nothing about the candidate’s qualifications did, that’s the algorithm’s own signature on the outcome, isolated from whatever the applicant pool happens to look like this month.
A real audit doesn’t do this once as a spot check — it runs the comparison repeatedly across roles, seniority levels, and demographic signals, and reports the aggregate result, not a single anecdote. This is the fuller version of a test we outlined earlier in this series: not a metaphor about opening a black box, but a specific, repeatable procedure with a number at the end of it. An audit that runs one of these tests and not the other is answering half the question — and most published audits, quietly, only run the first half.
Why Sample Size Decides Whether Any Of This Means Anything
Small numbers distort results structurally, not through anyone’s intent. If only six candidates from a given demographic group applied for a role, one candidate’s outcome can swing that group’s entire advancement rate by fifteen or twenty points — a number that looks dramatic and means almost nothing, because the sample is too thin to support a conclusion in either direction. A rigorous audit sets a minimum sample threshold below which it declines to publish a rate for that group at all.
That decision to withhold a number reads as evasive to people who don’t know the math, and it’s closer to the opposite. A confident-looking statistic built on six data points isn’t more honest than no statistic — it’s less honest, dressed up with false precision. Publishing a number only when the sample can actually support the claim, and saying plainly when it can’t, is the harder and less quotable discipline.
Independent On Paper vs. Independent In Practice
“Independently audited” gets used loosely enough that it’s worth defining precisely. Real independence means three conditions hold at once: the auditor isn’t paid contingent on a favorable result, the auditor designs its own testing methodology rather than working from a script the vendor wrote for it, and the auditor itself is certified by a body that reviews auditors — not just their reports. That third piece is the one easiest to skip and hardest to verify from the outside unless a vendor names who did the work and how that auditor is credentialed. It’s also why testing a matching model once and citing the result for an interview product too would be exactly the shortcut a real audit process doesn’t take — a tool that ranks resumes and a tool that holds a live conversation make different kinds of decisions, and neither one’s results stand in for the other’s.
Seven Steps That Turn A Claim Into Reproducible Evidence
A fairness claim becomes evidence when every conclusion has a traceable record behind it:
- Define the tool, use case, and scope statement.
- Name the protected groups and outcomes being tested.
- Freeze the system version and time period under a dated configuration record.
- Check sample adequacy and document any excluded subgroups.
- Run both the impact-ratio comparison and counterfactual pairs, and retain the outputs.
- Have an independent party review the methodology, not just the result.
- Remediate where needed, retest, and publish — with a change log, not just a headline.
Functionally different tools need their own audits run through this same process. A favorable process description — an account of how a system is designed to work — is not a substitute for actually testing outcomes, and a results table with no visible methodology behind it is a claim about fairness, not evidence of it.
Purpose-Built Models Vs. General LLMs: What Passing A Bias Audit Actually Compares
Purpose-built hiring models and general-purpose language models aren’t interchangeable on this measure. In Eightfold’s own internal research comparing the two, the best general-purpose LLM tested reached a 0.773 intersectional impact ratio, against 0.906 for Eightfold’s purpose-built matching model — general models trained to sound fluent, not trained on verified hiring outcomes, tend to under-score the same way structurally, not occasionally. That’s an internal research comparison, not the third-party published bias audit itself, and it’s worth being precise about which is which when someone cites either number.
A Passing Result Is A Snapshot, Not A Certification
None of this is a one-time stamp. A model trained on last year’s outcomes, run against a candidate pool that’s shifted since, is a different system in practice even when the code hasn’t changed — which is exactly why frameworks like NYC’s Local Law 144 require a fresh audit annually rather than once at launch. An audit with no re-test date attached to it is a photograph: accurate for the moment it was taken, and silent about everything since.
Who’s Accountable Once The Audit Is Published
Publishing a result doesn’t distribute responsibility for it evenly, and a credible governance model says so explicitly:
Humans remain responsible for every material hiring decision regardless of what any audit shows — a favorable result is evidence for that judgment, not a replacement for it.
What This Still Doesn’t Answer
Knowing what a rigorous methodology looks like tells you how to judge an audit that’s already been published. It doesn’t tell you what to do with a vendor who hasn’t published one — which questions separate a real answer from a deflection, or what a red flag looks like in that conversation before it becomes a signed contract. That’s a different checklist than the one this piece just walked through, and it’s the one that actually gets used in a vendor evaluation. That’s next.