Does AI resume screening software actually work, and is it worth it (bias, accuracy)?

The strongest evidence comes from resume audit studies, and it points where human-hiring audits have long pointed: identical resumes rank differently depending on the name at the top. Kyra Wilson and Aylin Caliskan of the University of Washington ran more than 500 public resumes against 500 job descriptions across nine occupations using Massive Text Embedding models, the class behind semantic resume matching in commercial tools [1]. White-associated names were preferred in 85.1 percent of cases against 11.1 percent for female-associated names, and Black male-associated names were disadvantaged in up to 100 percent of comparisons [1].

A follow-up experiment tested what happens with a person in the loop. Across 528 participants and 1,526 screening decisions, people shown no recommendations picked across racial groups at roughly equal rates, but when the model favored a group they picked from it up to 90 percent of the time, even when they called the recommendations poor quality [2].

Newer models complicate the picture rather than closing it. A June 2026 paired-resume audit of fourteen language models, at 24,024 paired postings each, found the one 2023-vintage model reproduced a pro-white callback gap of 2.12 percentage points, while every model from 2024 onward showed no significant gap or a reversal of up to 3.01 points [3].

The case people cite most is older and different in kind. Amazon started building a system in 2014 to score applicants from one to five stars, trained on ten years of resumes it had received; because most came from men, it learned to penalize the word "women's" and the names of certain all-women colleges, and Amazon scrapped it in 2017 [6]. That came from supervised learning on hiring records, not from a language model.

Keyword and boolean matching

Keyword filters do what the query says and nothing more, so their errors are a recruiter's errors made faster: a string requiring "CPA" drops the candidate who wrote "Certified Public Accountant." Bias here is written into the requirements rather than learned from data, so it is the easiest kind to fix. Pull the live query on an open requisition and count how many applicants each rule removes.

Statistical ranking trained on past hiring outcomes

This category produced the Amazon failure, and it is where bias is structural rather than incidental. The model learns from records of who was previously advanced or rated well, so patterns in those decisions become patterns in the score, and stripping name, gender, and race from the inputs does not remove them because proxies survive in schools, employers, and employment gaps [5]. Upturn documented a screener that had learned the name "Jared" and high school lacrosse predicted success, a real statistical association with no causal link to the work [5].

Language model and embedding evaluation of free text

These tools carry associations from pretraining rather than from your hiring history, which is why disparities appear with no employer data involved: the Washington audit ran on off-the-shelf embedding models with no fine-tuning and the name gaps appeared anyway [1]. Those associations move between releases, so treat the model version and prompt as a fixed configuration and re-audit whenever either changes [3].

Structured parsing and field extraction

Parsing carries the lowest risk of the four because it produces no score and expresses no preference. New York City's rule draws the same line, excluding from its definition of a simplified output tools that only translate or transcribe existing text, such as converting a resume from a PDF [8]. The real risk is quiet data loss, since a mangled skills section becomes a thin record every downstream filter inherits.

"Works" has no agreed meaning in this market, and that is the first thing to settle before reading any accuracy claim. A screening model is trained against a target variable someone chose, and in commercial tools that choice usually belongs to the buyer. Raghavan, Barocas, Kleinberg, and Levy reviewed 18 vendors of algorithmic pre-employment assessments and found vendors leaving it to clients to decide what outcomes to predict, naming performance reviews, sales numbers, and retention time as examples [4]. Vendor websites generally do not make clear whether models are validated or how validation data was selected [4].

So a tool can post an excellent accuracy figure and still be worth nothing to you. If the label it learned was "this applicant was advanced to a phone screen last year," high accuracy means it agrees with last year's recruiters. It has automated the existing screen, including whatever that screen was getting wrong, with no evidence about who does the job well.

A tool trained on real performance data still meets a ceiling that predates this software. Sackett and colleagues re-examined the meta-analytic validity literature and concluded that range restriction corrections had substantially overstated the validity of most selection procedures, cutting mean estimates by .10 to .20 points, with the structured interview highest ranked [7]. Nothing in personnel selection predicts job performance with anything close to precision, so a tool claiming to find the best candidate is claiming something the field has never shown.

The measurement standard already exists in federal enforcement practice: compare selection rates across groups and flag any group below four-fifths of the highest rate. Under 29 CFR 1607.4(D), such a rate "will generally be regarded by the Federal enforcement agencies as evidence of adverse impact" [12], and the Uniform Guidelines require validity evidence only where a procedure adversely affects a group's opportunities [13].

New York City turned that arithmetic into a filing requirement. A bias audit under the DCWP rule must calculate the selection rate and impact ratio for every EEO-1 category, separately for sex, for race and ethnicity, and for intersectional categories, and must report how many assessed people fell into an unknown category [8]. Where the tool scores rather than selects, scoring rate above the sample median replaces selection rate [8].

How often any of this happens is a separate question. A 2024 study at the ACM Conference on Fairness, Accountability, and Transparency sent 155 investigators to 391 employer websites and found audit reports for 18 employers and transparency notices for 13 [9]. The authors call that "null compliance" rather than non-compliance, since employers decide for themselves whether a tool is in scope [9].

Three jurisdictions impose specific obligations on employers using automated tools to screen candidates, while the federal baseline stays ordinary discrimination law rather than an AI statute. This is general information, not legal advice; confirm current requirements with counsel before deploying anything.

New York City

Local Law 144, implemented by DCWP rules effective July 5, 2023, bars use of an automated employment decision tool unless it has had a bias audit within the past year, a summary of results is published on your site before use, and notice reached candidates at least 10 business days beforehand with instructions for requesting an alternative process or accommodation [8]. The definition reaches any tool whose simplified output is relied on alone, weighted more heavily than any other criterion, or used to overrule human conclusions, so a ranking tool sitting beside recruiter judgment can still be covered [8].

Illinois

HB 3773 became Public Act 103-0804 and took effect January 1, 2026, amending the Illinois Human Rights Act to bar artificial intelligence that discriminates on the basis of a protected class or that uses zip codes as a proxy for one, and requiring notice to workers when AI is used in hiring, promotion, discipline, and other employment decisions [10].

Colorado

Colorado's position moved twice, so older summaries are unreliable. SB26-189, signed May 14, 2026, repeals the 2024 Colorado AI Act and reenacts it as a disclosure and rights framework effective January 1, 2027 [11]. Deployers will have to notify workers before covered technology influences an employment decision, give a plain-language explanation of an adverse outcome within 30 days, and allow meaningful human review [11].

Federal

The Uniform Guidelines on Employee Selection Procedures still govern, and they reach any procedure used to make an employment decision, including application screening [13]. The EEOC's 2023 technical assistance document on adverse impact in algorithmic selection no longer resolves on eeoc.gov, so anchor your analysis to the Uniform Guidelines and 29 CFR 1607 instead [12][13].

AI screening holds up where applicant volume exceeds human capacity, requirements are concrete, the tool ranks instead of rejecting, and someone recalculates impact ratios on a set cadence.

Conditions where it holds up

High-volume hourly and entry-level requisitions are the clearest case, because the realistic alternative is a recruiter skimming a thousand resumes at a few seconds each. A ranking that surfaces candidates for a human to read, with the full pool still reachable, changes the order of review instead of the outcome. A forklift certification is a fact you can verify, and a tool sorting on facts is auditable in a way a tool scoring "culture fit" is not [4]. The configuration that survives scrutiny is ranking, plus human review of the list, plus a standing adverse impact calculation [12].

Conditions where it is hard to defend

Small requisitions rarely justify the exposure, since forty applications are readable by a person and the audit overhead outweighs the time saved. Subjective and senior roles fit poorly, because the qualities that matter are the ones a model reaches through proxies, where the documented disparities live [1][5]. Automatic rejection without review turns a ranking error into a final decision, and it sits inside the NYC definition of a tool relied on alone [8]. Without category-level rates on your applicant flow, you cannot see whether the four-fifths threshold is being crossed [8][12].

Ask for five things in writing, and treat a refusal as its own answer.

The most recent bias audit, with impact ratios by category

Request the audit output itself rather than a compliance badge: selection or scoring rates and impact ratios for each sex, race and ethnicity, and intersectional category, plus the audit date, data source, and count excluded as unknown [8]. An audit on the vendor's pooled data across many employers is not the same as one on your applicant flow, and the NYC rule expects the auditor to get your own historical data [8].

The target variable and the training population

Ask what the model was trained to predict and over which people, in one sentence each. Vendors hand the outcome choice to the client and rarely publish validation methodology, so the answer will not be on the website [4]. If the target is a past decision or rating, the tool measures agreement with prior screening, and should be judged on that.

An explanation you could give a rejected candidate

Ask the vendor to produce, for one real applicant, the reason that applicant ranked where they did. Colorado will require a plain-language explanation of an adverse decision within 30 days plus meaningful human review [11], so a tool that cannot generate one leaves that obligation to you.

The model version and the change policy

Language-model behavior varies by release, and the fourteen-model comparison found demographic gaps flipping direction across generations [3]. Get the version identifier, the change-notice policy, and a written commitment to re-audit after an upgrade.

The evidence behind any performance number

Performance figures are marketing until the underlying study is on the table. Workday markets HiredScore AI for Recruiting with a claimed "54% increase in recruiter capacity within 10 months of launch" and "unbiased, AI-driven candidate grading" [14]; both are vendor claims, not published validation of screening accuracy. Ask which customers, over what period, against what baseline.