| name | ai-claims-and-stakeholder-audit |
| description | Use when auditing how an AI/ML personnel assessment is described and how it affects people — Components 7-9 (information & perceptions) of the Landers & Behrend (2023) framework. Covers first-party developer claims (do they honestly and transparently follow from the audit evidence?), second-party effects on those assessed (candidate reactions, justice, false positives vs. false negatives, what is communicated), and third-party understanding (employment-law experts, regulators, community, public). Triggers: "developer marketing claims vs evidence", "candidate reactions to AI hiring", "applicant fairness perceptions", "false positive vs false negative impact", "what do regulators/public think", "transparency of AI hiring claims". |
| version | 1.0.0 |
| author | OpenMatter-Network |
| license | MIT |
| category | research |
| tags | ["Community","io-psychology","ai-assessment","auditing"] |
| permissions | [] |
AI claims & stakeholder audit (Components 7–9)
The model can be technically sound and still be mis-described or harmful in use. This category
shifts from the model's internals to how information about it is presented, understood, and
experienced by three parties. It leans heavily on the individual-attitudes (justice) lens — see
ai-fairness-lenses.
Component 7 — First-party interpretation (developer claims)
What it is: the messaging the algorithm developer puts out about the model.
Questions to ask: Does all messaging from the developer logically, honestly, and
transparently follow from answers developed elsewhere in the audit?
Apply it (focal example): Does the developer claim the model predicts job performance? What
evidence in the audit forms the basis of that claim? Are important details left out?
Audit emphases:
- Cross-check every public/marketing/sales claim against the model-audit findings (Components
1–6). A claim unsupported by the evidence — or that omits material caveats (range restriction,
k-fold-only validation, untested intersectional subgroups) — is a finding.
- This is informational justice in action: transparency about what is assessed and what is done
with the data shapes how fair the system is perceived to be.
Component 8 — Second-party effects (those assessed)
What it is: impact on the people directly affected by the algorithm's use (candidates).
Questions to ask: Who is directly affected, and how have their outcomes and reactions been
assessed? What is the relative impact of acting on false positives versus false negatives on
second parties?
Apply it (focal example): How do non-selected applicants react to learning the algorithm did
not score them high enough to be selected? What information is communicated to them, and how do
they evaluate that information?
Audit emphases — use justice theory (Lens 1):
- Procedural justice rules the developer may be violating: opportunity to perform, job-relatedness/
face validity (do candidates believe facial expressions predict performance?), reconsideration/
appeal (algorithmic decisions that can't be appealed), two-way communication (AI replacing
human interaction), and propriety (some find AI decisions morally inappropriate).
- Distributive justice: which rule (equality/need/equity) do affected people apply to the outcome?
- Interactional justice: interpersonal (respect/dignity) and informational (adequate explanation,
e.g., an explanatory video before data collection).
- False positives vs. false negatives are not symmetric for second parties. A false negative
(a qualified candidate wrongly screened out) harms the individual; weigh it explicitly against false
positives rather than optimizing a single accuracy number. Tie this to the error-cost reasoning in
selection-decisions-and-scoring.
Component 9 — Third-party understanding (outside observers)
What it is: how outside observers perceive and evaluate the system.
Questions to ask: How have perceptions and evaluation by outside observers been assessed and
incorporated? Have outside regulatory groups and community organizations been consulted?
Apply it (focal example): How do experts in employment law view the documentation and
performance of the algorithm? How does the public view this use of algorithms?
Audit emphases:
- Solicit employment-law review (differential prediction, disparate impact, differential
treatment — Lens 2) and community/regulatory input, rather than assuming internal sign-off
suffices.
- Public trust is part of the value proposition; transparent, outward-facing evaluation raises it
(and is a stated benefit of normalizing audits).
Pitfalls
- Auditing the model but never checking the claims made about it against the evidence.
- Optimizing overall accuracy while ignoring the asymmetric cost of false negatives to candidates.
- Treating candidate reactions as PR rather than justice evidence relevant to fairness.
- No appeal/reconsideration path; no two-way communication.
- Assuming legality without consulting employment-law experts or affected communities.
Checklist
See also
ai-fairness-lenses (justice; legal lens) · ai-audit-reporting (communicating to these audiences) ·
ai-audit-meta-components · fairness-and-bias-analysis ·
selection-decisions-and-scoring (error costs) ·
administration-documentation (candidate communications, feedback)
Source: Landers & Behrend (2023), Table 1 (Components 7–9, "Components relating to information and
perceptions").
