Everyone carries bias. Jurors carry bias. Judges carry bias. So do the rest of us. The harder question for business today is whether the artificial intelligence tools we increasingly rely upon carry bias too, and if so, what that means for the companies that use them.
The National Association of Insurance Commissioners (NAIC) sounded this alarm early. In its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, circulated to insurance commissioners nationwide, the NAIC warned that unfettered reliance on AI exposes insurers to claims of unfair trade practices, unfair discrimination, unfair claims settlement practices, and violations of state insurance regulations—unless insurers take specific, documented steps to identify and correct AI bias.
That warning has proven prescient. Insurers, employers, social media companies, health systems, and other businesses have already been sued for breach of contract, violations of state and federal anti-discrimination law, unfair business practices, and biometric privacy violations arising out of the AI tools used in the ordinary course of business.
How AI Actually Works
To understand where bias comes from, it helps to understand what the AI is actually doing. According to the American Bar Association's Task Force on Law and Artificial Intelligence, the AI most commonly used in law firms and other professional settings today is “extractive AI”—AI that does not search the open internet in real time, but instead makes predictions, provides analysis, and creates AI products based only on the set of data fed into the AI model. Even so, most of today's extractive AI tools are built on large language models (“LLMs”), which is what allows them to read a prompt written in plain English and respond in kind.
An LLM is trained on an enormous volume of text—books, articles, websites, licensed content, and other material—before it is ever put to use. When a person types a prompt, the model does not retrieve a pre-written answer. It generates its response one token at a time (a token being a word or a piece of a word), continually re-scanning everything it has “learned” to decide which token is statistically most likely to come next, given everything that came before. It does this thousands of times in the course of a single response, weighing multiple possible continuations at each step and selecting the one that its training tells it best fits the prompt, the conversation, and its own internal consistency. In that sense, it “thinks” the way we do—by drawing on accumulated experience to predict what comes next. And, like us, it is susceptible to bias.
Where the Bias Comes From
Bias can enter an AI system at three distinct points, and a business evaluating its own AI risk should think about all three separately.
Bias in the data. An LLM is only as balanced as the material it was trained on. Books, news archives, corporate records, social media, and popular culture all carry the fingerprints of the eras and the authors that produced them—including whatever disparities or stereotypes existed at the time. If a historically underrepresented group appears less often in the training data, or appears there in a stereotyped way, the model can absorb—and later reproduce—that same skew, even where nobody involved intended it.
Bias in the algorithm. The mathematics underneath an LLM is built to reward consistency and to find patterns that repeat. That is a feature, not a bug; it is what allows the model to generalize instead of merely memorizing. But an algorithm optimized to find the most common pattern in its data will, by design, favor the majority pattern over the exception. When that “majority pattern” happens to track a demographic characteristic—the gender associated with a job title, the geography of a customer, the age of a claimant—the algorithm can reproduce a discriminatory pattern as if it were simply the statistically correct answer.
Bias in training and fine-tuning. After initial training, developers fine-tune a model's behavior—often using human reviewers to rate and correct its output. That process can introduce its own bias, depending on who the reviewers are, what they are told to look for, and what the developer decides counts as an acceptable answer. A model can be trained to hedge, acknowledge uncertainty, and present multiple viewpoints in an effort to avoid stereotyping, and developers increasingly do exactly that. But none of these safeguards eliminates bias; they only reduce its more obvious manifestations.
Unlike a human being, an AI is not introspective. It has no capacity to recognize that its own answer reflects bias, because it has no concept of “bias” at all—only patterns and probabilities. It simply assigns a higher probability to some outputs than others, based on what it learned. If the underlying data or the fine-tuning process was skewed, the model will replicate that skew with complete, and completely undeserved, confidence.
An example makes the point. When one of the authors of this article asked an AI model to generate a slide deck depicting treatment providers for soft-tissue injuries, every professional character the model generated was a white male even though the prompt said nothing about race or gender, and the model never asked. It simply generated the images its training and algorithms told it were most likely to satisfy the prompt, and those happened to reflect a stereotype baked into its data. The same dynamic plays out in higher-stakes settings: An AI tool built to rank job applicants can mirror the demographics of the resumes it was trained on and inadvertently screen out qualified candidates who do not fit the historical pattern. It can also surface—and improperly weigh—information an employer has no legal right to consider in the first place, such as an applicant's age, sexual orientation, or medical history, simply because that information exists somewhere in the data available to it.
Liability for AI Bias
The lawsuits are no longer hypothetical, and they are not limited to the companies that build the AI—they extend to the businesses that license and deploy it.
In Mobley v. Workday, Inc., 740 F. Supp. 3d 796 (N.D. Cal. 2024), an applicant sued Workday, a human-resources software vendor, alleging that Workday's AI-driven applicant-screening tools discriminated against him and a putative class on the basis of race, age, and disability. The court refused to dismiss the case, holding that Workday's software did not simply implement criteria set by its customers but actively participated in deciding which applicants would move forward—conduct the court said went to the heart of equal access to employment opportunities. The court pointed specifically to the plaintiff's allegation that Workday's AI relied on biased training data as part of what made the discrimination claim plausible. Discovery in Mobley is ongoing, and in 2025 the court permitted a collective action to proceed under the federal age discrimination statute.
Liability is not limited to the vendor. In EEOC v. iTutorGroup, Inc., No. 1:22-cv-02565 (E.D.N.Y. Aug. 9, 2023), the Equal Employment Opportunity Commission pursued, and settled, a claim against a tutoring company whose licensed AI screening software allegedly rejected applicants automatically based on age and gender. In Harper v. Sirius XM Radio, No. 2:25-cv-12403 (E.D. Mich. filed Aug. 4, 2025), a putative class alleges that Sirius XM's licensed AI hiring tool screened out qualified applicants on the basis of race. And in Liapes v. Facebook, Inc., 95 Cal. App. 5th 910, 313 Cal. Rptr. 3d 330 (2023), review denied (Cal. Jan. 2024), the California Court of Appeal reinstated a class action against Facebook (now Meta) under California's Unruh Civil Rights Act, holding that the plaintiff had adequately alleged that Facebook's ad-delivery algorithm, not the advertisers themselves, used age and gender to decide which users would ever see insurance advertisements in the first place.
Health and disability coverage decisions carry their own exposure. In Estate of Lokken v. UnitedHealth Group, Inc., 766 F. Supp. 3d 835 (D. Minn. 2025), the estates of deceased Medicare Advantage members allege that UnitedHealthcare used an AI tool, nH Predict, to cut off post-acute care despite the insurer's own plan documents promising that coverage decisions would be made by “Clinical Services Staff and Physicians.” The court allowed the breach of contract and bad-faith claims to proceed, reasoning that those claims turn on whether the insurer actually did what its policy said it would do, regardless of any Medicare Act preemption defense. The court has since ordered broad discovery into the insurer's internal AI governance and oversight practices, underscoring that an insurer's internal AI files are now squarely within the scope of coverage litigation.
Biometric data adds another layer of exposure. In Carpenter v. McDonald's Corp., 580 F. Supp. 3d 512 (N.D. Ill. 2022), a putative class action under the Illinois Biometric Information Privacy Act (BIPA) survived a motion to dismiss based on allegations that an AI-driven, voice-recognition drive-through ordering system collected customers' “voiceprints” without the consent BIPA requires. The case is a reminder that AI tools which listen to, watch, or otherwise measure customers or employees can create biometric-privacy exposure entirely separate from any bias claim. Illinois is not the only state with a biometric statute on the books.
Federal regulation adds a further layer. Under the Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. pt. 1607, federal enforcement agencies require covered employers to evaluate their selection procedures — including AI-driven screening tools — for adverse impact on protected groups, and agencies are authorized to pursue enforcement wherever an AI system perpetuates a pattern of discrimination. See 29 C.F.R. § 1607.4.
The Regulatory Patchwork
State and local governments are moving quickly to regulate AI, and a business operating across several jurisdictions—as most insurers, TPAs, and self-insured programs do—needs to track more than one rulebook.
- New York City's Local Law 144 (N.Y.C. Admin. Code §§ 20-870 to 20-874) requires employers using an “automated employment decision tool” to obtain an independent bias audit each year and publish the results.
- California's Civil Rights Council has adopted regulations under the Fair Employment and Housing Act that directly address employer use of AI and other automated decision systems in hiring, and California's new regulations under the California Consumer Privacy Act governing “automated decision-making technology” take effect Jan. 1, 2027.
- The Illinois Human Rights Act likewise prohibits the use of AI that results in unlawful discrimination against employees or applicants.
- Maryland's Facial Recognition Act, Md. Code Ann., Lab. & Empl. § 3-717, bars the use of facial-recognition technology during job interviews absent the applicant's written consent.
Two states have gone further and enacted comprehensive AI statutes that reach private business generally, not just employment. The Colorado Artificial Intelligence Act, Colo. Rev. Stat. § 6-1-1701 et seq., regulates both the developers and the “deployers” of high-risk AI systems and, after a legislatively delayed start, took effect June 30, 2026. It requires deployers to exercise reasonable care to avoid algorithmic discrimination and to document their risk-management program.
The Texas Responsible Artificial Intelligence Governance Act (“TRAIGA”), effective Jan. 1, 2026, prohibits developing or deploying an AI system with the intent to unlawfully discriminate against a protected class, among other prohibited uses, and offers covered businesses an affirmative defense for those who can show substantial compliance with a recognized framework such as the NIST AI Risk Management Framework. More states are expected to follow; dozens of AI-related measures have been enacted nationwide over the past year alone.
Why This Matters
For insurance companies, TPAs, and self-insured employers, the exposure described above is not abstract. It sits directly on top of risks the industry already knows well: bad-faith claims handling, discriminatory underwriting, and unfair or deceptive marketing practices are all long-established causes of action, and each one now has an AI-shaped version. A claims system that leans on AI to evaluate, triage, or deny claims, without meaningful human review, can turn an ordinary coverage dispute into a bad-faith claim, a regulatory inquiry, or, if the pattern repeats across a book of business, a class action.
The Lokken discovery order shows exactly how that plays out: An insurer's internal AI training data, model-validation studies, and override statistics are now discoverable, and gaps in that record can look a great deal like evidence of bad faith. Marketing and underwriting are exposed, too. Liapes shows how an algorithm that merely decides who gets to see an insurance advertisement can trigger civil-rights liability, entirely apart from anything an underwriter or claims adjuster ever decided.
These risks compound in ways that make litigation expensive even when the underlying claim is ultimately defensible: putative class certification, statutory fee-shifting and punitive-damages exposure under several of the statutes discussed above, and the discovery burden of producing training data, model documentation, and audit logs that most companies were never built to track in the first place.
The NAIC Model Bulletin, in Detail
Because the NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers is the single most important piece of AI guidance for insurers, TPAs, and self-insured programs, it is worth unpacking in more detail. The NAIC adopted the Model Bulletin on Dec. 4, 2023, through its Innovation, Cybersecurity, and Technology (H) Committee. As of this writing, the NAIC has not adopted a superseding or amended version of the Bulletin itself; the December 2023 text remains current. What has changed since 2023 is not the Bulletin's language but the machinery being built around it, discussed below.
The Bulletin is guidance, not a self-executing law. It only takes effect in a given state once that state's own insurance department separately issues it, typically by circulating substantially the same text under the department's own bulletin number. It is built on, and incorporates by reference, several other NAIC model acts already embedded in state insurance law. The Bulletin's central point is that none of these existing obligations changes because an insurer used AI, a predictive model, or a third-party vendor to reach a decision. The insurer remains just as responsible for the outcome as if a person had made the call by hand.
The core mandate: a written AIS Program. The Bulletin expects every insurer to develop, implement, and maintain a written program—the Bulletin calls it an “AIS Program”—for the responsible use of AI systems that make or support decisions tied to regulated insurance practices such as underwriting, rating, marketing, claims handling, and fraud detection. The AIS Program should be scaled to the insurer's actual use of AI and to the potential harm a bad decision could cause a consumer—a low-stakes marketing tool warrants less rigor than an AI system that decides whether a claim gets paid.
Governance. The Bulletin calls for a documented governance structure—often a cross-functional committee drawing on underwriting, claims, actuarial, data science, compliance, and legal—with clear accountability running up to senior management. That structure should define who has authority at each stage of an AI system's life cycle, from design through retirement, and should include training, monitoring, escalation, and audit protocols so that a problem surfaces internally before a regulator or plaintiff's lawyer finds it first.
Risk management and internal controls. This is where bias mitigation lives in the Bulletin's structure. Insurers are expected to document their data practices—including data lineage, quality, and, specifically, “bias analysis and minimization”—and to maintain an inventory of the predictive models and AI systems they use, with documentation of how each was developed, validated, and monitored. The Bulletin calls for validating and re-testing AI output on an ongoing basis, including evaluating “model drift,” or, the tendency of a model's accuracy to decay over time as the world it is making decisions about changes but the model does not.
Third-party AI systems and data. Most insurers do not build their own AI; they license it, the same way Workday's customers licensed Workday's screening tools or naviHealth's customers licensed nH Predict. The Bulletin squarely anticipates this and expects insurers to conduct due diligence on a vendor's data and AI system before using it, and to build contract terms giving the insurer audit rights and requiring the vendor to cooperate with regulatory inquiries. An insurer cannot outsource its AI and its accountability at the same time.
Notably, the Bulletin also identifies the NIST AI Risk Management Framework, Version 1.0, as an example of an existing third-party standard an insurer may incorporate into its own AIS Program. That is the same framework Texas's TRAIGA offers as a path to an affirmative defense, discussed above — which means an insurer that builds its governance program around the NIST framework is, in effect, buying itself credibility with regulators and litigants on two fronts at once.
What a regulator will actually ask for. Section 4 of the Bulletin sets out, in granular detail, the kinds of documents a state insurance department may demand during an investigation or market conduct action, with or without a specific complaint prompting it. That includes the written AIS Program itself; records of how and when it was adopted; documentation showing the program is tailored to the insurer's actual risk; policies, procedures, and training materials; the insurer's model inventory and model-specific documentation, including the data source, lineage, and bias analysis for any model under review; validation, testing, and model-drift records; and, where a third-party system is involved, the insurer's vendor due diligence file and the underlying vendor contract. In short, an insurer using AI should assume that everything described above is discoverable in litigation and producible on regulatory demand — which is exactly what played out in the discovery order in Lokken discussed earlier in this article.
State Adoption
As of mid-2026, roughly half of all U.S. jurisdictions—more than two dozen states plus the District of Columbia—have formally adopted the NAIC Model Bulletin, generally with little or no material change to the NAIC's text.
The practical takeaway for our insurance, self-insured, and TPA clients: a single, well-documented AIS Program built to the NAIC's specifications will satisfy the expectations of every state, adopted or not, and is far more efficient than trying to maintain state-by-state variations.
It is also worth watching one live development that is not a new version of the Bulletin, but changes how it will be enforced: in 2026 the NAIC's Big Data and Artificial Intelligence (H) Working Group is piloting an AI Systems Evaluation Tool—a standardized set of questions examiners will use to review an insurer's AIS Program during a market conduct examination. Twelve states are participating in the pilot, which runs from January through September 2026, and the Working Group anticipates the Tool will come up for formal adoption at the NAIC's 2026 Fall National Meeting. Insurers and TPAs should expect that, going forward, a market conduct exam will include structured, specific questions about the AIS Program; not just a general inquiry into whether one exists.
Bias Remediation and Governance
The NAIC's Model Bulletin offers a useful starting template, and its core message is one every business using AI should take to heart: the decision has to remain a human one. The Bulletin instructs insurers that decisions cannot be based on inaccurate, arbitrary, capricious, or unfairly discriminatory recommendations, and that AI can increase that risk unless it is actively supervised.
Concretely, the NAIC recommends that:
- An oversight board or committee of senior management—drawing on underwriting, claims, actuarial, data science, compliance, and legal—select and continuously monitor the AI tools the company uses, watching specifically for signs of biased outcomes;
- The company adopt a written AIS Program—a governance framework tailored to its actual use of AI—and revisit it periodically as the technology, and the company's use of it, evolves;
- Data going into any predictive model or AI system be evaluated for bias, lineage, and quality before it is used, and that AI output be independently validated, tested, and re-tested on an ongoing basis—including monitoring for model drift—to confirm it remains free of discernible bias;
- Third-party AI vendors be vetted before engagement and bound by contract terms giving the insurer audit rights and requiring cooperation with regulatory inquiries; and
- The company maintain documentation—the AIS Program itself, model inventories, validation records, vendor due diligence files—sufficient to produce on a regulator's request, since state insurance departments are now examining for exactly this.
Every company using AI, inside or outside the insurance industry, should take a version of these steps: form an internal task force responsible for AI compliance; adopt a written AI use policy; audit the data going into the AI tool for discriminatory content; test the AI's output for bias before and after deployment; and, above all, make certain that final decisions affecting hiring, firing, claims handling, coverage, and underwriting are made, and can be shown to have been made, by a trained human being who understands that AI, like the rest of us, comes with bias built in.