AI in Healthcare Raises Health Equity, Algorithmic Bias and Disability Risks

Artificial intelligence systems can improve diagnostic accuracy and healthcare efficiency without necessarily reducing disparities between patient groups, according to research examining algorithmic bias, clinical decision-making and the treatment of people with disabilities.

Georgetown University found that only 22 of 166 peer-reviewed articles addressing AI and health equity provided an explicit definition of health equity. Researchers also identified limited attention to individual medical conditions and a substantial concentration of research in the United States.

The findings come as healthcare providers introduce increasingly sophisticated AI systems for diagnosis, patient management and treatment decisions. Although these technologies may expand access to medical services, their performance depends on the data used for training, the outcomes they are designed to predict and how clinicians apply their recommendations.

Key highlights

• Only 22 of 166 peer-reviewed articles on AI and health equity explicitly defined health equity, according to Georgetown University researchers, while 70% had US-based first authors.
• Research shows that higher AI diagnostic accuracy does not necessarily reduce healthcare disparities, particularly when algorithms rely on biased data or clinicians respond differently to recommendations.
• Autonomous medical AI and predictive algorithms raise additional risks for patients with disabilities, prompting calls for representative datasets, human oversight and stronger equity-focused governance.

For patients with disabilities and other underrepresented populations, these factors can determine whether AI improves access to care or reinforces existing inequalities (see How is Artificial Intelligence Modernizing the HealthCare Industry?).

Most AI healthcare research lacks a clear definition of health equity

Georgetown University researchers examined 166 articles published between 2018 and early 2025, including commentaries, editorials, literature reviews and other publications providing guidance on equitable AI use in healthcare delivery, health systems and health services research.

The team searched four academic databases in January 2025 and found that just 22 articles explicitly defined health equity.

Most of those definitions followed the World Health Organization’s description of health equity as the absence of unfair, avoidable or remediable differences between population groups. Fifteen of the 22 articles offering definitions were written by US-based author teams.

When health equity is vaguely defined, there is risk that resources will be primarily allocated to the most advantaged group, rather than the group who needs the resources the most.

Caroline R. Efird, Ph.D., lead author and assistant professor in Georgetown University’s Department of Health Management and Policy

The review also identified substantial geographic differences in the research.

First authors affiliated with US institutions accounted for 116 articles, or 70% of the total. The United Kingdom contributed 10 articles and Canada nine. No first authors were based at institutions in South America, while Africa and Australia each accounted for one publication.

AI and health equity research: key findings

Research indicator	Findings
Articles reviewed	166
Publication period	2018-2025
Articles explicitly defining health equity	22 (13.3%)
Articles without a disease-specific focus	68%
Commentaries, editorials and opinion pieces	64%
Articles with US-based first authors	116 (70%)

Most publications also addressed health equity broadly rather than examining particular diseases or clinical specialties.

Some 68% discussed AI and health equity without focusing on a specific medical condition. Cardiovascular disease, chronic diseases and dermatological conditions, including skin cancer, each represented 5% of the articles (see about Cancer Health Insurance).

COVID-19 accounted for 3%, while ageing, diabetes and radiology each accounted for 2%.

The researchers argued that more condition-specific expertise is needed because AI can reproduce disparities already present in medical evidence.

Dermatology provides one example. Clinical textbooks and public image databases have historically included more examples of skin lesions on lighter skin tones. AI models trained primarily on those images may have greater difficulty recognising conditions on darker skin.

Specialty-specific expertise (e.g., radiology, dermatology, cardiology) is crucial for reducing the risk of worsening health disparities when AI is used in healthcare delivery.

She noted that medical specialists may be better positioned than data scientists to recognise diagnostic problems arising from training datasets that do not adequately represent diverse patient populations (Raising Deductibles vs. Employer-Sponsored Reimbursements: Weighing the Pros and Cons).

The authors recommended involving radiologists, oncologists, gynaecologists, neurologists and other clinicians alongside programmers and health equity specialists throughout model design, development, implementation and evaluation.

Higher AI accuracy does not always reduce healthcare inequalities

Evidence from existing medical algorithms demonstrates why overall predictive accuracy is an insufficient measure of fairness.

A 2019 study published in Science examined a widely used US healthcare algorithm intended to identify patients who would benefit from additional medical support.

Researchers found that Black patients were substantially sicker than White patients assigned the same risk score.

The problem originated partly in the outcome the algorithm was trained to predict. Rather than directly measuring illness or unmet healthcare needs, the system predicted medical expenditure (Global Employer Medical Plan Costs Will Climb Worldwide). Because healthcare spending can reflect unequal access to treatment, predicting costs did not reliably identify which patients required additional care.

The algorithm could therefore perform well against its selected target while producing unequal results for patients. This distinction also matters for European healthcare systems considering AI for patient outreach and resource allocation.

Historical appointment records, diagnoses and service utilisation may appear to provide useful information about patients’ needs. Yet limited healthcare activity can indicate either good health or barriers preventing someone from accessing treatment.

An algorithm trained on those records may consequently overlook patients whose unmet needs are poorly represented in existing data.

Healthcare organisations need to establish whether the prediction target accurately represents the clinical or operational decision the model is intended to support.

Medical specialties and conditions covered in AI equity research

Clinical focus	Share of articles
General healthcare, without a specific condition	68%
Cardiovascular conditions	5%
Chronic diseases	5%
Dermatology and skin cancer	5%
COVID-19	3%
Ageing	2%
Diabetes	2%
Radiology	2%

Patients and frontline healthcare professionals can contribute to this assessment by identifying circumstances that administrative records or conventional performance measures fail to capture.

Their participation can influence the model’s objectives before developers begin evaluating accuracy.

Digital health companies are adopting AI faster than insurance programs are adapting to the resulting exposures, according to Beazley’s 2026 Digital Health & Wellness report. The study surveyed 600 executives across Europe, North America and Asia, while its claims analysis draws on roughly a decade of Beazley’s own healthcare loss data.

Beazley writes digital health insurance and therefore has a commercial interest in the market examined by the report. Its longer claims history nevertheless provides a useful comparison between the risks executives worry about and the losses appearing most often in practice.

Medical negligence ranks as both the most frequent and most severe cause of loss in Beazley’s healthcare claims book. Improper supervision ranks second by frequency, even though executives surveyed were more focused on cyberattacks and workforce competency.

Clinical use of AI can widen diagnostic disparities

Even when AI provides useful diagnostic assistance, the interaction between clinicians and algorithms can produce unequal results.

A 2024 Nature Medicine experiment involving 389 dermatologists and 459 primary-care physicians from 39 countries examined diagnostic decisions in a simulated teledermatology setting.

Participants assessed medical images with different forms of AI decision support.

Although assistance improved diagnostic accuracy overall, the experiment found that it widened the accuracy gap between light and dark skin among generalist physicians.

The findings demonstrated that higher average performance did not necessarily translate into more equitable diagnosis.

The experiment used images and limited clinical context rather than measuring patient outcomes during routine care. Its design also varied the decision support provided to clinicians to examine how they responded to algorithmic recommendations.

The research therefore showed the importance of assessing combined human and AI performance instead of relying exclusively on model-level accuracy.

Hospitals evaluating AI tools could compare clinical decisions before and after algorithmic assistance, examining whether physicians accept, modify or reject recommendations differently across patient groups.

Differences could arise from the information displayed, interface design or clinicians’ familiarity with particular clinical presentations.

Evaluation also needs to account for the consequences of different errors. A missed diagnosis, unnecessary referral and delayed treatment decision may have substantially different effects on patients.

Evidence of algorithmic bias in healthcare AI

Research or investigation	Key finding	Health equity concern
Science study, 2019	Black patients were sicker than White patients at the same algorithmic risk score	Healthcare spending used as a proxy for medical need
Nature Medicine study, 2024	AI improved overall diagnostic accuracy but widened the light-versus-dark skin accuracy gap among generalists	Unequal diagnostic performance
Georgetown University review, 2026	Only 22 of 166 articles explicitly defined health equity	Inconsistent standards for evaluating fairness
STAT’s Denied by AI investigation	Algorithmic predictions contributed to decisions cutting off rehabilitation coverage	Potential loss of medically necessary care for older and disabled patients

Disability representation creates additional risks for medical AI

The growing use of artificial intelligence in healthcare raises particular concerns for people with disabilities, whose characteristics may be poorly represented in training data.

AI systems often rely on patterns identified across large datasets. When those datasets primarily represent people without disabilities, systems may perform less accurately for individuals whose speech, movement, eye behaviour, communication or health information differs from the assumed norm.

Such differences can affect biometric identification, healthcare assessments and other automated decisions.

Training datasets may also contain inaccurate classifications of disability-related information or reflect discriminatory assumptions already present in healthcare records.

Research has identified potential disability-related bias in clinical notes and other medical documentation. Models trained on these sources may reproduce or reinforce existing assumptions even when their outputs appear neutral.

Patients with disabilities who also belong to other marginalised groups may face overlapping forms of discrimination.

Those risks can be particularly relevant for women, Indigenous people, LGBTQI+ individuals and younger patients with disabilities, whose experiences may be inadequately represented across several dimensions of a dataset.

Accessibility therefore extends beyond whether patients can operate a particular AI interface. It also concerns whether automated assessments accurately recognise their circumstances and whether recommendations produce equitable clinical outcomes.

AI-driven healthcare decisions have already raised patient safety concerns

Concerns about algorithmic discrimination are supported by documented cases involving automated healthcare decisions.

STAT’s Denied by AI investigation examined Medicare Advantage patients whose rehabilitation coverage was discontinued based on algorithmic predictions, despite evidence that they still required treatment.

The investigation also reported that UnitedHealth pressured medical staff to follow predictions about rehabilitation stays and terminate payments for seriously ill patients, including older Americans and people with disabilities.

These cases involved predictive algorithms rather than newer autonomous AI agents, but they demonstrated how automated recommendations can influence access to medically necessary care.

Similar questions are emerging as healthcare organisations and federal agencies consider deploying more advanced systems.

US federal health agencies have been accelerating pilot programmes involving autonomous AI agents designed to diagnose conditions and prescribe treatment.

More than 1,600 AI-enabled medical devices have received FDA approval or authorisation for specific applications, including identifying skin cancer from medical images and estimating patients’ risk of heart attacks.

The regulatory approach to foundation models and agentic AI, however, remains less established as these technologies begin performing tasks traditionally reserved for physicians.

Dr. John Whyte, chief executive of the American Medical Association, warned that autonomous medical AI models are “not ready for prime time”, noting that much of the available evidence comes from simulations.

The FDA has also sought public feedback to inform its approach to regulating generative AI-enabled medical devices.

Concerns surrounding advanced AI include unpredictable autonomous behaviour, patient safety, data privacy and the possibility that disadvantaged patients could receive lower-quality automated services.

For people with disabilities, expanding access through autonomous systems without first addressing accessibility and algorithmic bias could create separate care pathways that provide less effective treatment.

The question is therefore whether systems are designed and evaluated against the needs of diverse patient populations before they are used in decisions affecting health and access to services.

Healthcare AI governance differs between Europe and the US

Regulatory approaches to medical AI have developed differently across jurisdictions.

The European Union’s AI Act, adopted in 2024, establishes requirements for high-risk AI applications, including medical uses covered by the legislation.

These include risk-mitigation measures, appropriate data quality, information for users and human oversight.

Georgetown researchers contrasted the European framework with US policy developments since early 2025, which they characterised as placing greater emphasis on deregulation.

They cited President Donald Trump’s January 2025 executive order revoking the Biden administration’s 2023 framework on safe, secure and trustworthy artificial intelligence (see How Will the EU AI Act Affect the Insurance Industry?).

Separate US healthcare regulations also address discriminatory risks associated with automated decision-making.

Under a 2024 rule implementing Section 1557 of the Patient Protection and Affordable Care Act, covered healthcare entities were required to make reasonable efforts to identify patient care decision support tools using race, colour, national origin, sex, age or disability as input variables.

The rule also required covered entities to mitigate discrimination risks associated with those tools.

Those provisions took effect on May 1, 2025, and the relevant regulatory text remained unchanged in the federal government’s electronic code as of October 23, 2026. The Georgetown review did not examine that rule.

Legal responsibility for harm involving AI-assisted medical decisions also remains unsettled.

Human oversight and specialist involvement remain essential

Across the Georgetown review, publications generally recommended that AI support clinical judgment rather than replace healthcare professionals.

The reviewed articles emphasised direct interaction between patients and clinicians, particularly where diagnoses and treatment decisions require contextual understanding, professional judgment and empathy.

Several authors warned against reducing medical practice to algorithmic recommendations that cannot account for all the circumstances surrounding individual patients.

Healthcare professionals using AI should therefore receive training in its underlying principles, capabilities and limitations.

The review also called for health systems to develop institutional policies that prioritise preventing harm from AI applications.

In the US, where health outcomes can vary by race and gender, the authors discussed the use of race-conscious and gender-conscious algorithms as potential ways to address existing disparities.

They also recommended interdisciplinary development teams combining technical expertise with clinical knowledge and patient participation.

AI could improve access to healthcare for underserved patients

Despite the documented risks, AI technologies may provide benefits for patients who encounter barriers to conventional healthcare services.

A survey of patients, patient advocates and caregivers identified several potential advantages, including improved disease screening, fewer human errors, reduced bias, more accurate diagnoses and better prediction of future health problems.

Respondents also identified personalised treatment and improved healthcare accessibility as possible benefits.

AI-enabled services could help address long waiting times and treatment costs, particularly where specialist resources are limited.

Examples include Limbic, which provides technology supporting mental healthcare, and Curai Health, which offers AI-assisted primary care services.

Such applications may expand access to services for underserved populations, including patients with disabilities.

Their benefits, however, depend on whether the technology is accessible, appropriate for individual patients and supported by sufficient evidence.

Greater availability of automated services does not necessarily establish that the resulting care is equitable.

Health systems also need to examine whether particular groups receive less effective assessments, experience additional barriers or have fewer opportunities to challenge automated recommendations.

Researchers call for stronger health equity standards in AI development

The Georgetown authors recommended that research funders and academic journals require AI studies to define health equity explicitly.

They also urged researchers to report results for population subgroups and marginalised communities rather than relying exclusively on aggregated performance measures.

Such reporting could help identify circumstances in which average improvements conceal deteriorating outcomes for particular patients.

The authors recommended that healthcare organisations establish equity-focused AI governance programmes with continuous oversight and that national regulators incorporate frameworks based on the WHO definition of health equity into their evaluations.

For people with disabilities, researchers and policy specialists have also proposed incorporating accessibility throughout the AI development lifecycle.

Greater transparency about how AI is used in healthcare decisions and patient communications is another priority, particularly for applications affecting privacy, safety or legal rights.

Patient involvement during development may help identify problems that would otherwise become apparent only after deployment.

The evidence from predictive algorithms, clinical decision-support experiments and disability research indicates that healthcare AI requires evaluation beyond aggregate technical accuracy. Its effects also depend on the outcomes selected for prediction, the populations represented in training data and the clinical decisions made using its recommendations.

………………..

EXPERTS: Caroline R. Efird, Ph.D. – lead author and assistant professor in Georgetown University’s Department of Health Management and Policy, Dr. John Whyte – chief executive of the American Medical Association

by Nataly Kramer – Lead Insurance Editor at Beinsure, Peter Sonner – Lead Tech Editor at Beinsure