On his first day in office, President Trump revoked former President Biden’s Executive Order 14110. Issued October 30, 2023, this now-revoked EO sought to promote certain principles regarding the development, use, safety and security of AI.
In its place, President Trump issued EO 14179. Titled Removing Barriers to American Leadership in Artificial Intelligence, this EO seeks to broadly deregulate AI at the federal level and shift the policy of the United States to “global AI dominance.” This shift in federal policy may lead to more activity at the state level to regulate the development and use of AI.
Ensemble is tracking the state legislative sessions of our full outsource partners across the country. Below are some state bills that we’ve identified so far this year seeking to regulate the use of AI in healthcare.
As it relates to use of AI by insurers, legislative trends include prohibitions against use of AI without any human oversight, particularly with respect to determinations of medical necessity, and a focus on the prevention of patient discrimination.
Connecticut
Legislators in the Connecticut House have proposed two house bills which would restrict health insurers’ use of artificial intelligence in the denial of health insurance claims:
- Proposed H.B. No. 5587 seeks to prohibit AI’s use as the primary method of denial, while
- Proposed H.B. No. 5590 seeks to prohibit its use more generally with respect to denial of health insurance claims.
Connecticut Senators have proposed similar bills:
- Proposed S.B. No. 817 would prohibit health insurers from using AI, or a similar tool or algorithm, to automatically downcode or deny a health insurance claim submitted by a healthcare provider without detailed review by a clinical peer.
- Proposed S.B. No. 447 would prohibit health carriers from using AI in the evaluation and determination of patient care to safeguard patient access to testing, medications and procedures.
Lastly:
- S.B. No. 2 seeks to broadly protect consumers from the risk of algorithmic discrimination and unfair treatment posed by AI.
The most recent version of this bill (S.B. No. 2) indicates its requirements would apply when AI is used to make, or is a substantial factor in making, a “consequential decision” which is defined as including any decision which has a significant effect on the provision or denial of healthcare services to a consumer.
In such cases when the consequential decision is adverse, there must be disclosures to the consumer and opportunities to examine the personal data used by AI to make the decision and correct it as needed. There may also be an opportunity to appeal where that personal data was inaccurate.
As currently written, this law would exempt covered entities to the extent that they are providing healthcare recommendations that (A) are generated by AI, (B) require a healthcare provider to take action to implement such recommendations, and (C) are not considered to be high risk.
Florida
In Florida, one related bill was introduced recently in March:
- SB 794 would require insurers generally to ensure that any decision to deny a claim is reviewed, approved and signed off by a “qualified human professional.” It would prohibit the use of AI, an algorithm or an automated system as the basis for determining whether to deny a claim. The qualified human professional would be required to analyze the facts of the claim and the terms of the insurance policy independently of AI and to review the accuracy of any output generated by AI.
Insurers would be required to keep documentation of the basis for the denial and to identify the qualified human professional who reviewed the denial decision in all communications to the claimant.
Illinois
Illinois has the Artificial Intelligence Systems Use in Health Insurance Act (SB1425):
- SB1425 proposes providing for regulatory and enforcement oversight of health insurers’ use of AI systems when those systems are used to make or support adverse determinations.
Regulatory review would include, for example, investigations and market conduct actions regarding health insurers’ development, implementation, and use of AI systems. This Act would also prohibit health insurers from using AI exclusively when it comes to denying or reducing benefits. Where AI is used in any decision-making process for that adverse determination, the Act would require meaningful review by an individual with authority to override the AI system’s determinations. Lastly, this Act would require health insurers to disclose their AI system utilization.
On the provider side in Illinois:
- SB2259 would amend the Medical Practice Act to require hospitals and providers that use generative AI to generate written or verbal patient communications pertaining to patient clinical information adhere to certain criteria and that such communications be read and reviewed by a licensed provider.
Indiana
In Indiana:
- HB 1620 proposes requiring both healthcare providers and insurers to disclose the use of AI technology to the patient or insured
(1) when it’s used to make or inform any decision involving the patient or insured, or
(2) when it’s used to generate any part of a communication to the patient or insured.
Tennessee
In Tennessee:
- SB 1261/HB1382 proposes requiring health insurance issuers using AI (or algorithm or other software tool) for utilization review or management ensure that AI bases its determinations on:
- An enrollee’s medical or clinical history
- Individual clinical circumstances presented by the requesting provider
- Other relevant clinical information in the enrollee’s medical record
Health insurance issuers would be required to ensure that the AI does not supplant the healthcare provider’s decision-making, discriminate or directly or indirectly cause harm to the patient. It also proposes prohibiting AI from denying, delaying or modifying healthcare services based on medical necessity.
Instead, such determinations may only be made by a licensed physician or a licensed healthcare professional competent to evaluate the patient’s specific clinical issues involved and the services requested by the patient’s provider.
New Mexico
In New Mexico:
- HB 60, the Artificial Intelligence Act, seeks to require AI developers to protect consumers from algorithmic discrimination. Deployers, or users, of AI systems making a “consequential decision” (defined as including any decision affecting the provision or denial of healthcare services) would be required to implement policies and programs to protect consumers from foreseeable risks of algorithmic discrimination.
In addition to providing consumers with general notice of their AI use, deployers would also be required to provide direct notice and opportunity for appeal to consumers when AI is used to making adverse consequential decisions.
Taken together, this bill may impose new requirements on health insurers using AI in their decision-making concerning the provision or denial of healthcare services to their members.
Texas
The Texas Legislature meets every two years, so there is a bit more activity in this state.
- HB 1709, The Texas Responsible Artificial Intelligence Governance Act, seeks to comprehensively and broadly regulate the development and use of AI with respect to “consequential decisions,” defined as including those decisions involving a healthcare service or treatment.
- SB 815/HB 2922 would prohibit utilization review agents from using automated decision systems (defined as an algorithm that may be incorporating AI) to make, wholly or partly, adverse determinations. The House bill prohibits the sole use of automated decision systems and would further require that determinations about medical necessity or the appropriateness of healthcare services be made by a physician or licensed healthcare provider.
- SB 1822/HB 4018 would require health insurers and health maintenance organizations to disclose to its insureds and contracted providers whether it uses AI in conducting utilization review. It would also require that any AI used minimizes the risk of bias and complies with evidence-based clinical guidelines.
- SB 1411 would require health benefit plan issuers to disclose its use of AI in utilization review to members and contracted healthcare providers. This bill would also require submission of AI-based algorithm and training data sets used in utilization review to the Texas Commissioner for Insurance, and that a healthcare provider be involved in the utilization review that uses AI. This bill would also establish certain oversight and enforcement authority of its requirements.
The bottom line
Many of these proposed state regulations aim to prohibit the use of AI without any human oversight. This is a safeguard we support to ensure the ethical, effective application of AI in healthcare.
We believe AI with human oversight has the power to eliminate administrative barriers standing between patients and providers. Ensemble’s approach to AI aims to:
- Reduce the time and energy spent on administrative tasks by clinicians and staff to mitigate burnout and be more responsive to patient needs and enable access to care. This includes automating routine and manual tasks and providing AI-powered intelligence to prioritize and provide next best action recommendations. This enables providers to tackle more complex tasks in areas including insurance collections and appeals.
- Reduce denials, expedite patient access to care and accelerate accurate reimbursement by identifying billing issues, preventing denials and automating the appeals process.
- Improve the patient experience by reducing call wait times and resolve patient needs faster by transcribing calls so human operators can focus on addressing patient needs instead of taking notes.
Our experts will continue to track proposed legislation that impacts the use of artificial intelligence in the healthcare sector. We are committed to sharing this information — as well as information around our continued efforts to put AI to use for the benefit of providers.
These materials are for general informational purposes only. These materials do not, and are not intended to, constitute legal or compliance advice, and you should not act or refrain from acting based on any information provided in these materials. Neither Ensemble Health Partners, nor any of its employees, are your lawyers. Please consult with your own legal counsel or compliance professional regarding specific legal or compliance questions you have.
