Doctor Michelle Mello highlights AI governance challenges in California healthcare

The Assembly Joint Hearing of the Health Committee and the Privacy and Consumer Protection Committee convened on May 28, 2025, to discuss the integration of artificial intelligence (AI) in healthcare and the regulatory frameworks necessary for its safe and effective use. The meeting featured expert testimonies highlighting the current state of AI in healthcare, the challenges faced, and potential pathways forward.

The session began with a presentation emphasizing the need for increased investment in AI technologies within the healthcare sector. Experts noted that private sector investment is hindered by uncertainty regarding reimbursement from insurers. This uncertainty limits the adoption of advanced AI tools that could enhance patient care, such as algorithms capable of early cancer detection. The speaker proposed that if Medi-Cal could establish clear payment structures for these technologies, it would incentivize their development and use.

Following this, Dr. Michelle Mello from Stanford University discussed the governance of AI tools in healthcare. She outlined three key messages regarding the role of state regulation. First, while healthcare organizations are interested in experimenting with AI, they lack sufficient incentives to establish robust governance systems. Second, despite the promising capabilities of AI models, ongoing monitoring is essential to ensure their safe use, which presents significant challenges. Lastly, she advocated for a regulatory framework that mandates healthcare facilities to implement governance structures as a condition of licensure.

Dr. Mello further elaborated on the current landscape, noting that many healthcare organizations do not adequately vet or monitor AI tools, leading to a lack of information on their safe and equitable use. The costs associated with establishing governance committees and monitoring systems deter many organizations from pursuing these necessary steps. She highlighted that while AI is often perceived as a cost-saving measure, it frequently results in increased expenses due to the need for oversight.

The discussion also touched on the liability issues surrounding AI development. Dr. Mello pointed out that developers often include disclaimers in their terms of service, limiting their financial responsibility for errors. This creates a situation where the liability for AI-related issues typically falls on healthcare providers rather than developers, complicating the incentive structure for ensuring safety and accountability.

In conclusion, the meeting underscored the critical need for a collaborative approach between government, healthcare organizations, and researchers to foster the responsible development and implementation of AI in healthcare. The discussions highlighted the importance of establishing clear regulatory frameworks and financial incentives to support the safe use of AI technologies, ensuring that they can effectively enhance patient care while addressing equity concerns. The committees plan to continue exploring these issues in future sessions, aiming to create a supportive environment for innovation in healthcare technology.