How Duke Operationalizes Responsible AI in Healthcare

Artificial intelligence (AI) is revolutionizing healthcare, unlocking new possibilities to enhance patient care, improve diagnostics, and streamline workflows. However, with this immense potential comes significant challenges, particularly in ensuring fairness, transparency, and safety. This requires a comprehensive approach to ethical AI in clinical decisions. AI systems must not only function effectively but also align with ethical principles, a necessity that becomes even more pronounced in life-critical fields like healthcare.

In this article, we explore Duke Health’s groundbreaking efforts to operationalize responsible AI in healthcare. Guided by Dr. Nicoleta Economou, director of Duke Health’s AI Evaluation and Governance Program, we uncover how Duke is translating abstract principles into practical systems to ensure AI’s safe, fair, and effective integration into clinical workflows. These insights have far-reaching implications for organizations navigating the complexities of integrating AI into clinical care and other high-stakes environments.

Duke Health’s Framework for Responsible AI: The ABCDS Program

The centerpiece of Duke Health’s AI governance efforts is the Algorithm-Based Clinical Decision Support Oversight Program (ABCDS), established in 2021. This framework was designed to address the lack of finalized regulations for AI in healthcare at the time, ensuring that any algorithm deployed in patient care is robustly evaluated and closely monitored throughout its lifecycle.

Key Goals of the ABCDS Program:

1. Evaluation and Monitoring: AI algorithms are assessed for safety, fairness, and utility before and after deployment. Continuous monitoring ensures performance consistency and relevance.
2. Lifecycle Commitment: From registration to post-deployment management, the ABCDS program governs algorithms at every stage of their lifecycle.
3. Transparency and Accountability: End-user education and clear documentation are critical components, empowering clinicians to understand and trust the AI tools they use.

This structured approach has enabled Duke Health to proactively address challenges such as algorithmic bias, data drift, and a lack of standardized processes, laying the foundation for a coordinated, ethical deployment of AI in healthcare.

sbb-itb-f5765c6

A Multi-Stakeholder Approach to Governance

One of the most significant hurdles in AI adoption is the fragmented understanding of AI ethics and governance across stakeholders. Clinicians, data scientists, informaticists, and ethicists often operate with different perspectives and terminologies, creating gaps in communication and implementation.

Duke Health’s governance model addresses this by establishing a multidisciplinary review committee, ensuring that diverse subject-matter expertise is applied to every phase of AI evaluation. This collaborative structure mirrors the complexity of developing and deploying AI systems, where technical, clinical, and ethical considerations must align.

Operational Rigor and Continuous Monitoring

Dr. Economou highlights the importance of operational rigor in ensuring responsible AI deployment:

• Risk-Based Evaluation: Algorithms are evaluated based on the risk they pose, with higher-risk systems subjected to more stringent validation and monitoring. This is particularly critical for real-time health risk alerts that require immediate clinical action.
• Post-Deployment Monitoring: AI tools are tracked and reevaluated periodically to ensure they remain effective and safe even as data evolves.
• Transparent Documentation: A focus on clear and accessible documentation builds trust and ensures compliance across teams.

Broadening Impact: The Role of Industry Coalitions

Recognizing that responsible AI requires collective effort, Duke Health has played a leading role in two major coalitions: the Coalition for Health AI (CHAI) and the Trustworthy and Responsible AI Network (TRAIN).

CHAI: Defining National Standards

CHAI’s mission is to establish national guidance for trustworthy AI through consensus-driven methods. It brings together academia, industry, government, and patient advocates to align on what responsible AI should look like and how it can be achieved.

TRAIN: Enabling Practical Implementation

Where CHAI defines the principles, TRAIN focuses on the "how." This coalition provides a collaborative space for healthcare organizations to share tools, frameworks, and best practices for operationalizing AI governance.

Dr. Economou explains how these partnerships are complementary: "CHI defines what trustworthy AI should look like, while TRAIN delivers the practical solutions to implement those principles."

SAGE: Automating Governance for Wider Adoption

To scale its governance frameworks for other organizations, Duke Health collaborated with Microsoft and Avanade to develop the Smart AI Governance Engine (SAGE). Built on Microsoft’s Azure platform, SAGE streamlines the registration, evaluation, and monitoring processes for AI tools, making it easier for healthcare organizations to adopt responsible AI practices.

Responsible AI in Action: Principles and Practice

According to Dr. Economou, responsible AI is built on core ethical and quality principles, including:

• Safety: Ensuring that algorithms produce accurate, reliable, and clinically useful results.
• Effectiveness: Verifying that AI tools achieve their intended purpose without causing harm or unnecessary burden.
• Fairness: Addressing potential biases in data and algorithms to prevent unequal outcomes across different patient populations.
• Transparency: Clearly documenting how algorithms function, how decisions are made, and how risks are mitigated.

Challenges of Generative AI in Healthcare

The rise of generative AI, such as large language models (LLMs), introduces new opportunities and risks. Duke Health is optimistic about its potential, particularly for applications like ambient scribing, which reduces clinician burnout by automating medical documentation. However, rigorous governance remains critical:

• Human-in-the-Loop Validation: Ensuring that clinicians can review and verify AI-generated content.
• Defined Use Cases: Not all AI methodologies are suitable for every clinical application. Each use case must be carefully evaluated for risks and benefits.
• Monitoring for Drift: Algorithms must be continuously monitored to detect and address performance drift over time.

Implications for Insurance and Health System Decisions

AI’s expanding role in healthcare extends beyond clinical applications to areas like insurance coverage and reimbursement decisions. Here, transparency and accountability are especially critical. Insurers and health systems must ensure that AI-based decisions are:

• Transparent: Patients and clinicians should understand when AI is being used and how decisions are made.
• Fair: AI models must be tested to avoid disadvantaging certain populations.
• Safe: Continuous monitoring is essential to maintain trust and ensure patient safety.

Duke Health’s governance model offers a roadmap for achieving these goals, emphasizing the importance of aligning AI deployment with ethical principles and cost considerations.

Key Takeaways

• Lifecycle Governance is Essential: AI in healthcare requires robust evaluation and oversight from registration through post-deployment monitoring.
• Multidisciplinary Collaboration Works: Successful AI governance depends on input from diverse stakeholders, including clinicians, data scientists, and ethicists.
• Operationalizing Governance is Key: Abstract principles like fairness and transparency must translate into practical processes that organizations can implement.
• Coalitions Drive Progress: Partnerships like CHAI and TRAIN bring academia, industry, and government together to define and implement responsible AI standards.
• Generative AI Shows Promise: While tools like ambient scribing can reduce clinician burnout, they require rigorous validation and monitoring to ensure safety and effectiveness.
• Monitor for Drift: Algorithms evolve over time, and continuous monitoring is critical to maintaining performance, fairness, and safety.
• Transparency Builds Trust: Clear documentation and accountability are crucial for earning the trust of patients, clinicians, and regulators alike.

Conclusion

Duke Health’s efforts to operationalize responsible AI demonstrate how principled governance can drive ethical and impactful AI adoption in healthcare. Their frameworks, coalitions, and tools offer a scalable model for institutions navigating the complexities of introducing AI into sensitive environments.

By prioritizing safety, transparency, and fairness, Duke Health is not only addressing today’s challenges but also shaping the future of trustworthy AI in healthcare. For organizations seeking to integrate AI responsibly, the lessons from Duke serve as both inspiration and a practical guide to success.

Source: "Operationalizing Responsible AI in Healthcare: Lessons from Duke Health" - TrustedAI, YouTube, Sep 2, 2025 - https://www.youtube.com/watch?v=4GDaVBmjTTY