Samenvatting

The workload in the healthcare sector is increasing, requiring the need for innovative solutions. One such
solution is for AI to assist in clinical decision-making by extracting information from patient’s records. To ensure
healthcare professionals stay in the lead, the reasoning of the AI should be transparent, creating the need for
explainable AI (XAI). As this XAI representation should fit the users’ needs and workflows, the user needs to
be included in the design process. This research focuses on a case study using the Design Thinking method
for generating an XAI representation for predicting the start of the palliative phase in patients with chronic
obstructive pulmonary disease (COPD) or heart failure. This paper presents knowledge about and experiences
with the design practices used, focusing on the ideation, prototype, and test phases. This contributes to the
understanding of the needed design process to design XAI representations in the healthcare sector