| Abstract: |
Collaborative AI systems, which combine both forms of intelligence (i.e., human and machine), are attracting increasing interest from the scientific and medical communities, with various applications in radiology (clinical decision support systems) and surgery (robot-assisted surgery). However, despite their promise, these systems face significant challenges in integrating into clinical practice due to a lack of transparency, trust, and clinical validation. Drawing on the case of robotic surgery, the aim of this work was to analyse the scientific evidence for ten surgical robots currently on the market (i.e., CE-marked or FDA-cleared/approved) that meet the definition of a collaborative AI system. We found a low number of peer-reviewed publications and a lack of transparency from authors and manufacturers, particularly regarding the functioning of their devices, which are often considered as ‘black boxes’. Furthermore, the term ‘artificial intelligence’ is under-utilised in scientific publications, regulatory submissions, and commercial materials. Based on these findings, we propose three recommendations to promote the integration of these medical devices: 1) promote the transparency, explainability, and comprehensibility of AI devices by encouraging manufacturers to provide more detailed information about their systems and their functioning, including the interrelationship with the user; 2) promote randomised controlled multicentre trials to provide stronger evidence on the performance and safety of these devices; 3) encourage the publication of scientific results in peer-reviewed journals to expose them to scientific scrutiny and improve transparency. These recommendations have been carefully formulated to cover a wide range of AI/ML-enabled medical devices, beyond the case of surgical robots reviewed here. |