Modern surgery faces challenges due to the anatomical complexity of organs, especially in areas such as the liver, where the intricate vascular network makes every movement critical. Surgical predictability, ensured by detailed mapping of hepatic segments, aims to avoid vascular damage. Technological advances have turned preoperative 3D reconstructions into a powerful tool, allowing surgeons to anticipate operating room conditions. Currently, augmented reality (AR) and virtual reality (VR) offer innovative solutions to bridge the gap between planning and execution, enhancing the surgeon’s understanding and contextualization of anatomical structures directly in the field of view.
Medical preceptorship, which involves supervised practice integrated into resident training, stands out as a crucial element for developing clinical and surgical skills, aligning real-world experience with practical feedback. The use of physical or digital 3D technologies adds a new layer to medical education by enabling the visualization of scans, offering a deeper understanding of anatomical structures and their spatial relationships.
The integration of traditional preceptorship with VR environments presents complementary aspects. In the in-person context, direct supervision allows for detailed observation of the procedure and immediate feedback. Under this model, residents watch, perform under supervision, and subsequently teach, fostering the development of progressive skills. In the virtual environment, the immersive experience provides a safe space for practice, allowing residents to make mistakes without patient risk and repeat procedures multiple times. This overcomes real-world limitations such as time and resource availability. Thus, combining these approaches optimizes learning and delivers more comprehensive, effective training.
In the first preoperative experiment, the liver was reconstructed in 3D, with all eight hepatic segments and their vascularization clearly identified to facilitate surgical planning. During the activity, the preceptor supervised at the computer while the residents manipulated the 3D model in VR, presenting their approaches and justifications. The feedback provided by the surgeon highlighted the residents’ successes and mistakes, promoting the improvement of their skills.
In the subsequent experiment, featuring a liver surgery simulator, the residents were first introduced to the controls. They then performed the procedure on the 3D liver model, facing anatomical challenges. Afterwards, they practiced techniques individually, such as using the electrocautery and contouring the vasculature. However, the lack of abdominal tension simulation impaired realism. Even so, the metaverse’s social aspect for case discussion proved valuable for introducing future procedures, though the practical simulation failed to faithfully reproduce the surgical experience.
Moving forward, we explored an experiment integrating these innovations directly into the intraoperative setting, allowing the use of augmented reality (AR) during surgical procedures. The project was made possible by available technology: the HoloLens 2, launched in 2019, represented a major advance in AR devices. The COVID-19 pandemic, which began in 2020, accelerated the adoption of remote technologies. Meanwhile, Facebook—renamed Meta—launched the Quest 2 in 2021, and Microsoft introduced Microsoft Mesh, bolstering immersive digital interaction.
Three types of visualizations were developed to solve the problem of non-occlusion of the user’s hands in the headset, which could hinder the procedure. One version had a transparent parenchyma and vessels; another displayed a wireframe of the parenchyma, allowing for the visualization of hands alongside the virtual overlay; and a third showed only vascular structures, used when anchoring the virtual object. A second operator switched among these views as needed. The device was used mainly at the beginning of the procedure, after opening and exposing the organs, to identify the relevant vasculature. The option to raise and lower the display made it easy to check the HoloLens content, allowing the surgeon to operate with the visor up for most of the time.
For future projects addressing this same topic, it may be important to add other functionalities, such as the ability to view and interact with medical imaging in augmented reality. Additionally, remote interaction between physicians—enabling real-time collaboration through annotations and overlays on the surgeon’s view—could be beneficial. These advances will help increase procedural accuracy and optimize teamwork, ultimately improving surgical outcomes.
