Augmented reality in orthopedics

The digital revolution has long since arrived in medicine. Various computer-assisted procedures have also been used in orthopedic surgery for some time. An emerging one of these techniques is the so-called augmented reality, whose history dates back to 1968 when Sutherland first described it [1]. Augmented reality is characterized by a computer-assisted extension of reality perception, which specifically means that the real environment is extended for the user by virtual information. This contrasts with virtual reality, in which the user is exposed to a completely virtual environment. These technologies are mainly applied via a monitor-based system or using so-called data glasses. The latter are becoming increasingly popular, justified by the fact that the use of data glasses offers a high degree of flexibility, the user can move freely in the room and always keeps the virtual information in view. The increasing commercialization of available data glasses such as Google Glass (Google Inc., Mountain View, California, USA) or Microsoft HoloLens (Microsoft, Redmond, WA) (Fig. 1) has also contributed to this.

In orthopedic surgery, the planning and performance of surgical procedures are often based on X-ray or CT images obtained preoperatively or intraoperatively. The individual steps of the respective operation, such as placing an implant or performing an osteotomy, are guided by this imaging. Due to the often mechanical nature of the individual steps of an orthopedic procedure, augmented reality consequently seems predestined for use as a navigation aid in orthopedics. This is probably an important reason why a globally increasing interest in this technology can be observed in orthopedics [2]. In addition to the advantage of intraoperative navigation assistance, augmented reality in orthopedics also offers other potential applications, such as use in surgical education and training of specific procedures.

Various studies showed that the application of augmented reality in orthopedic surgery has the potential to shorten procedure times, reduce radiation exposure for the patient as well as the treating orthopedic surgeon, and increase the precision of the respective procedures performed. The following is now an overview of the main applications of augmented reality in orthopedic surgery to date.

Navigation for the placement of instruments and implants

During an orthopedic procedure, the orthopedic surgeon often has to rely on his three-dimensional vision to correctly position or insert the required instruments and implants. The use of intraoperative fluoroscopy (fluoroscopy) provides only two-dimensional images and the treating orthopedic surgeon is presented with the mental challenge of translating the two-dimensional images to the actual three-dimensional anatomy of the patient. In such cases, augmented reality offers the potential to reduce treating surgeon-dependent variables for postoperative outcomes. Specifically, this can be done, for example, when using data glasses by superimposing the preoperative planning into the surgeon’s field of vision or by superimposing the correct instrument or implant positioning by means of an “overlay hologram”. For example, in a Sawbone study, it was shown that using augmented reality via data goggles that superimposed the anatomy onto the intraoperative situs increased the accuracy of pedicle screw insertion and reduced the incidence of pedicle perforation [3]. This seems particularly important due to the anatomical proximity to the nerve roots and spinal cord in this delicate surgical procedure, which is very frequently used in spinal surgery in the context of spondylodesis.

Another therapeutic modality, which is frequently used in the treatment of patients with spinal diseases, is so-called facet joint infiltrations. Such infiltrations are usually performed under fluoroscopy or navigation using computed tomography. Here, it was also shown in a Sawbone study that lumbar facet joint infiltrations can be successfully performed using augmented reality, without the risk of needle misplacement and also with significantly reduced intervention time compared to computed tomography navigated infiltrations [4]. However, in addition to the use of augmented reality in spine surgery for instrument or implant positioning, this technology is already being used in other orthopedic subspecialties. For example, a phantom study demonstrated that the accuracy of acetabular positioning during implantation of a total hip prosthesis, one of the most common procedures in orthopedic surgery, could be improved by using augmented reality [5]. Another common procedure in orthopedic and traumatologic surgery is intramedullary nail osteosynthesis, which is used for various femoral or tibial fractures, among others. A challenging surgical step in such procedures is the percutaneous insertion of the distal locking screws under fluoroscopic control. Under augmented reality navigation, this step was shown to be successful with an average of only 2 fluoroscopy images, in contrast to the conventional method, with which it has been described that up to 25 fluoroscopy images are needed to correctly place a locking screw [6].

Numerous development steps will still be necessary before broad application of augmented reality for so-called “direct” holographic navigation. The “direct” holographic navigation means that a hologram is superimposed on the surgeon directly on the corresponding anatomical structure, thus allowing navigation. Prior to this, registration steps are obligatory in order to couple reality with virtuality. The first direct holographic navigated spine surgery in the world has been performed at Balgrist University Hospital in December 2020 as part of a randomized controlled trial.

Performance of osteotomies

Osteotomies have always been used in orthopedic surgery to correct a wide variety of bony deformities. Osteotomy is defined as the targeted surgical cutting of a bone in order to subsequently correct its position so that the bone consolidates in a biomechanically and/or functionally optimal position. Preconditions for such corrective osteotomies are precise preoperative planning and accurate intraoperative implementation of the osteotomy as well as the correction, so that the patient benefits maximally from such an intervention and does not suffer from a persistent malposition. One of the most common corrective osteotomies in orthopedics today is the corrective osteotomy for the presence of hallux valgus. If such a procedure is not planned accurately or if the osteotomy and correction are not performed precisely, shortening of the great toe can potentially occur, which in turn can lead to transfer metatarsalgia and thus persistent forefoot pain. Because of this, great efforts are being made to avoid such consequential problems, and it was shown in a phantom model study (Fig. 2) that using augmented reality, even less experienced orthopedists can perform a corrective osteotomy for hallux valgus more precisely compared to the conventional method [7].

Another and probably one of the most challenging osteotomies in orthopedics is the so-called periacetabular osteotomy. This is used in selected patients with hip dysplasia and femoroacetabular impingement. In this procedure, the dysplastic pelvic socket (acetabulum) is detached from the rest of the pelvis by means of technically demanding osteotomies and reoriented as physiologically as possible. A proof of concept study has already demonstrated that the use of augmented reality via holographic superimposition of the correct osteotomy planes into the surgeon’s field of view (Fig. 3) improves the accuracy of this challenging corrective osteotomy for less experienced surgeons [8].

Orthopedic tumor surgery

Orthopedic tumor surgery is another predestined application area for augmented reality. Especially in malignant tumors, a complete resection of the tumor is absolutely necessary for therapeutic success. Often, however, the exact extent of a bone tumor is not directly visually apparent intraoperatively. Due to this, in such cases the surgeon is again dependent on his three-dimensional imagination. The mental transformation of the extent of the tumor obtained in the preoperative imaging into the intraoperative situs can be very challenging, but it is crucial for the patient. For this reason, there are ongoing efforts to further improve the precision of tumor resections. A potential advance here is also the use of augmented reality to continuously visualize the exact tumor extent intraoperatively to the treating orthopedist. In animal models, it has already been shown that using augmented reality by means of a monitor-based system (tablet PC), the resection margin of simulated bone tumors in long tubular bones [9] as well as in pelvis [10] could be removed more accurately and with a more reliable adherence to the required resection margin compared to conventional resection.

Education and training

Most orthopedic procedures rely on a certain learning curve. The more frequently a procedure is performed by an orthopedic surgeon, the better its results can be expected to be. However, at some point in their career, every orthopedic surgeon is still in the ascending branch of the learning curve for a certain procedure. In order to achieve ideal results right from the start, efforts are underway to make this possible by means of the application of augmented reality. For example, it has been shown that residents who were trained regarding facet joint infiltrations using augmented reality were subsequently able to perform facet joint infiltrations in the conventional method more accurately than residents who were trained using the conventional method only [11]. Furthermore, the possibility of so-called “tele-guided” orthopedics has already been described. Here, an experienced orthopedist supports a less experienced colleague by being virtually connected to the operation via augmented reality without actually having to be physically on site. This has already been successfully demonstrated in the context of various technically demanding shoulder procedures [12,13].

Take-Home Messages

The application of augmented reality in orthopedic surgery offers the potential to shorten procedure times, reduce radiation exposure for the patient and the treating surgeon, and increase the reproducibility and precision of procedures.
The main benefit of augmented reality to date is that relevant additional information can be superimposed on the field of vision of the treating orthopedist during defined surgical steps without having to repeatedly turn away from the surgical field.
The existing body of studies on augmented reality in orthopedics mainly includes phantom and cadaver studies. The first direct patient holographic navigation was performed in December 2020 in Zurich as part of a controlled study.

Literature:

Sutherland IE: Head-mounted three dimensional display. Proc Fall Joint Comput Conf 1968; 757-764.
Jud L, Fotouhi J, Andronic O, et al: Applicability of augmented reality in orthopedic surgery – A systematic review. BMC Musculoskelet Disord 2020; doi: 10.1186/s12891-020-3110-2.
Dennler C, Jaberg L, Spirig J, et al: Augmented reality-based navigation increases precision of pedicle screw insertion. J Orthop Surg Res 2020; doi: 10.1186/s13018-020-01690-x.
Agten CA, Dennler C, Rosskopf AB, et al: Augmented reality-guided lumbar facet joint injections. Invest Radiol 2018; doi: 10.1097/RLI.0000000000000478.
Fotouhi J, Alexander CP, Unberath M, et al: Plan in 2D, execute in 3D: An augmented reality solution for cup placement in total hip arthroplasty 2018; arXiv.
Londei R, Esposito M, Diotte B, et al: Intra-operative augmented reality in distal locking. Int J Comput Assist Radiol Surg 2015; doi: 10.1007/s11548-015-1169-2.
Viehöfer AF, Wirth SH, Zimmermann SM, et al: Augmented reality guided osteotomy in hallux valgus correction. BMC Musculoskelet Disord 2020; doi: 10.1186/s12891-020-03373-4.
Kiarostami P, Dennler C, Roner S, et al: Augmented reality-guided periacetabular osteotomy – proof of concept. J Orthop Surg Res 2020; doi: 10.1186/s13018-020-02066-x.
Cho HS, Park YK, Gupta S, et al: Augmented reality in bone tumor resection: An experimental study. Bone Jt Res 2017; doi: 10.1302/2046-3758.63.BJR-2016-0289.R1.
Cho HS, Park MS, Gupta S, et al: Can augmented reality be helpful in pelvic bone cancer surgery? An in vitro study. Clin Orthop Relat Res 2018; doi: 10.1007/s11999.0000000000000233.
Yeo CT, Ungi T, U-Thainual P, et al: The effect of augmented reality training on percutaneous needle placement in spinal facet joint injections. IEEE Trans Biomed Eng 2011; doi: 10.1109/TBME.2011.2132131.
Ponce BA, Jennings JK, Clay TB, et al: Telementoring: use of augmented reality in orthopaedic education: AAOS exhibit selection. J Bone Jt Surg – Am Vol 2014; doi: 10.2106/JBJS.M.00928.
Ponce BA, Menendez ME, Oladeji LO, et al: Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Orthopedics 2014; doi: 10.3928/01477447-20141023-05.