Edit this page

NA-MIC Project Weeks

Back to Projects List

SlicerVR: interactive UI panel in the VR environment

Key Investigators

Project Description

A key infrastructural element that is still missing from SlicerVR is the ability to show and interact with arbitrary Qt widgets in the VR scene. Up until recently, a blocking issue was an incomplete implementation of the vtkQWidgetWidget class in the VTK version that Slicer was using. Now that Slicer has been migrated to VTK9, the implementation of this in-VR widget continued.

A blocking issue for this to happen is the ability to build the SlicerVR extension against Slicer built with VTK9. Currently, build fails because the build system does not support building VTK remote modules externally (this issue breaks the build of other important extensions as well, such as SlicerVMTK).

Once build is fixed and the in-VR widget is added, arbitrary UI elements of Slicer can be used from within VR, thus basically exposing the entire Slicer functionality within the VR environment.

Objective

  1. Build SlicerVR against VTK9
  2. Add interactive Qt panel to VR scene SlicerVR#43
  3. Make use of the in-VR widget via laser pointer and VR-optimized widgets

Approach and Plan

  1. Update SlicerVR CMake files to build with VTK9
    1. Utilize new VTKExternalModule infrastructure to build vtk openvr rendering
    2. Extract the said module from VTK proper with history (JC)
    3. Change main CMake file to use this instead of the VTK remote module approach (Adam?)
    4. Fix build issues arising from the switch to VTK9 in SlicerVR (Csaba?)
  2. Try vtkQWidgetWidget in SlicerVR, confirm that it now works (Csaba)
  3. Explore existing possibilities for using a laser pointer emanating from the controllers to control the Qt-based widget (press, click, drag&drop, etc) (?)
  4. Add the already implemented but dormant VR-optimized widgets in the SlicerVR user interface (Csaba)

Progress and Next Steps

  1. Build SlicerVR against VTK9 :heavy_check_mark:
    1. KitwareMedical/SlicerVirtualReality#84: Update build system to support building against VTK9
    2. Created KitwareMedical/VTKExternalModule for externally building any built-in or remote VTK module outside of the VTK source tree.
    3. vtk/vtk#8123: vtkModule: Do not generate files in source tree when building module externally
  2. In-VR UI widgets
    1. Rebased VR widgets branch to the latest master to a new branch
    2. Built the SlicerVR branch succesfully with VTK9
    3. The vtkQWidgetWidget test still crashes unfortunately
  3. Proposed some hooks to enable customization of VR interaction from python code in Slicer. Pull request here for reference.

Illustrations

In-VR user interface

Background and References

  1. Commit that broke 3rd party module build here
  2. Discussion on remote modules vs. external modules here
  3. How to make downstream VTK modules here
    1. Links to example here
  4. https://github.com/KitwareMedical/VTKExternalModule
  5. VR virtual widget branch here