AcuVision: AI + Mixed Reality TCM Training System
AcuVision is an innovative educational platform that combines Azure Cloud NLP services and Mixed Reality techniques to revolutionize Traditional Chinese Medicine (TCM) training. As Team Leader, I led the development of this award-winning system that features real-time acupoint detection and an intelligent medical assistant.
Project Overview
Traditional Chinese Medicine relies heavily on precise knowledge of acupoints and complex diagnostic principles. Training new practitioners traditionally requires years of hands-on practice with experienced mentors. AcuVision transforms this educational process by creating an immersive mixed reality environment where students can visualize acupoints, practice treatments, and receive real-time feedback and guidance.
Technology Stack
Mixed Reality
- Microsoft HoloLens 2
- Mixed Reality Toolkit (MRTK)
- Spatial Mapping
- Hand Tracking
Development
- Unity 3D
- C# Programming
- Unreal Engine
- C++ Programming
Azure Services
- Azure Language Understanding (LUIS)
- Azure Cognitive Services
- Azure Cloud Infrastructure
- Azure Custom Vision
Machine Learning
- Hand Acupoint Detection Algorithm
- Natural Language Processing
Core Features
Real-time Acupoint Detection
One of the most significant technical achievements of this project is the development of an algorithm that can accurately identify and track 99 acupoints on the human hand in real-time. This required:
- Creating a detailed 3D model of the human hand with precisely mapped acupoint locations
- Implementing advanced hand-tracking algorithms to maintain accurate positioning regardless of hand orientation
- Optimizing the detection system to operate with minimal latency on the HoloLens 2's hardware
- Developing visual overlays that clearly indicate acupoint positions without obstruction
Acupoint Detection Visualization
Real-time visualization of the 99 acupoints detected on a human hand
Intelligent Medical Assistant
The natural language dialogue system leverages Azure LUIS (Language Understanding Intelligent Service) to:
- Automatically record and transcribe medical consultations
- Extract key medical terminology and diagnostic information
- Provide real-time suggestions and information about potential treatments
- Maintain a structured medical record that can be referenced and reviewed
- Integrate with the visual acupoint system to suggest relevant treatment points
Implementation Approach
Challenges
- Achieving high accuracy in acupoint detection despite hand movement and rotation
- Optimizing performance for real-time operation on HoloLens 2 hardware
- Creating a natural language system capable of understanding specialized TCM terminology
- Integrating mixed reality visuals with practical training scenarios
- Ensuring intuitive user interaction for non-technical medical practitioners
Solutions
- Developed a custom bone-anchored positioning system for acupoint placement
- Created custom Azure LUIS models trained on TCM medical corpus
- Designed interactive training scenarios with expert practitioners
- Conducted extensive user testing with medical students for interface refinement
Project Timeline
Conducted initial research, formed the team, and defined project objectives. Consulted with TCM experts to identify key educational challenges.
Developed the foundational systems including hand tracking, acupoint mapping, and initial LUIS models. Created HoloLens application architecture.
Integrated all components, refined the user interface, and optimized performance. Conducted initial user testing with medical students and practitioners.
Completed final development, prepared documentation, and participated in various competitions and exhibitions, winning multiple awards.
Awards and Recognition
Technical Implementation Details
Mixed Reality Development
The AcuVision project leverages the capabilities of HoloLens 2 to create an immersive learning environment:
- Implemented spatial mapping to integrate virtual content with the physical environment
- Used MRTK's hand interaction system to enable natural manipulation of virtual objects
- Developed custom shader systems for visualizing acupoints with depth-appropriate rendering
- Created holographic UI elements that follow ergonomic best practices for mixed reality
Azure Cloud Integration
The system leverages several Azure cloud services to provide intelligent assistance:
- Azure LUIS for natural language understanding of medical terminology
- Azure Speech Services for voice recognition and transcription
- Azure Functions for serverless processing of complex queries
- Azure Storage for maintaining patient records and training data
System Architecture
Architecture diagram showing the integration between HoloLens, Unity, and Azure services
Results and Impact
The AcuVision system represents a significant advancement in TCM education technology:
- Enhanced Learning Experience: Students can visualize acupoints and practice treatments in an immersive environment, accelerating the learning process.
- Precision Training: The real-time feedback system helps students develop greater precision in identifying and targeting acupoints.
- Standardized Education: Provides a consistent educational experience that can be deployed across multiple institutions.
- Reduced Learning Curve: Initial testing shows a significant reduction in the time required for students to achieve proficiency in locating hand acupoints.
Conclusion
AcuVision demonstrates the powerful potential of combining mixed reality and artificial intelligence in medical education. By creating an immersive, interactive learning environment, we've transformed how Traditional Chinese Medicine can be taught and practiced. The recognition received through multiple awards validates the innovation and effectiveness of this approach.
The system's success has opened opportunities for expanding into other areas of medical education and training. Future development will focus on expanding the anatomical coverage beyond hand acupoints and enhancing the AI assistant's capabilities with more advanced treatment suggestions.