Background: In recent years, the integration of Virtual Reality (VR) with Digital Twins (DTs) has paved the way for a more immersive, interactive, and intuitive representation and control of real-world entities. 

Research Objectives: The objective of this project is to develop a virtual reality based digital twin for advanced manufacturing system integration. We have develop a preliminary version of the digital twin (Figure 1, right) for the manufacturing station (Figure 1, left). Each component of the system is connected either physically or wirelessly to a Linksys EA4500 Dual-Band Wi-Fi router. Two UR3e collaborative robot arms are connected via ethernet connection to the router. Three Raspberry Pi’s are connected wirelessly to the system. Two of these are connected physically to a LulzBot TAZ Workhorse 3D Fused Deposition Modeling (FDM) printer. Moreover, these devices are hosting Octoprint, an OpenSource 3D printer monitoring and control software. The final Raspberry Pi hosts a WireGuard VPN server, enabling remote access to the local network from anywhere which is connected to the university’s network. The virtual environment is developed in Unity, and it simulates the physical twin in real time.  

Figure 1: Sample snapshots from the VR-based digital twin (left: physical twin, right: digital twin)

The 3D printers are fully modeled in the VR system, and their extruder and bed temperature are successfully represented within the DT. In addition, the status of each printer and the time the printing operation is being performed is displayed in real time in the DT. This is accomplished through Hypertext Transfer Protocol (HTTP) requests from Unity to the Raspberry Pi hosting Octoprint. Figure 2 (left) shows the extruder and bed temperature display on the virtual printer as it samples the physical printer’s material extruder. Through the HTTP protocol, the physical printers can be issued G-Code commands, enabling fully remote printing. Monitoring and programming of the Collaborative robots is implemented through a client interface native to the device. The DT collects real time data from the physical system and simulates the same process in VR. Figure 2 (right) shows the print duration by manually observing the printing operation and comparing it against the print duration in the VR-based DT. The DT gives a reliable status of the printing status of the physical system.

Figure 2: State transition (left) and recurrence plot (right)

Research Plan: (1) continue the development of the VR based digital twin and install more sensors to the system to collect real-time data regarding the different system parameters, (2) preform a usability study of the VR environment, (3) collect and analyze real-time data from the physical system and develop predictive models in the VR environment for system monitoring and control. 

REU Student Outcomes: This project fits a student with coding and automation skills (e.g., computer science, electrical engineering). REU student will gain hands-on experience in system simulation and automation and virtual reality. Student will be supervised by Dr. Faisal Aqlan (Industrial Engineering) and participate in writing conference and journal papers.

Required Skills: Knowledge of Python, Unreal Game Engine (C++),  Machine Learning, Raspberry Pi, Arduino.