Developing a Virtual Reality Environment for Mining Research

Abstract

Recent advances in computing, rendering, and display technologies have generated increased accessibility for virtual reality (VR). VR allows the creation of dynamic, high-fidelity environments to simulate dangerous situations, test conditions, and visualize concepts. Consequently, numerous products have been developed, but many of these are limited in scope. Therefore, the National Institute for Occupational Safety and Health researchers developed a VR framework, called VR Mine, to rapidly create an underground mine for human data collection, simulation, visualization, and training. This paper describes the features of VR Mine using self-escape and proximity detection as case studies. Features include mine generation, simulated networks, proximity detection systems, and the integration and visualization of real-time ventilation models.

1. Introduction

There are many definitions of virtual reality (VR) that include various levels of immersion and features [1]. Some definitions refer to specific technologies [2], while others are limited to visualization [3]. For this paper, a more general definition of VR will be adopted. VR is defined as the collection of hardware and software that interactively engages users and stimulates their senses within a synthetic or virtual environment. From a hardware standpoint, this can include anything from a phone to a head-mounted display (HMD) to a large-scale theater as well as speakers, microphones, haptic sensors, and interaction devices. The software used to create virtual environments (VEs) generally includes a game engine, custom software components and libraries, and two-dimensional and three-dimensional graphical assets.

Recent hardware and software advances have increased the fidelity and availability of VR. New and improved technologies including HMDs and high-resolution monitors and projectors can display sharper, more detailed visualizations. For example, in the past 5 years between the release of the Oculus Rift DK1 (2013) and the HTC Vive Pro (2018), there was a threefold increase in resolution along with a 50% increase in the refresh rate. Furthermore, despite early claims that low-cost VR technology was readily available [4], computational power has not been sufficient to handle the large amount of data generated by these systems [5]. In the past 5 years, however, rendering speeds have more than doubled and memory has more than tripled [6], allowing these visualizations to be incorporated into higher-complexity and higher-fidelity VEs that more closely mirror the real world. Similarly, software capabilities have also significantly improved through game engine advancements and feature development. In the past several years, game engines have added and improved features such as physics, lighting, platform support (e.g., Android), and asset development integration. Some of the current leading commercial game engines are Amazon Lumberyard, CryEngine, Unity, and Unreal, but this can quickly change as is evident in the literature [7, 8].

As VR accessibility has increased, so has its adoption. VR is currently being used across numerous industries and environments, most recently for post-traumatic stress treatment [9], lighting design [3], emergency preparedness [10], and safety training [11] to name a few. The mining industry has also recently seen new applications of VR related to big analytics (i.e., analysis of large, complex data sets for underlying patterns and associations) and production optimization. Dassault Systèmes released its 3DEXPERIENCE Mine that uses VEs to visualize mine design, simulation, and monitoring. LlamaZoo has also developed a similar VR tool—MineLife VR—which runs on a desktop computer or HMD. Codelco even collaborated with Microsoft on a contest to design an underground mine using Minecraft [12]. However, widespread implementation and acceptance of VR in the mining industry remains elusive.

Several barriers exist to the stable and pervasive use of VR in mining. First, VR development has not been strategic nor collaborative. VR has been used in mining research for over two decades [13] specifically targeting mine safety training [4, 14, 15], but the scope and application of these projects is limited. In an assessment of thematurity ofVR in the mining industry, Stothard and Swadling identify lack of collaboration and unification of VR work as a major issue. They state that “[VR] development and implementation are ad hoc and often mining simulation does not follow a road map or strategic plan” [16]. The lack of cohesive development is also a problem because many VR projects require significant time and resources. Development projects typically last several years and often require a team of several developers to complete [17, 18]. Though these simulations offer some flexibility and content can be repurposed, non-trivial redevelopment is required to update the applications to include new scenarios. Furthermore, the rapid pace of technology advancement necessitates continual support to ensure simulations do not quickly become obsolete [16]. However, life cycle support can be especially difficult with changing priorities, limited or uncertain funding, and changes in personnel. This is especially critical because another barrier is the fact that VR development requires skilled personnel. While some applications claim to minimize the technical requirement through visual programming [7], complex, dynamic VEs require a detailed understanding of the relationships and mathematics behind what is being simulated. Lastly, VR development, especially for mining, has been limited in application. Related to the other barriers, much of the VR work in mining has been limited to training. However, there are other opportunities in education, risk assessment, product development, mine planning, mine monitoring, and emerging technologies [16, 19, 20]. Expanding the use cases of VR helps offset the investment and delivers a greater return. In any application or domain, VR has the potential to aid in solving complex problems as an intuitive mechanism to integrate humans with information.

In order to address these barriers and demonstrate new use cases for the mining industry, research can lay the groundwork for the adoption of VR. Therefore, the aim of this work is to create a VR framework to facilitate the rapid generation of custom VR software in support of research activities.

2. Benefits of Virtual Reality Research

VR is a critical tool and methodology for research that allows the creation of dynamic, high-fidelity VEs to do things that would otherwise be impossible, too costly, or too dangerous in the real world [2, 21]. VR can visualize invisible concepts and complex data, help to evaluate prototypes, and simulate dangerous or non-routine situations. These are typically accomplished through data collection and visualization projects including human subject data collection, iterative simulation, co-visualization applications, and training applications.

Though often overlooked, VR provides huge advantages to objective data collection. First, VR allows researchers to integrate synchronized sensors into a protocol. For example, motion tracking, eye tracking, and interface interactions can be easily used in VR experiments because they take place in a clean, controlled environment with reliable access to power and connectivity infrastructure. However, this type of data collection in the field would be difficult and nearly impossible in underground coal due to permissibility requirements. Second, VR allows researchers to have complete control over the environment and scenario being simulated. Though VR, especially in training, is only typically used as the background, it can also be implemented as the stimulus for experimental work [22]. Similar to sensor outputs, VR simulations can trigger events or monitor exposure. Third, VR can provide enhanced participant presence and engagement. Because of the improvements in fidelity, VR can provide a higher level of immersion than other laboratory mock-ups and typically at a lower cost [19, 21]. The increased engagement can also improve participants’ experiences and maintain higher accuracy, because boredom and disinterest have the potential to confound behavioral results just as severely as poor training or technical errors [21]. Lastly, VR can help increase the ecological validity of an experiment by making VEs closer to reality, thus eliciting a more natural response from participants. In situations where natural behavior is limited or too contrived, VR can provide a mechanism for gamification or the application of other pressures (e.g., time); these strategies have been shown to induce responses closer to natural decision-making under stress that would not otherwise be possible, while still providing a safe environment [21].

VR can also provide benefits as a visualization and communication tool. By co-visualizing different data types and making it more accessible to a non-technical audience, VR can improve collaboration [23]. Further, VR can enhance communication between geographically dispersed people [19]. Because software is easier to send than people or equipment, multiplayer simulations can be used to share concepts, prototypes, scenarios, or training around the world. Visualization techniques are also increasing in importance in the wake of big data. Without a mechanism to distill and interpret large quantities of data, embedded information may be lost to obscurity.

3. Technical Considerations for Virtual Reality Research

While VR affords many benefits to research, in order to fully achieve these benefits, research requires a higher level of rigor. Researchers should carefully consider the technical implications of accuracy, flexibility, and control. These requirements are briefly explained below and highlighted in the feature descriptions.

3.1. Accuracy

In order to draw conclusions from performed work, it must be accurate. Namely, it must be clear when, what, and how something was done. Therefore, VR application development requires significant attention to detail with respect to stimulus generation, internal calculations, and outcome data. For example, in research, it is necessary to generate a consistent, repeatable stimulus. Additionally, the sequence of when physics (e.g., positions, forces) are computed, applied, and rendered can vary depending on the application. A configuration focused on simulation rather than visualization may update the position of an object multiple times per rendering. Therefore, logging should be done with care to associate the correct data with the correct timestamp.

3.2. Flexibility

Flexible VR architectures for research need to be both modular and scalable, and the underlying framework should support further development. For example, strategic development of base VR features should be platform agnostic. Though user interfaces would need to change based on a final use case, minimizing the need for content adaptation increases the utility. To this end, a VR application could easily be made for an HMD or a large-scale theater. Another example of flexibility is the use of physically based rendering. Assets developed with this technique are able to be used in most lighting conditions without modification. This would allow content (e.g., a conveyor) to be used in both underground and surface mining applications.

3.3. Control

Control in this case refers to timing and responsiveness. As previously discussed in the “Benefits of Virtual Reality Research” section, in order to take full advantage of VR for research, the VE would be part of the stimulus and therefore should not vary between participants or groups. This means, for example, that some of the approximations (e.g., prerendered frames) used in video games that prioritize experience would have to be overridden. Similarly for responsiveness, in order to maintain immersion and validity, dynamics within the simulation should respond with consistent and minimal delay. This requirement is true for both the hardware and software.

4. Current Use Cases

Two ongoing research projects at the National Institute for Occupational Safety and Health’s (NIOSH’s) Pittsburgh Mining Research Division drove the development of this VR framework: self-escape and proximity detection systems. While neither project is the focus of this paper, this section describes the research motivation and goals of the projects that guided feature development.

4.1. Self-Escape

The primary goal of NIOSH’s Self-Escape from Underground Coal Mines Training Initiative was to characterize the essential components of the mine emergency escape system, develop competency standards, and identify best practices to improve system preparedness and self-escape training of underground coal mining personnel. Development of VR Mine features related to this work was primarily completed to prepare for future human subject decision-making work by developing the basic underground environment. The main outcome of the project has been to identify and determine current levels of miner competence in critical knowledge, skills, abilities, and other attributes [24].

4.2. Proximity Detection Systems

The primary goal of NIOSH’s Design of Proximity Systems for Underground Mobile Equipment project was to develop recommendations for the design and implementation of proximity detection systems (PDSs) for mobile haulage equipment in underground coal mines. Development of VR Mine features related to this work was in support of communicating the results of the technical and behavioral studies. Because electromagnetic fields are invisible, VR can be used to make the result more tangible for a larger audience. Some of the main findings of this work identify and characterize interference with the proximity detection systems [25, 26].

5. VRMine

Implemented as a Unity project, VR Mine is a VR framework that allows the rapid generation of VR software applications for use in research. Researchers chose to develop VR Mine using the Unity game engine because it is a stable commercial game engine with a large user base, which is important to maintain life cycle support. While open source engines may be able to provide similar communities, upgrade and integration of base features are more difficult. Researchers also selected Unity because the team had previous experience with it, it included the necessary basic features (e.g., physics, lighting, cluster rendering), and at the time was the most affordable commercial option. Software applications built from VR Mine can be run in single-player or multiplayer configurations as a participant or an observer. All of these configurations were natively included to accommodate data collection and data communication.

Because VR Mine is a workflow and not a limited application, researchers developed features for the Unity editor as well as the applications. Researchers and developers have the flexibility to incorporate, update, and modify instances of these features to fit current research needs. From the Unity editor, executable applications can be created that include specific mine layouts and the incorporated interactive features for the end users. The sections below describe some of the key features developed specifically for research applications.

5.1. Logging

Accurate and timely extracting and saving of data is a necessary function to support most research activities. The VR Mine uses a robust logging system to ensure fast and reliable recording of large volumes of data. Each item placed in the mine (e.g., mine geometry, equipment) can include a log handler component (e.g., script) that sends data specific to the item to be logged (e.g., a shuttle car sends speed data). During runtime, the log handlers send data either continuously or when specific events occur, depending on the specific research need. The data is written to a file continuously to avoid loss. However, the writing takes place in a separate process that does not disrupt the simulation.

5.2. Mine Creation and Population

As part of the framework to aid in the rapid creation of VEs, VR Mine includes semiautomated tools to build a mine. These features include building mine geometry, adding ventilation controls, adding mine infrastructure (e.g., cables, lifeline), and placing equipment. Developers are currently able to autogenerate large sections of a room-and-pillar underground coal mine through the use of tileable mine segments. Once created, mine segments can be swapped or built upon using a custom interface within Unity’s editor (Fig. 1). Ventilation and infrastructure can likewise be added throughout the mine using a point and click interface. These features are scalable and could include other commodities and configurations with the development of the 3D assets.

Fig. 1.

Editor interface that allows developers to add a new mine segment to the selected interface (highlighted yellow cube). The developer can select between various entry configurations (i.e., 4-way, 3-way, 2-way, end cap) and rock dust levels (i.e., full, medium, none)

5.3. Mine Networks

VR Mine also includes the ability to create network systems. Researchers implemented networks as a generic feature in order to maintain maximum flexibility, as many types of these systems exist. Currently, assets and logic exist that use this generic network architecture to create the following communications and tracking systems: mine phone, leaky feeder, mesh node, and radio frequency identification (RFID) tracking (Fig. 2).

Fig. 2.

Examples of the mine phone, leaky feeder, and mesh node 3D art assets for communication systems implemented in VR Mine, where the blue boxes along the cable also represent communication network nodes and the green box represents the mesh network nodes

In the Unity editor, instances of wired communication systems are created by designating cables as phone lines or leaky feeder lines. The cables will then automatically be populated with nodes that contain communication logic. Mesh style communication systems are created by placing mesh nodes within the mine; these nodes communicate with other mesh nodes within defined areas of influence (Fig. 3). Since all of the communication systems function as separate networks, multiple systems can be placed within a scene. During runtime, messages can be entered into the communication system via access points such as a mine phone or a handheld radio. These messages are transmitted along the communication systems according to each network’s connectivity and displayed to other users on the network.

Fig. 3.

A visual depiction of the areas of influence of an example mesh node communication system. According to the implemented logic, the areas in green have full coverage that degrades through yellow, orange, and red to no coverage in dark areas

Similarly for tracking, the RFID node tracking system utilizes a last past logic, where each node has a defined area of influence. Tracking nodes can be placed throughout the mine and their areas of influence can be modified. Additionally, tracking networks also update users’ locations during runtime to a small map of the mine that can be viewed from the observer interface.

5.4. Experiment Configuration

In order to facilitate simulation experiments, VR Mine includes a system to run experimental trials, varying one or more independent variables and recording the results of the simulation. To more rapidly test a large number of trials, the physics simulation update was decoupled from the normal game time clock, allowing hundreds of simulation steps to be performed for each rendered frame. For each of these simulation steps, both the physics engine and any relevant simulation components are updated with a fixed time step and then the state of the simulation is logged.

The initial experiment performed using this system considered PDSs on various underground mining haulage vehicles. For this experiment, the physical properties of the machine and parameters of the PDS were varied to determine the effect on vehicle performance as the vehicle approached a minerwearable component and triggered the PDS on the machine, causing it to slow (yellow zone) and stop (red zone) (Fig. 4). Variables such as speed, slope, load weight, braking torque, and proximity zone size were varied, and both the underlying physics as well as the control and PDS logic were simulated for each time step. The resulting position, speed, tire slip, and other critical parameters were logged.

Fig. 4.

A screenshot of the PDS performance stopping distance simulation, where the mineworker is wearing the miner-wearable component that triggered the PDS implemented with a box zone model on an unloaded battery scoop on a 10° downhill slope

Researchers included physics simulation capabilities in VR Mine in order to capitalize on the high-fidelity assets, flexibility of scenario creation, and real-time simulation and visualization capabilities. While many vehicle dynamic simulations exist (e.g., TruckSim), they typically do not contain the highlevel, mining-specific graphics already developed for VR Mine, nor do they typically run in real time. The discrete simulation capabilities provide a robust platform on which controlled simulations can be developed. Furthermore, physics calculations can also be easily decoupled, allowing researchers to use an alternative physics library if necessary.

5.5. Proximity Detection Systems

VR Mine can simulate PDS zones in three different ways— box, shell, and deformable—allowing researchers the flexibility to select the best approach depending on the requirements of the machine and PDS (Fig. 5).

Fig. 5.

From left to right, examples of the box, shell, and deformable PDS zone model on a battery hauler

The simplest PDS zone modeling method is a rectangular box of a given size and position offset for the yellow and red zones of PDSs. The box modeling method creates the most controllable and repeatable PDS. The second method approximates the electromagnetic fields of the yellow and red zones of a PDS by using shell equations for each generator [27]. The shape and position of the zones are specified by generator locations, magnetic field strength, and shape constants, which are then used to compute the final shape of the shell. The shell modeling method incorporates complex mathematics to produce a more accurate simulation that may be required for more complex scenarios and machine positioning (e.g., severe slope).

The final and most flexible configuration method is a deformable PDS. In this case, the zone shapes are configured by first specifying a base shell model with generator positions, magnetic field strengths, and shape constants. The base shell can then be deformed into almost any shape to simulate interference or match real-world measured values. The deformation is controlled by a customized brush tool within the Unity editor.

In addition to zone shape and position, VR Mine also includes features to control the state and applicability of the PDS system on each vehicle. PDS control logic allows the zone shape and position of each PDS system to change based on parameters such as machine speed, operation state (e.g., cutting, loading), mineworker (e.g., car operator), and distance fromanother machine. The zone change can either be a blend between two shapes based on some continuous variable or a discrete transformation. Zone blending is meant to simulate interference, where the zones would change based on the distance away fromanother machine or a trailing cable [25, 26]. Discrete changes are instantaneous applications of new zone shapes that can even include different modeling methods (i.e., box, shell, or deformable) in order to simulate PDS logic.

5.6. Real-time Ventilation Model

Researchers implemented the ability to simulate ventilation, smoke, and gas propagation through the mine in real time. To do so, VR Mine integrates MFIRE, “a dynamic, transient-state, mine ventilation network simulation program” [28]. Based on the mine geometry, VR Mine generates MFIRE nodes and an airway network, then uses MFIRE to calculate air velocities, gas concentrations, and contaminant concentrations. MFIRE calculates these values in discrete time steps of approximately 10 s. In order to maintain the accuracy of the data on both sides, VR Mine and MFIRE are synced, and any changes in either simulation are reflected in the next time step. For example, a user could change the ventilation system by adding or removing a stopping. The results of this change will show up within a few seconds, potentially drastically changing ventilation paths, methane concentrations, or contaminant levels. Similarly, the changes in ventilation data can then be used to visualize smoke levels, display gas concentrations on a multi-gas meter, or potentially to detect if a ventilation change causes flammable gas to flow over an ignition source. Figure 6 displays an example visualization of air velocities, methane, and contaminant concentrations due to methane liberation at the face and a coal fire.

Fig. 6.

An overhead depiction of a section of a mine with methane liberation coming from the face of the fourth visible entry from the left and a fire two crosscuts outby as modeled by MFIRE. The small particle system represents the airflow velocity and methane concentration (i.e., blue = inert, red = explosive, white = none). The large spheres indicate level of contaminant by size and color (i.e., bigger = higher concentration; red = high, white = none)

6. Future Directions

As a result of VR Mine development, researchers and developers are now able to generate a useable software application in less than a week. The flexible, accurate, and rigorous architecture of VR Mine also supports the development and refinement of additional features. VR Mine can create applications for laboratory-based data collection, a VR training tool for the field, or anything in between. As VR expands further into the mining industry, this framework can support new applications and projects. VR Mine is poised to encourage collaboration among researchers and stakeholders. Some short-term possibilities include integration with real-time atmospheric monitoring systems (AMS), virtual illuminance measurement, the creation and usage of a digital twin, and behavioral studies of haul truck operators, which will be discussed below.

The integration of real-time data from anAMS in VR Mine has the potential to improve our understanding of emergency and routine environmental conditions. Recent research indicates that AMS data can be used to identify, locate, and track a fire in an underground mine [29]. By pairing this type of data with ventilation simulations as well as historical data, researchers may be able to gain a fuller understanding of what is occurring. The realistic, intuitive nature of VR Mine applications may help to highlight underlying patterns and make this data more accessible to mine operators. An application such as this could even be used to improve mine monitoring on a regular basis.

VR also has the potential to improve underground illumination work in mining. Because of the recent improvements in the lighting pipeline thanks to the gaming industry, dynamic lighting can be simulated with reasonable accuracy in a VE. With additional work in quantifying incident and reflected light in simulation (e.g., importing photometric data), researchers can use applications developed from VR Mine to study how, when, and where light is needed to perform tasks as well as to evaluate lighting prototypes. This work could build on existing methodologies such as using static evaluations of glare using physically based rendering [30], and the results could inform lighting technology development and policy.

As evidenced by the recent development of commercial products, the mining industry is increasingly using big data analytics to better understand and address logistics and safety concerns. VR Mine can play a pivotal role in helping to visualize and transform big data. Linked with real-time data, VR Mine could be used to create digital twins of underground environments. Among other things, this connection of mine design and production data could aid in the identification and mitigation of process and design limitations.

Lastly, haul truck operators’ perceptions and decisionmaking could be studied using VR mine. Haul truck-related accidents are continually a problem in the mining industry; however, it is currently unclear why [31]. VR Mine could be used to create an HMD application that could be taken into the field to further study this problem. By leveraging the logging, experiment configuration, PDS, and networking systems features among others, researchers could easily expand into surface and other commodities. The results of this work could help researchers to gain insights into what design limitations may be contributing to these accidents.

Overall, VR still offers great potential to the mining industry. The key will be in strategically developing and maintaining technical capabilities to take advantage of this potential. Continued work in the area can drive innovation beyond our current reality.