Remote operation is an emerging technical component of vehicle automation. It allows a human operator to support a connected and automated vehicle from a distance. However, today’s remote operator interfaces are predominantly only visual. So, what about sound?
A summary of our officially published research paper
In this article, we have summarised our research paper, which is now officially published for the world to explore. Our paper underwent an extensive peer-review process and has been evaluated by experts in the field.
We believe that science and technology have the power to transform our world for the better. For that reason, we are super excited to see how our research can pave the way for advancements and innovations that will benefit society at large.
If you want to get into the nitty gritty of our research, you can access the full paper here.
But for now, let’s dive in!
Can sound improve remote operator interfaces?
The purpose of our research was to explore how sound can improve remote operator interfaces. We wanted to test if an auditory display could enhance speed perception, sense of self-motion, situational awareness, and presence. To find out, we carried out an experiment with 28 participants in a simulated environment, using both propulsion sound from the vehicle and spatial augmented sounds from surrounding road users.
The development of automated road vehicles
To get some background, let’s first discuss the progression of automated road transport.
In recent years, there has been a significant increase in the development of automated road vehicles. Automated Driving (AD) offers benefits to our society like reduced environmental impact, enhanced traffic safety, and more efficient mobility systems. However, challenges remain, especially in handling unexpected scenarios, sensor and system failures, and situations beyond the system’s design limits.
To address these challenges, remote operation, where a human operator assists the vehicle remotely, has been developed. (Remote operation involves three categories: remote driving, remote assistance, and remote supervision. In our research, we focus on remote driving.) Nevertheless, remote operation presents challenges, including technology and human factors. These issues involve situational awareness, increased cognitive load, speed and acceleration estimation, and video feed quality.
While current remote operator interfaces heavily rely on visual cues, it is essential to explore other senses, like sound, to improve the overall experience and effectiveness of remote operators.
Why include sound in a remote operating station?
Now, let’s discuss some key reasons for including sound in a remote operating station.
Enhanced operator performance
Modern vehicles use various interface sounds to guide and warn drivers, like turn signals, parking assistance, lane departure warnings, and drowsiness alerts. By adopting these sounds in remote operations, it is possible to enhance the operator’s performance.
Reduced risk of excessive speeds
Accurate speed perception is crucial for safe remote vehicle operation. In remote operator stations with limited visual cues, sound from the operated vehicle becomes vital for speed regulation. Studies show that in-car noise has an impact on perceived speed, suggesting that a lack of propulsion sound feedback may lead to excessive speeds.
Increased sense of self-motion
For a remote operator, a sense of actually moving is essential for awareness. While visual stimuli alone can create such sensations, auditory stimuli, including propulsion sound, contribute to self-motion sensations and increased auditory-visual display sensations.
Improved speed perception
Situational awareness is vital for remote operators. An auditory display, specifically through the vehicle’s own sound, can improve speed perception, enhancing all three levels of situational awareness (perception, comprehension, and prediction). Hearing the remote environment aids in identifying, localizing, and predicting the trajectory of road users.
Increased sense of presence
The sensation of presence, feeling ”being there,” is crucial in remote vehicle operation. Auditory displays, including spatialized sound and individualized Head Related Transfer Functions (HRTFs), play a significant role in achieving a sense of presence. Feeling more present in the remote environment may lead to safer vehicle control.
In summary, auditory displays, focusing on propulsion and augmented sound, have the potential to improve the situation for remote vehicle operators. Our research evaluates their impact on situational awareness, speed, self-motion perception, and the sensation of presence in various scenarios and use cases.
Our hypothesis – sound will improve for remote drivers
We wanted to test two main hypotheses related to an auditory display prototype:
- Hypothesis 1: Propulsion sound and augmented sound both contribute to self-motion sensation and speed perception.
- Hypothesis 2: Propulsion sound and augmented sound contribute to situational awareness and presence, with augmented sound having a greater impact.
Augmented sound (AS) helps the user understand the surrounding objects, where they are (perception) and where they are going (comprehension). Propulsion sound (PS) helps the user understand the remote car’s speed, facilitating the projection of how surrounding objects move in relation to the remote car through augmented sounds.
The experiment – an interactive driving simulator
Now, it is time to dive into the experiment. This is how we conducted it.
Full factorial design
We employed a full factorial design with spatial augmented sounds and propulsion sound as independent variables. Dependent variables encompassed speed regulation performance, self-rated self-motion, speed perception, driving performance, situational awareness, and presence.
Driving simulator and auditory display
We used an in-house driving simulator based on CARLA open-source software featuring three curved screens, a steering wheel, and pedals. The auditory display prototype LAVA (Layered Augmented Vehicle Audio) rendered sound.
Introduction to the experiment
We introduced the participants to the experiment and driving simulator, ensuring they understood their task of remotely controlling the car at 50 km/h. We also collected demographic data and information about their driving and computer game experience.
Driving sessions
The participants then experienced four driving sessions, each comprising practice and real trials, under four different auditory display conditions: no sound, PS on, AS on, or both on. The participants used headphones and a head tracker for rendering in relation to their head movements. Vibrotactile feedback enhanced the low-frequency sensation of the propulsion sound through electrodynamic shakers in the chair. They navigated through scenarios in the map provided by the CARLA software, encountering situations like a car turning in front of them, pedestrians crossing, and cyclists biking. Navigation arrows guided the participants, and their task was to follow instructions and drive as if controlling a real car.
Visual stimuli
The visual stimuli used the map in a nighttime setting to reduce the visual detectability of surrounding road users, emphasizing the importance of auditory cues. The viewpoint included the vehicle’s partially visible hood for ease of handling. A digital-style speedometer was presented during the practice sessions to familiarize participants with the 50 km/h perception. The speedometer was not shown in the subsequent real trials.
Auditory stimuli
Auditory stimuli included the propulsion sound of the vehicle and augmented sound representing surrounding road users. Propulsion sound with low-frequency vibrational feedback mimicked the interior of an electric car at various speeds. Augmented sound comprised three source types: car, pedestrian, and cyclist sounds, spatialized and enhanced with head tracker input for realism.
Scales, questionnaires and interviews
After each driving session, we included a break for the participants to complete questionnaires with rating scales. After the experiment, we conducted interviews to capture the participants’ overall experience, perception of sound, and potential improvements.
Questionnaire for presence
1. I had a sense of “being there” in the virtual environment.
2. There were times when the virtual environment was the reality for me.
3. The virtual environment seemed like images that I’d seen or somewhere I visited.
4. I had a stronger sense of being elsewhere/being in the virtual environment.
5. I consider the virtual environment much like the other places I’ve been today.
6. During the experience I often thought that I was really driving in the city.
Questionnaire for situational awareness
1. I was able to localize pedestrians around me.
2. I understood in which direction pedestrians were walking.
3. I was able to localize cyclists around me.
4. I understood in which direction cyclists were going.
5. I was able to localize cars around me.
6. I understood in which direction cars were driving.
7. I knew what was going to happen within the next few seconds.
What were the results?
Now, it is time to reveal the results!
Presence
In the first scale item of the questionnaire, “I had a sense of ’being there’ in the virtual environment,” we observed a significant main effect of propulsion sound. Ratings were markedly higher with the propulsion sound turned on versus off. This trend persisted across all presence scales. Similarly, augmented sound demonstrated a main effect on the first presence item, with significantly higher ratings when augmented sound was on versus off. Augmented sound also had significant effects on scale items 2, 5, and 6.
Situational awareness
Augmented sound also showed main effects on situational awareness items, indicating an enhanced ability to localize pedestrians, cyclists, and cars. This effect was also observed for item 7, “I knew what was going to happen within the next few seconds.”. Propulsion sound led to higher ratings on items 1, 5, 6, and 7, indicating an improved ability to localise pedestrians and cars as well as understand their directions.
Driving performance, speed perception, and motion perception
Propulsion sound significantly boosted the participants’ certainty about their speed. It heightened the sensation of self-motion and resulted in higher self-rated driving performance. Augmented sound positively influenced self-motion sensation.
Speed regulation performance
Propulsion sound significantly reduced the mean speed, deviation from the target speed of 50 kph, and standard deviation of speed. However, augmented sound showed no significant effects on any speed regulation measures.
To sum it up, propulsion sound had a positive impact on presence, situational awareness, subjective speed perception, and driving performance. Augmented sound enhanced presence, situational awareness, and self-motion sensation.
Sound does improve remote vehicle operation
Based on our interviews, the participants experienced the propulsion sound positively, enhancing realism and aiding speed-related decisions. They associated specific pitch changes with reaching 50 kph, which contributed to a smoother driving flow and heightened spatial experience.
The augmented sounds influenced the participants’ awareness of the surroundings, providing hints of events and enhancing presence. They created a sense of space but had an ambiguous effect on comprehension. Some found it difficult to interpret information, citing overlapping or indistinguishable sounds. Limited field of view exacerbated these challenges, leading to split attention and increased cognitive load.
Despite challenges, the participants experienced that driving with auditory feedback felt better than in a silent environment. They preferred a combination of propulsion and augmented sound, as it fostered a sense of safety, eased driving, and increased connection to the virtual environment.
The future of remote vehicle operation
As we approach the future of remote vehicle operation, our experiment serves as a step toward a more elevated driving experience. The promising outcomes underscore the potential of sound as a catalyst for advancements in this domain.
The experiment also gave us valuable insights, pointing towards the need for for improvements in how the auditory display represents nearby road users. This includes adjustments such as narrowing the sound range and adjusting volume levels based on importance in different situations. These improvements will not only reduce irrelevant information but also make remote vehicle operation smoother and more intuitive.
Looking ahead, future research should explore these improvements in more realistic settings, assessing long-term safety benefits and investigating operators’ preferences in daily operations. Needless to say, the role of warning and information sounds in remote operation should remain a key focus for further experimental exploration.
I hope you enjoyed this summary of our research. Feel free to reach out with any questions or to delve deeper into the future of remote vehicle operation.