Abstract

The concept of sensory substitution involves supplying sensory stimuli through a sensory modality usually used for gathering different sensory data. An example of this would be echolocation in blind humans. It is usually our eyes which help us decide how far away objects are, by processing the sensory stimuli we receive visually. However, in this example, this same information, of how far away different objects are, is instead being received via the human’s sense of hearing. Over time, blind humans who make use of echolocation are able to improve their perception of how far away objects are. Though this example shows a non artificial method of sensory substitution, external devices are usually used. These devices are known as sensory substitution devices, and they simply make use of sensors to detect stimuli, conduct some processing, and output a different form of stimuli. For this previous example, a theoretical device might use a proximity sensor to detect how far away objects are, conduct processing, and use a vibration stimulus to convey this information to the blind person. In this experiment, a custom Sensory Substitution Device (SSD) was designed to approximate the vestibular system via a gyrometer placed on the back of the head, the readings from which were used to deliver appropriate haptic feedback using a custom haptic belt. This SSD is open source and provides a platform for other sensory substitution experiments. The wearer’s sense of balance was tested before and after a weeklong period, during which the SSD was worn, using sharpened Romberg tests, and there was indeed a very statistically significant increase in performance after this period (p<0.001).

Introduction

Vestibular function in humans is paramount to our daily lives. Activities such as walking¹, balancing, and posture² are all significantly affected by our vestibular system. Thus, those with vestibular impairments, such as certain types of vertigo, are significantly impacted by their reduced ability to carry out these tasks. The concept of sensory substitution involves supplying sensory stimuli through a sensory modality usually used for gathering different sensory data³ – and has an important role to play in the rehabilitation of those with vestibular impairments. Sensory substitution requires the use of a specific device, which uses artificial sensors to collect sensory data, and delivers this data to a subject via one of their sensory modalities (e.g touch, audio, visual). Such devices are called Sensory Substitution Devices (or SSDs), and due to the custom nature of these devices and the difficulties associated with designing them⁴, there are relatively few studies where the ability for sensory substitution devices to provide measurable improvement in completing tasks is tested. However, the studies which have taken place, such as those with the BrainPort artificial vision device^{5 6}, the feelSpace belt⁷, and the research of Maria C Dadarlat⁸, have all shown that SSDs can cause a measurable and significant improvement in completion of relevant tasks. The BrainPort and feelSpace belt (pictured below) studies show that when this artificial stimulus, in the form of haptic feedback, is related to the input from an artificial sensor, human subjects are able to adapt to these new stimuli (which has qualitative significance, as it is related to a sense), and improve their ability at tasks which required the use of a sense similar to the artificial sensor used, as well as experience qualitative improvements in their ability to sense.

Figure 2 – The feelSpace belt device.

Figure 1 – The BrainPort device.

The BrainPort used a camera to help improve those with impaired vision’s sense of sight, and the feelSpace belt used an artificial compass to help the blind with their navigation ability. This study aims to contribute to the small, but incredibly important body of research with an original device and investigation. A belt capable of delivering haptic feedback to any one of 15 locations around itself, with adjustable strength, was created for this experiment. When coupled with a triple-axis rotational acceleration sensing unit (gyrometer), the belt (henceforth referred to as the SenseBorg Belt) is able to provide tactile feedback in the direction in which the participant’s head is rotationally accelerating. This aims to approximate the sense of balance, as the semicircular canals in the ear effectively sense rotational acceleration (amongst other things)⁹, and thus to test whether the wearing of this belt can improve the wearer’s sense of balance, as the wearer begins to associate the haptic feedback with concepts of balance, and thus adjusts their motor skills to utilise this new data. This experiment will be conducted on those without impairments to their sense of balance in order to investigate whether the use of this artificial stimuli will improve their already existing vestibular functions through sensory substitution. If the experiment succeeds, there is a high chance that this apparatus would prove even more useful for those with vestibular impairments, like vertigo¹⁰, and potentially even Parkinson’s postural sway¹¹.

Hypotheses:

Hypothesis: The wearing of the SenseBorg belt over a week-long period, for 12 hours every day, will result in a statistically significant improvement in Romberg test results (longer duration spent without falling).

Methods and Materials

Materials

The main material used for this experiment was the SenseBorg belt. The SenseBorg belt was designed from scratch, taking inspiration from the feelSpace belt. The SenseBorg belt contains many vibration elements spaced evenly around a belt. When the belt and an accompanying headband are worn, the belt vibrates in the direction in which it senses the most rotational acceleration, thereby providing the wearer with information about their balance, haptically. To design the belt, a technical document of the feelSpace belt¹² was analysed, and decisions were made based on electronic knowledge and budget considerations. Below is a table showing the differences between the SenseBorg belt and the feelSpace belt.

	feelSpace Belt	SenseBorg Belt
Vibration Motor Count	30	15
Battery Life (h)	20	12
Max vibration strength (G)	1.2	1.15
Type of vibration motor	Pancake	Pancake
Type of Battery	Lithium-Ion	Lithium-Ion

Table 1—A comparison between the technical specifications of the feelSpace belt and the SenseBorg belt.

Thus, it is clear that the general specifications of my custom belt are similar to that of the feelSpace belt. Notable differences are a lower vibration motor count, battery life, and slightly lower max vibration strength. A lower vibration motor count was chosen as after it was observed that later models of the feelSpace belt had cut their number of vibration motors, and because I felt an angular precision of 24° would be sufficient to provide the wearer with sufficient directional feedback. The reduced battery life and haptic motor strength were both budget considerations, but after conducting my own pre-experiment, in which the ability of a subject to point in the direction where they felt vibration was tested (headphones with loud music were worn to eliminate sound as an aid), the strength and number of vibration motors chosen was deemed satisfactory.

In order to best approximate the mechanisms by which the semicircular canals work, a gyrometer was mounted in a headband, which would be worn by the subject, positioning this sensing unit on the back of the head. As the semicircular canals primarily sense rotational acceleration⁷, it was decided that this would be the factor which would determine the vibration direction and strength on the belt. Custom software was written, which constantly read sensory input from the gyroscope and calculated the direction and magnitude of the current rotational acceleration vector the wearer was experiencing. The vibration motor closest to the exact direction of the acceleration was turned on with strength correlating to the magnitude of the acceleration. Only one motor was ever turned on at a time, as it has been shown that multiple vibrating elements vibrating at once can cause distress to the wearer (due to the incidence of beating rhythms)¹³. As vibration motors cannot generally be run with variable strengths, the common way to vibrate at “lower strengths” is to use PWM (pulse width modulation), a technique which involves turning the motor on and off very quickly, many times a second, so that on average, the motor’s strength is weaker. I chose a frequency of 300Hz, so that the vibrations would be best detected by the Pacinian Corpuscles⁹ (a type of mechanoreceptor, which detects vibrations in the range of 250Hz-350Hz). I then varied the duty cycle between 0% and 75%, depending on the magnitude of the rotational acceleration. Thus, the SenseBorg belt was engineered from a biological perspective, to give the experiment its greatest chance of succeeding in a meaningful way. Below is a drawn diagram of the belt, headband, and individual vibration unit, with each part’s capabilities written out.

Figure 3 —A drawn diagram of the SenseBorg belt, gyroscope headband, and of individual vibration units, with technical specs written out.

Methods

There are many different ways of testing the human sense of balance¹⁴. However, many of these require specialised apparatus or access to medical facilities, both of which are outside my current resources, especially given the current pandemic. Whilst more rudimentary, it is easy to carry out the Romberg test¹⁵, which requires no external apparatus to conduct. The basic Romberg test consists of having a patient stand with their feet together, arms by their sides, and eyes closed. As the Romberg test is usually administered to those with vestibular impairments, after conducting a small pre-experiment on an independent subject, a sharpened Romberg test was decided on. This simply involves adjusting the position in which the subject must remain. To conduct a single Romberg test, a subject was instructed to place their arms on the opposite shoulders, place one foot in front of the other (both facing forwards, toe to toe), tilt their head back as far as they could, and close their eyes. The subject was then timed for how long they could stay in this position, with any movement of arms off the shoulders, eyes opening or feet moving counting as a lapse of position. The closing of the eyes not only made the test harder, but it also eliminated the visual aspect of the sense of balance, creating a greater reliance on the subject’s vestibular sense, the sense the belt was meant to approximate.

In order to see if the wearing of the SenseBorg belt over a week-long period could contribute to a better sense of balance, a first series of sharpened Romberg tests was conducted on the subject, before they had had any interaction with the belt. A series of 30 sharpened Romberg tests in a row was conducted, with the subject being allowed to rest for a minute halfway through to eliminate muscular fatigue as a factor. Further, the subject was instructed to switch which foot they had placed forward halfway through, to combat muscular imbalances as a factor in the tests. The subject was also made to wear a turned off SenseBorg belt during these tests. The belt was turned off for the whole duration but was worn just in case the slight weight of the belt had an impact on the results. Shoes and socks were not worn, and the tests were carried out on a wooden floor.

After this initial testing, the subject wore the (turned on) SenseBorg belt for 7 days, 12 hours each day (on average). After this week, 30 more sharpened Romberg tests were conducted with the subject wearing the belt (turned on), and the same considerations listed above taken into account. The tests were also taken at the same time of day as the original tests, to attempt to eliminate the contribution of daily mental fatigue to the subject’s results. They were also taken with the same footwear and floor surface. The two sets of data were then compared to see whether the week spent wearing the belt had any effect.

Results

Figure 4 —A histogram of the results of the Romberg tests. Blue represents the results of tests taken before the belt was worn for a week, and red is after.

Above is Figure 4, showing the results collected from this experiment as a histogram. The raw data which was analysed took the form of two distributions, each comprising 30 measurements. The two distributions were the set of Romberg test results from before the belt was worn, and the set of results from after the weeklong period of wearing the belt. Figure 4 was generated by counting the number of data points present within each 5s wide class, for both distributions, and plotting the number present for each class on the above graph.

As the aim of this experiment was to investigate whether the wearing of the device caused a statistically significant increase in the wearer’s sense of balance, it was necessary to conduct some kind of statistical analysis comparing these two distributions, and whether they varied significantly. A t-test was conducted on these two distributions, using the below formula:

Figure 5 – The formula for a t-test. x1 and x2 are the means of each distribution, s1 and s2 are the standard deviations of each distribution, and n1 and n2 are the total number of data points for each distribution.

Using the calculated mean and standard deviations, a t-test value was calculated to be 4.731. The critical value at a p-value of 0.0005 (with 58 degrees of freedom) was found to be 3.4663, therefore my findings are extremely statistically significant at the p=0.05 level.

Discussion and Limitations

This experiment has shown (with p<0.0005) that the wearing of the SenseBorg belt for a week-long period, for 12 hours at a time, resulted in statistically significant improvement in Romberg tests. The Romberg test is a relatively strong indicator of the ability of a subject’s sense of balance. Thus, it is reasonable to conclude that this experiment shows that the wearing of the SenseBorg belt over a week resulted in an improvement to the wearer’s sense of balance. Whilst it can only be speculated, I think it is likely the wearer developed new sensorimotor contingencies through the week, learning and adapting to the signals they were receiving through the belt, in conjunction with their own pre-existing sense of balance. Through this process, they began to associate the haptic feedback of the belt with the idea of balance. Thus, they were able to intuitively interpret the signals they were receiving through the belt as information on their state of balance and use this to avoid falling during the second round of Romberg tests. In other words, the belt caused them to develop a better sense of balance.

There were also no anomalous results with all results being relatively consistent with the rest of the data, showing that the testing method was rigorous and well enforced.

There were a few limitations with this experiment. The number of test subjects was 1, and if this were to be increased, the data produced would be much more reliable. The method of testing balance was repeated Romberg tests. Though Romberg tests work well enough, there are better ways of testing the sense of balance, though these require specialised equipment which was not available at the time. Finally, there was some human error introduced due to the recording of times by a human with a stopwatch. Whilst there were some limitations, the experiment was generally designed well, and its results are strong.

Conclusion

Thus, whilst there were some limitations to my experiment, it has still shown that the wearing of the SenseBorg belt for a week resulted in a statistically significant increase in Romberg test results, implying that the subject’s sense of balance has improved. This research has implications in rehabilitation for those with vestibular disorders, as the permanent wearing of a device like this, could result in a much better sense of balance. This would eliminate the need for any kind of invasive surgery, extensive therapy, or medicine with side effects. In addition to this specific use case, this research adds to the currently small body of research on sensory substitution, a field which more generally has applications throughout the field of rehabilitation, as well as new technologies used for optimising information flow to human workers. Research from BrainPort’s creators¹⁶ explores the possibility of using SSDs to deliver new sensory information during human-machine interactions, in an attempt to reduce the brain overload problem. By adding to this body of research, the case for sensory substitution has been strengthened, and a new platform on which further research could build upon has been provided, given that the SenseBorg belt is completely modular and open-source (unlike the aforementioned SSDs). Not only is the belt open source, but a full build guide can be found on Github¹⁷. In the future, research on sensory substitution could be carried out using the SenseBorg belt in conjunction with a different sensor, and different software translating sensor inputs to vibration outputs – i.e this belt could be used for other experiments in which the ability to improve other senses is tested. Whilst the results shown here are significant, more research with larger sample size, as well as research on those with vestibular impairments must be done to further investigate the applications of this experiment.

Acknowledgements

Many thanks to Dr Evans (Project Supervisor) for her help through this project, to Prabha Kotiswaran for being a test subject, and Thomas Fossati for his technical advice.

Bibliography

1 Kim, Soo Chan, Joo Yeon Kim, Hwan Nyeong Lee, Hwan Ho Lee, Jae Hwan Kwon, Nam beom Kim, Mi Joo Kim, Jong Hyun Hwang, and Gyu Cheol Han. \”A quantitative analysis of gait patterns in vestibular neuritis patients using gyroscope sensor and a continuous walking protocol.\” Journal of neuroengineering and rehabilitation 11, no. 1 (2014): 1-9. doi:10.1186/1743-0003-11-58.

2 \”Physiology, vestibular system.\” In StatPearls [Internet]. StatPearls Publishing, 2018.\”. Ncbi.Nlm.Nih.Gov. https://www.ncbi.nlm.nih.gov/books/NBK532978/.

3 Simner, Julia, and Edward M. Hubbard, eds. Oxford handbook of synesthesia. Oxford University Press, 2013.

4 Kristjánsson, Árni, Alin Moldoveanu, Ómar I. Jóhannesson, Oana Balan, Simone Spagnol, Vigdís Vala Valgeirsdóttir, and Rúnar Unnthorsson. \”Designing sensory-substitution devices: Principles, pitfalls and potential 1.\” Restorative neurology and neuroscience 34, no. 5 (2016): 769-787. doi:10.3233/rnn-160647.

5 Nau, Amy C., Christine Pintar, Aimee Arnoldussen, and Christopher Fisher. \”Acquisition of visual perception in blind adults using the BrainPort artificial vision device.\” American Journal of Occupational Therapy 69, no. 1 (2015): 6901290010p1-6901290010p8. doi:10.5014/ajot.2015.011809.

6 Lee, Vincent K., Amy C. Nau, Charles Laymon, Kevin C. Chan, Bedda L. Rosario, and Chris Fisher. \”Successful tactile based visual sensory substitution use functions independently of visual pathway integrity.\” Frontiers in human neuroscience 8 (2014): 291. doi:10.3389/fnhum.2014.00291.

7 König, Sabine U., Frank Schumann, Johannes Keyser, Caspar Goeke, Carina Krause, Susan Wache, Aleksey Lytochkin et al. \”Learning new sensorimotor contingencies: Effects of long-term use of sensory augmentation on the brain and conscious perception.\” PLoS One 11, no. 12 (2016): e0166647. doi:10.1371/journal.pone.0166647.

8 Dadarlat, Maria C., Joseph E. O\’doherty, and Philip N. Sabes. \”A learning-based approach to artificial sensory feedback leads to optimal integration.\” Nature neuroscience 18, no. 1 (2015): 138-144. doi:10.1038/nn.3883.

9 Purves, Dale, George J. Augustine, David Fitzpatrick, Lawrence C. Katz, Anthony-Samuel LaMantia, James O. McNamara, and S. Mark Williams. \”Circuits within the basal ganglia system.\” In Neuroscience. 2nd edition. Sinauer Associates, 2001.

10 Strupp, Michael, and Thomas Brandt. \”Diagnosis and treatment of vertigo and dizziness.\” Deutsches Ärzteblatt International 105, no. 10 (2008): 173. doi:10.3238/arztebl.2008.0173.

11 Park, Jeong-Ho, Yeo-Jeong Kang, and Fay Bahling Horak. \”What is wrong with balance in Parkinson’s disease?.\” Journal of movement disorders 8, no. 3 (2015): 109. doi:10.14802/jmd.15018.

12 Kärcher, Silke Manuela, Sandra Fenzlaff, Daniela Hartmann, Saskia Kathi Nagel, and Peter König. \”Sensory augmentation for the blind.\” Frontiers in human neuroscience 6 (2012): 37. doi:10.3389/fnhum.2012.00037.

13 Nagel, Saskia K., Christine Carl, Tobias Kringe, Robert Märtin, and Peter König. \”Beyond sensory substitution—learning the sixth sense.\” Journal of neural engineering 2, no. 4 (2005): R13. doi:10.1088/1741-2560/2/4/r02.

14 Renga, Vijay. \”Clinical Evaluation of Patients with Vestibular Dysfunction.\” Neurology research international (2019): 1-8. doi:10.1155/2019/3931548.

15 Khasnis, A., and R. M. Gokula. \”Romberg\’s test.\” Journal of postgraduate medicine 49, no. 2 (2003): 169. https://pubmed.ncbi.nlm.nih.gov/12867698/.

16 Danilov, Yuri, and Mitchell Tyler. \”Brainport: an alternative input to the brain.\” Journal of integrative neuroscience 4, no. 04 (2005): 537-550. doi:10.1142/s0219635205000914.

17 Belur, Bhuvan. 2020. \”Bhuvan21/Senseborg\”. Github. https://github.com/bhuvan21/SenseBorg.

About The Author

Bhuvan Belur is a high school student in the UK. He loves to tinker with electronics and make things, and this interest expands into his love for biology. Recently he\’s worked on the official ratings app for the english chess federation, and creating his own budget scoring box for fencing (sport). In the future he hopes to study computer science or engineering, though he\’s not yet decided.