ORIGINAL RESEARCH
published: 05 November 2021
doi: 10.3389/fpsyt.2021.739742
Frontiers in Psychiatry | www.frontiersin.org 1 November 2021 | Volume 12 | Article 739742
Edited by:
Preethi Premkumar,
London South Bank University,
United Kingdom
Reviewed by:
Daryl Wayne Niedermoser,
University Psychiatric Clinic
Basel, Switzerland
Séamas Weech,
McGill University, Canada
*Correspondence:
Stéphane Bouchard
Specialty section:
This article was submitted to
Psychological Therapies,
a section of the journal
Frontiers in Psychiatry
Received: 11 July 2021
Accepted: 07 October 2021
Published: 05 November 2021
Citation:
Bouchard S, Berthiaume M,
Robillard G, Forget H,
Daudelin-Peltier C, Renaud P, Blais C
and Fiset D (2021) Arguing in Favor of
Revising the Simulator Sickness
Questionnaire Factor Structure When
Assessing Side Effects Induced by
Immersions in Virtual Reality.
Front. Psychiatry 12:739742.
doi: 10.3389/fpsyt.2021.739742
Arguing in Favor of Revising the
Simulator Sickness Questionnaire
Factor Structure When Assessing
Side Effects Induced by Immersions
in Virtual Reality
Stéphane Bouchard
1,2,3
*
, Maxine Berthiaume
2
, Geneviève Robillard
1,4
, Hélène Forget
1
,
Camille Daudelin-Peltier
1
, Patrice Renaud
1
, Caroline Blais
1
and Daniel Fiset
1
1
Département de Psychoéducation et de Psychologie, Université du Québec en Outaouais (UQO), Gatineau, QC, Canada,
2
School of Psychology, University of Ottawa, Ottawa, ON, Canada,
3
Centre de recherche du Centre Intégré de Santé et des
Services Sociaux de l’Outaouais, Gatineau, QC, Canada,
4
Innovation, Science and Economic Development Canada,
Ottawa, ON, Canada
Two issues are increasingly of interest in the scientific literature regarding unwanted virtual
reality (VR) induced side effects: (1) whether the latent structure of the Simulator Sickness
Questionnaire (SSQ) is comprised of two or three factors, and (2) if the SSQ measures
symptoms of anxiety that can be misattributed to unwanted negative side effects induced
by immersions in VR. Study 1 was conducted with a sample of 876 participants. A
confirmatory factor analysis clearly supported a two-factor model composed of nausea
and oculomotor symptoms instead of the 3-factor structure observed in simulat ors.
To tease-out symptoms of anxiety from unwanted negative side effects induced by
immersions in VR, Study 2 was conducted with 88 participants who were administered
the Trier Stress Social Test in groups without being immersed in VR. A Spearman
correlation showed that 11 out of 16 side effects correlated significantly with anxiety. A
factor analysis revealed that items measuring general discomfort, difficulty concentrating,
sweating, nausea, and vertigo loaded significantly on the anxiety factor comprised
of items from the State-Trait Anxiety Inventory. Finally, a multiple regression indicated
that the items measuring general discomfort and difficulty concentrating significantly
predicted increases in anxiety. The overall results support the notion that side effects
associated with immersions in VR consist mostly of a nausea a nd an oculomotor latent
structure and that a few items are confounding anxiety and cybersic kness. The data
support the suggestion to revise the scoring procedures of the Simulator Sickness
Questionnaire when using this instrument with immersions in VR.
Keywords: simulator sickness questionnaire, simulator sickness, cybersickness, trier stress social test, anxiety,
virtual reality
Bouchard et al. SSQ Factor Structure for VR
INTRODUCTION
Unwanted negative side effects induced by an immersion in
virtual realit y (VR) are not uncommon. Cobb et al. (1) and
Wilson (2) summarized the results of a comprehensive research
program conducted on 148 civilians and 75 non-civilians in
VR. Overall, they found that 20% of their civilian participants
did not notice any side effects during an immersion in VR.
Among the remaining participants, only a few (5% of the
total sample) experienced side effects severe enough that they
had to stop the immersion. The other participants reported
side effects t hat, usually, were mild, occurred within the first
15 min of the immersion, and subsided within 10 min after
the immersion. These side effects are often referred to in the
popular literature as cybersickness, although people experiencing
them are not actually sick. In a litera ture review on the topic,
Lawson et al. (
3) concluded that about 5% of users immersed
in VR will report symptoms that are significant enough to
warrant stopping the immersion, about 5% will not experience
any symptoms at all, and the remaining users (between 70
and 90%) may experience some mild symptoms caused by the
immersion in VR. Lawson (
4) reported that between 50 and
100% of users immersed in VR experienced some dizziness
and between 20 and 60% of users experienced some abdominal
symptoms. The frequency of other symptoms appeared less
documented but included oculomotor problems. These findings
are consistent with Cobb et al. (1), Nichols and Patel (5),
and Stanney and Kennedy (6). Similarly, Sharples et al. (7)
found that 60 to 70% of participants reported an increase in
unwanted VR-induced symptoms from pre- to post-immersion.
Several more recent studies (8–11) and a systematic review
(12) confirmed that many participants experience unwanted
negative side effects induced by immersions in VR, even
when current generation VR head-mounted displays (HMDs)
are used.
According to Kennedy et al. (13), the temporary negative
side effects associated with immersion in VR include general
discomfort, difficulty focusing, increased salivation, sweating,
fullness of head, stomach awareness, and burping and are
regrouped into three factors: (1) oculomotor problems (i.e.,
eyestrain, blurred vision, headache), (2) disorientation (i.e.,
vertigo, imbalance), and (3) nausea (i.e., vomiting, diz ziness).
The ocular problems are lik ely related to the display system,
such as wearing the HMD (e.g., the HMD is too heavy or
too tightly strapped to the head) or the eyestrain of looking
for a long period of time at computer monitors that are
located at a closed and fixed distance from the eyes. The
nausea and the disorientation side effects are temporary and
often associated with motion sickness symptoms. The most
often reported explanation for symptoms of nausea is a conflict
between information provided by the otolith organs (linear
acceleration of the head), the semic ircular canals (angular
acceleration of the head), the visual system (position and
Abbreviations: HMD, Head mounted display; SSQ, Simulator Sickness
Questionnaire; TSST, Trier Stress Social Test; TSST-G, Trier Stress Social
Test in Groups; VE, Virtual environment; VR, Virtual reality.
orientation of the body with respect to the visual environment),
and the kinesthetic system (limb and body position) (
14–
19). The sensory conflict t heory is not without its critics e.g.,
(20, 21) and other theories may explain some of the nausea
symptoms, such as difficulty maintaining postural stability in
virtual environments (VEs) (
21, 22). For the disorientat ion
symptoms, t h e most important symptoms, as measured by
the Simulator Sickness Questionnaire [SSQ; (13)], are dizziness,
blurred vision, difficulty focusing, and nausea. Lawson et al.
(3) and Lawson (4) mentioned a fourth cluster of symptoms
described as the Sopite syndrome. The Sopite syndrome is a
form of motion sickness manifesting it self solely by signs of
fatigue (drowsiness, difficulty concentrating, and apathy). It
is possible that this syndrome involves the vestibular system.
Factors associated with the Sopite syndrome after an immersion
in VR remain poorly understood (3, 23, 24).
The SSQ (
13), mentioned above, i s the most widely used
measure to assess simulator sickness and unwanted negative side
effects following immersions in VR. The SSQ was developed
following the notion that symptoms experienced in flight
simulators are similar to sickness symptoms caused by traveling
(“kinetosis” or “naupathia”), but tend to be less severe, to have a
lower incidence (25), and to be more associated with the visual
system and the atypical interaction among the visual, vestibular,
and proprioceptive systems (26, 27). Sixteen symptoms are
listed in the SSQ, and their severity is rated from “0” (“None”)
to “3” (“Severe”). The SSQ was conceived for i mmersions in
various simulators, including those frequently used in VR,
such as HMDs and CAVE systems. The factor structure of
the SSQ is based on Lane and Kennedy’s (
28) and Kennedy
et al.’s (13) study on a sample of 1,119 military participants
who were immersed in a variety of Navy simulator training
exercises. The researchers wanted to determine which symptoms
demonstrated systematic changes before a nd after the virtual
immersions, and thus administered the SSQ before and after
each immersion. After performing a principal factor analysis
with a Varimax rotation and comparing factor solutions from
three to six f actors, and a hierarchical analysis to produce and
validate a general measure that avoids problems of collinearity,
a th ree-factor solution was identified: (1) oculomotor (i.e.,
general discomfort, eyestrain, difficulty concentrating, blurred
vision, difficulty focusing, etc.), (2) disorientation (i.e., dizziness,
vertigo, blurred vision, difficulty focusing, nausea , etc.), and
(3) nausea (i.e., nausea, burping, increased salivation, general
discomfort, difficulty concentrating, etc.). This factor structure
of the SSQ has been widely used since then e.g., (
3, 23). Other
instruments have also been developed over the years, such
as the Nausea Profile (29) and the Virtual Reality Symptom
Questionnaire (30), but they remain far less popular than
the SSQ.
Two themes of discussion are emerging in the literature
regarding side effects induced by immersions in VR and their
assessment with the SSQ: (1) the factorial structure of the
SSQ, when immersive technologies differ from flight or driving
simulators, and with users from the general population, and
(2) t he potential confound with anxiety symptoms and those
measured by the SSQ.
Frontiers in Psychiatry | www.frontiersin.org 2 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
As noted by Kennedy et al. (13), many items of t he SSQ
contribute significantly to more than one factor, which is a
problem according to classical factor analytic approaches (31, 32).
Five items are scored on two different subscales and, following
Kennedy et al.’s (
13) scoring procedure, are thus scored twice in
the calculation of the total score. The items “general discomfort”
and “difficulty concentrating” were assigned to both the Nausea
and Oculomotor subsc ales, the items “difficulty focusing” and
“blurred vision” were assigned to both the Oculomotor and
Disorientation subscales, and “ nausea” was assigned to both
the Nausea and the Disorient ation subs cales. Items scored
twice are essentially given more weight to the total score than
other negative unwanted side effects. Multiplying the results
by a constant during the scoring process further impacts the
intuitiveness of the interpretation of the results.
Given the slightly blurred factor structure of the SSQ,
Bouchard et al. (
33) sug gested that a two-factor solution is
probably more adequate than the three-factor solution proposed
by Kennedy et al. (13). The difference in factor structure may
be due to differences in the type of simulations (i.e., flight
simulators vs. HMD used in the treatment of phobias) or
the populations for which the SSQ was originally developed
(i.e., physically fit military personnel vs. general public). The
revised factor structure of the SSQ proposed by Bouchard et
al. (33) questions whether disorientation represents a relevant
and distinct group of symptoms, and has clinical and practical
implications (3 4).
When using the SSQ in a clinical context, such as with
people with mental illness or for physical rehabilitation, many
researchers have also noted an overlap between symptoms
of anxiety a nd unwanted side effects of immersions in VR
e.g., (
35–40). In one study, Pot-Kolder et al. (41) found that
anxiety partially mediated nausea and disorientation symptoms.
When using VR with clinical populations, or when inducing
anxiety in VR to confront feared stimuli, anxiety may better
explain symptoms such as sweating, discomfort, or fatigue
than immersion in VR per se. Given the increasing use of
VR in mental health applications, Bouchard et al. (35) studied
unwanted negative side effects induced by immersions in
VR in a clinical sample of 157 adults diagnosed with an
anxiety disorder. They found that 8 0% of their participants
reported no or only a few mild symptoms. Interestingly, many
symptoms usually associated with immersions in VR were
already present before the immersion [for an example with
another clinical population, see also (42)]. This sug gests that the
symptoms may have been anxiety-related instead of VR-related,
as participants were already experiencing them pre-immersion.
However, their methodology did not allow to discriminate
whether the symptoms were caused by anxiety experienced
during the immersions or to the immersion per se. To clarify
the specific role of immersions in VR and anxiety on the SSQ
and its factors, it would be necessary to induce anxiety without
immersing people in VR and assess the relationship between
anxiety and the symptoms of unwanted negative side effects
measured by the SSQ.
The current article reports on two studies questioning how the
SSQ is currently used to assess unwanted negative side effects of
immersions in VR: (1) testing the adequacy of the proposed two-
factor structure of the SSQ, and (2) documenting the potential
confound of symptoms caused by anxiety.
STUDY 1
Materials and Methods
Two research questions were addressed about the factor structure
of the SSQ: (1) Can the three-factor structure found by Kennedy
et al. (
13) be replicated in a large sample of non-military adults
immersed in VR? and (2) Would a factor analysis confirm
the adequacy of a two-factor solution? To this end, a French-
Canadian translation of the 16-item SSQ was used (
33).
Participants
The convenience sample consisted of 876 adults (551 women, 324
men) recruited in previous studies from the general population
and suffering from an anxiety disorder (n = 346), gambling
disorder (n = 77), or he althy controls (n = 453) (for specific
details about the sample, see the list of references in the
Supplementary Material). Participants were recruited via local
universities’ networks, advertisements in local newspapers and on
social media, and referrals from clinicians. Clinical participants
received their diagnoses using a structured diagnostic interview
and healthy controls were screened for the absence of
anxiety disorders, gambling disorder, psychotic disorder, and
schizophrenia. Among the 346 anxious participa nts, the most
frequent diagnosis was specific phobia, followed by social phobia,
generalized anxiety disorder, panic disorder with agoraphobia,
post-traumatic stress disorder (PTSD), and obsessive-compulsive
disorder. The mean age of the total sample was 34.88 (SD = 13.27,
range between 15 and 76).
Procedures
The participants completed the SSQ before and immediately after
immersions in VR, and only post-immersion questionnaires were
used in the current study. In the case of participants immersed
more than once in VR, only their first immersion was included
in the statistical analyses t o avoid multicollinearity. Participants
were immersed in VR with different technologies (i.e., HMD,
CAVE-like), in a variety of VEs (clinical participants were
immersed in VEs created for the treatment of anxiety disorders or
gambling disorder), had to perform different tasks (i.e., exposure
to feared stimuli, exploration and navigation, attention), and
were immersed for different durations (immersions lasted
between 5 and 60 min). For more details about the virtual
environments and the tasks t hat participants completed, see the
list of references in the Supplementary Material. Such variety
in procedures favors the generalization of the results. The UQO
Research Ethics Board approved the project and participants
had to remain in the waiting room 15 min after the immersions
before leaving the laboratory. While in the waiting room, they
received a handout describing what cybersickness is and provided
with contact information in the event that they experienced
after-effe cts or prolonged side effects after the studies.
Frontiers in Psychiatry | www.frontiersin.org 3 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
FIGURE 1 | Final confirmatory factor analysis model of the Simulator Sickness Questionnaire with two factors observed in adults after an immersion in virtual reality.
Measures
Structured Clinical Interview for DSM-IV [SCID-IV; (
43)]. This
is a semi-structured interview used to screen all participants and
diagnose mental disorders according to DSM-IV criteria (44).
Although the DSM-5 (45) has been published after the start of
the data collection, and the SCID-5 much later, all participants
from t he study listed above meet DSM-5 diagnostic criteria.
Simulator Sickness Questionnaire (
13). The 16-item SSQ
was used to assess participants’ sickness levels before and
after immersions in VR. Participants rated the se verity of
each symptom (e.g., dizziness, headache, sweating) on a 4-
point Likert scale (0 – “None” to 3 – “Severe”). To obtain a
total score, all raw items were summed (the items were not
weighted with Kennedy et al.’s formula to avoid inflating total
Frontiers in Psychiatry | www.frontiersin.org 4 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
scores, as suggested by) (33). A higher total score reflects more
severe sickness.
Results
The ratio of participants per variable was 55 to 1, confirming
that the sample met basic assumptions and criteria to perform
confirmatory f actor analyses. Only participants with no missing
data on all items were included in the analyses (nine participa nts,
or 1% of the initial sample, were excluded because AMOS
is unable to provide some fit indices when there are missing
data). Maximum likelihood estimation was used for structural
equation modeling and modification indices as well as a global
appraisal of tradit ional indexes and their critical values were
used, as suggested by Byrne (
46), Tabachnick and Fidell (32),
and Arbuckle (47): CFI (>0.90), PCFI (>0.75 ), GFI > 0.90
and RMSEA (<0.07). The statistical significance of the chi-
square is reported but should not be used given the known
limitation of this index with large samples (
32, 46). AIC and
BIC indices are reported to compare the models. The individual
values of these indices are not meaningfully interpretable per se,
as they are affected by sample size and other factors, but can
be used to compared models estimated with the same sample
(48). The smaller th e value, the better. Differences in AIC values
larger than 7 are considered very important, and differences
larger than 10 suggest the model with the highest value is not
supported (48).
To test t he three-factor structure wit h five items associated
to more than only one factor as proposed by Kennedy et
al. (
13), a confirmatory factor analysis was performed
using AMOS 27.0 (47). The fit indices did not support
the three-factor model proposed by Kennedy et al. (13):
χ2 (96) = 823.23, p < 0.001, CFI = 0.84, PCFI = 0.67,
GFI = 0.89, RMSEA = 0.09, RMR = 0.019. The AIC
(903.23) and BIC (1 094. 25) for th is model are reported
to compare the three- and two-factor solutions (see
below). Inadequate fit was also found when testing the
model only with clinical or non-clinical participants (see
Supplementary Material for the results), or after c onsidering
modification indices.
To test the adequacy of the two-factor model proposed by
Bouchard et al. (
33) with items loading solely on one factor, a
confirmatory factor analysis was performed using AMOS 27.0
(47), with maximum likelihood estimation. The final structural
equation model is presented in Figure 1, where circles represent
latent variable and rectangles represent me asured variables (Q
stands for Question or item number; e stands for error). The
plausibility of the two-factor solution was confirmed by the fit
indices (CFI = 0.89, PCFI = 0.74, RMSEA = 0.07, χ2 (100) =
578.15, p < 0.001), the examination of the modification indices,
the low value of the RMR (0.016), and a strong percentage of
variance explained (GFI = 0.92). Items with correlated errors
are items assessing dizziness with eyes open and dizziness with
eyes closed, difficulty focusing and blurred vision, and headache
and fullness of head. A comparison between the two-factor and
the original three-factor solutions confirmed that the two-factor
solution was more parsimonious, based on the AIC (AIC =
650.15, vs. 903.23 for the 3-factor solution) and th e BIC (BIC =
822.06 vs. 1 094.2 5 for the 3-factor solution) criteria. Smaller AIC
and BIC values indicate higher parsimony (
32).
The mean SSQ-Total raw score in the current sample was 4.38
(SD = 5.07, range between 0 and 42). The mean subscale scores
were 1.7 8 (SD = 2.79, range between 0 and 25) for the Nausea
factor and 2.60 (SD = 2.88, range between 0 and 17) for the
Oculomotor factor. The Cronb ach’s Alpha for the entire scale
was 0.87.
In summary, the aims of Study 1 were to test if the three-
factor structure of the SSQ found by Kennedy et al. (
13) was
replicable in a large sample of non-milit ary adults immersed in
VR and confirm if a two-factor st ructure was more adequate. The
results indicated that Kennedy et al. (13) three-factor structure
of the SSQ could not be replicated in a sample of non-military
adults immersed in VR. Instead, they suggested that a two-factor
structure is more parsimonious in this sample and favored a
factor solution where each item loads only on one factor.
STUDY 2
Materials and Methods
The aim of Study 2 was to document the potential confound of
symptoms caused by anxiety. The SSQ was administered before
and after an anxiety-inducing task conducted in vivo, the Trier
Social Stress Test in groups [TSST, (
49) TSST-G, (50)], with
no immersion in VR. The TSST-G is a validated standardized
stressful procedure (49, 50). If the SSQ specifically measures
unwanted negative side effects induced by immersions in VR,
scores on the SSQ should not be signific antly influenced by the
TSST-G t ask nor correlate significantly with symptoms of anxiety.
Participants
A total of 91 adult (18 to 30 years old, M = 23.81, SD = 3.87)
were recruited as part of other studies using the TSST-G. The first
42 participants were recruited for a study on facial expressions
conducted by Daudelin-Peltier et al. (
51). The remaining 49
participants were recruited for a subsequent study on v isual
perception from the same lab and matched for gender and
age. In both studies, procedures regarding recruitment, sample
characteristics, administration of the SSQ, and experimental
manipulations with the T SST-G were identical. The methodology
is reported in detail in Daudelin-Peltier et al. (51), and relevant
information is summarized below. The final sample consists of
88 participants with no missing data on the SSQ (two participants
did not show up and one participant had 43% of missing data).
Procedures
The UQO Research Ethics Board approved the project,
participants provided their informed consent, and the SSQ was
administered before and after the TSST-G (
50) conducted in vivo
(i.e., not in VR).
The TSST is essentially a social performance task to induce a
subjective and physiological stress response (49). The three main
components of the traditional TSST are: (1) an at mosphere of
high performance, (2) the task induces socio-evaluative threat,
and (3) participants have no control over the events (
49, 50, 52).
Participants completed a group TSST-G [i.e., three participants
Frontiers in Psychiatry | www.frontiersin.org 5 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
TABLE 1 | Correlation of SSQ items with the State Anxiety scale.
SSQafter the TSST-G State Anxiety after the TSST-G
Item 1 - General discomfort 0.67***
Item 2 - Fatigue 0.11
Item 3 - Headache −0.07
Item 4 - Eyestrain 0.15
Item 5 - Difficulty f ocusing 0.45***
Item 6 - Increased salivation 0.22*
Item 7 - Sweating 0.40***
Item 8 - Nausea 0.25*
Item 9 - Difficulty concentrating 0.49***
Item 10 - Fullness of head 0.48***
Item 11 - Blurred vision 0.29**
Item 12 - Dizzy (eyes open) 0.26*
Item 13 - Dizzy (eyes closed) 0.09
Item 14 - Vertigo 0.28**
Item 15 - Stomach awareness 0.24*
Item 16 - Burping 0.130
SSQ-Total Raw 0.60***
N = 88.
*p < 0.05, **p < 0.01, ***p < 0.001.
completed the tasks in turn in the same room with dividers
separating them; see (
51) for details]. Participants were called i n
front of an interview panel and had to: (a) give a speech on a
topic outside of their comfort zone for t hree min per participant
(for a total of nine min), and (b) perform an arithmetic task out
loud for three min per participant (for a total of nine min). The
two members of the interview panel displayed an emotionally
neutral attitude and provided minimal feedback, except to ask
participants to continue their presentation if t hey stopped before
their three min and to start over the arithmetic task when
they made an error. Anxiety and unwanted negative side effects
induced by an immersion in VR were measured before and after
the TSST-G.
To circumscribe the effect observed on the anxiety measure
to the stressful nature of the TSST-G, as opposed to general
characteristics of the TSST-G or the visit in the lab, a Control task
was also used. In the Control task, participants were in the same
room as for the TSST-G, re ad non-stressful magazines for nine
min and counted out loud, starting from zero and at a pace of
about one number per second, for three min each (for a total of
nine min). The order in which participants completed the TSST-
G and the Control task was randomly assigned. All measures
were also administered before and after the Control task. Data for
the Control task were analyzed only to confirm th e experimental
induction of anxiety with the TSST-G. The main analyses were
performed using only T SST-G data.
Measures
In addition to the French version of the SSQ (see Study 1),
participants completed the French version (
53) of the State
Anxiety scale from the State-Trait Anxiety Inventory (
54). The
State Anxiety scale of the State-Trait Anxiety Inventory consists
of 20 items, each assessing how the participant feels right now
on a scale from 1 (“Almost Never ” ) to 4 (“Almost Always”). After
reverse scoring relevant items, all items are summed. A higher
total score expresses more anxiety.
Results
The statistical analyses were performed using SPSS 27.0 (
55).
When reporting total scores for the SSQ, the scoring procedure
followed re commend ations by Bouchard et al. (33): the tot al non-
weighted score (i.e., SSQ-Total raw) was calculated by adding all
16 items of the SSQ only once and not multiplying the total by
a const a nt. Prior to performing the analyses, assumptions were
assessed using the SPSS Explore, Frequencies, and Regression
functions for each of the variables used in further analyses. The
assumption of normality was not met and there were univariate
outliers. The univariate outliers (z = ± 3) were winsorized to the
next most extreme score for the item in question. For example, if
a participant had a z-score of 3 for one item with a corresponding
raw score of 4, their raw s core would be winsorized to the
next most extreme raw score for that item (e.g., 3) by finding
the next most extreme z-score (e.g., 2). Although this corrected
extreme scores, the assumption of normality was still not met. As
such, non-parametric analyses were performed with and without
the univariate outliers, and the results did not significantly
change. Therefore, the results of the non-parametric analyses
with the univariate outliers not corrected (i.e., not winsorized)
are presented here. Although there were multivariate outliers as
shown by Mahalanobis dist ances above 20.09 [χ2 (8) = 20.09],
they were included in the statistical analyses as our a im was to
examine whether side effects induced without immersions in VR
overlap wit h anxiety.
As a manipulation check, the impact of the Control task
and the TSST-G on the State Anxiety scale were analyzed. In
the Control task, a Wilcoxon signed-ranks test showed that
participants’ scores on the State Anxiety scale significantly
decreased after completing the task [Z(88) = −2.40, p = 0.016],
from a mean of 27.69 (SD = 6.76) to 26.52 (SD = 5.94). After
participants performed the TSST-G, their scores significantly
increased [Z(88) = 6.65, p < 0.001], from a mean of 27.11 (SD
= 6.16) to 35.22 (SD = 12.18), indicating that the procedure
successfully induced anxiety. For the SSQ-Total raw scores in the
Control condition, a Wilcoxon signed-ranks test indicated that
participants’ scores significantly increased after completing the
task [Z(87) = 4.09, p < 0.001], from a mean of 2.86 (SD = 2.67)
to 4.30 (SD = 3.49).
Overlap With Anxiety
To examine the relationship between anxiety and symptoms
measured by the SSQ, the Spearman correlation between the total
state anxiety post-TSST-G and items of the SSQ post-TSST-G is
reported in Table 1. Results revealed that only five SSQ items
did not significantly correlate with anxiety post-T SST-G. The
eleven items that significantly correlated with anxiety were evenly
distributed between the Nausea ( items 1, 6, 7, 8, 14, and 15) and
Oculomotor (items 5, 9, 10, 11, and 12) factors. After applying
a Bonferroni correction (0 .05/16 = 0.003), correlations for items
number 1, 5, 7, 9 and 10 remain statistically significant.
Frontiers in Psychiatry | www.frontiersin.org 6 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
Another statistical approach to address the overlap between
SSQ scores and anxiety is to perform a factor analysis to see how
items aggregate on common or distinct f actors. The reasoning
behind the statistical analyses was that the items of the State
Anxiety scale and the SSQ should theoretically be orthogonal and
load on two distinct factors. Items of the SSQ significantly loading
on the anxiety factor would show that these symptoms were
associated with anxiety, or at least not specifically representing
side effects of immersions in VR. A principal component analysis
with a Varimax rotation was conducted on all post-TSST-G items
of the State Anxiety scale and the SSQ to determine the overlap
between a nxiety and VR-induced side effects. The Kaiser-Meyer-
Olkin measure of sampling adequacy was.82 and Bartlett’s test
of sphericity was significant [χ2 (630) = 2024.55, p < 0.001],
suggesting that it was appropriate to conduct a factor analysis
on this dataset. The ratio of participants per variable was low
(two participants per variable), which led to the decision to
be conservative and consider only loadings higher than 0.50 as
significant. The number of factors to extract was forced to two,
with the expectation that items of each questionnaire would
load on their respective factor with minimal cross-loadings. The
eigenvalues of the first and second factors were 12.59 (34.97% of
unique variance) and 2.91 (8.08% of unique variance), followed
by six other potential factors with eigenvalues greater than one
and accounting for a portion of unique variance ranging from
6.28% to 3.01%. Examination of the rotated loading matrix
revealed that two items of the SSQ loaded above 0.50 on the State
Anxiety factor: (1) item 1– General discomfort (cross-loading =
0.69), and (2) item 9 – Difficulty concentrating (cross-loading =
0.53). Three additional items had loadings approaching 0.50 on
the State Anxiety factor: item 7 – Sweating (cross-loading = 0.46),
item 8 – Nausea (cross-loading = 0.48), and item 14 – Vertigo
(cross-loading = 0.49). All these loadings were stronger on the
anxiety factor than on the SSQ factor (item 9 had a loading of 0.43
on the SSQ factor) and evenly distributed between the Nausea and
Oculomotor factors reported in Study 1.
Overlap With Increase in Anxiety
The intensity of symptoms assessed by the SSQ-Total raw was
low before the TSST-G, with a mean of 2.98 (SD = 2.73), and
significantly increased [Z(88) = 3.96, p < 0.001] to 5.01 (SD =
4.51) post-TSST-G. Wilcoxon signed-ranks tests were conducted
for all SSQ items pre- and post-TSST-G to examine which ones
had significantly increased after performing the TSST-G in vivo.
The results indicated that items number 1 [Z(88) = 5.02, p <
0.001], number 3 [Z(88) = 2.32, p = 0.020], number 5 [Z(88) =
3.98, p < 0.001], number 7 [Z(88) = 2.91, p = 0.004], number
9 [Z(88) = 4.48, p < 0.001], number 10 [Z(88) = 2.04, p =
0.042], and number 11 [Z(88) = 2.33, p = 0.020] significantly
increased post-TSST-G. After applying a Bonferroni correction
(0.05/16 = 0.003), increases on items 1, 5, and 9 remained
statistically significant.
To analyze increase in anxiety and increase in SSQ items
while maintaining sufficient statistical power, the next statistical
analysis focused on the seven symptoms measured by the SSQ
that changed following the TSST-G. Standardized pre-post TSST-
G residuals for each item and for the State Anxiety scale were
TABLE 2 | Multiple regression predicting (residualized) increase in State Anxiety
score with (residualized) change observed on selected items of the SSQ before
and after a TSST-G conducted in vivo.
SSQ Items
(residualized
change
scores)
β t p sr 95% Confidence intervals
Item 1 –
General
Discomfort
0.45 4.53 0.000 0.35 0.25 0.65
Item 3 –
Headache
−0.10 −1.20 0.235 −0.09 −0.26 0.07
Item 5 –
Difficulty
focusing
0.15 1.36 0.179 0.11 −0.07 0.38
Item 7 –
Sweating
0.11 1.23 0.222 0.10 −0.07 0.30
Item 9 –
Difficulty
concentrating
0.24 2.21 0.030 0.17 0.02 0.46
Item 10 –
Fullness of
head
−0.05 −0.50 0.620 −0.04 −0.27 0.16
Item 11 –
Blurred vision
−0.08 −0.74 0.461 −0.06 −0.28 0.13
N = 88.
used as change scores. A multiple regression was performed
with the residualized change pre/post TSST-G State Anxiety
scale as a dependent variable and the seven residualized change
pre/post TSST-G SSQ scores as predictors. The assumptions
of linearity, absence of multicollinearity (Tolerance and VIF
statistics), homoscedasticity, and independence of the residuals
(Durbin-Watson st ati stic) were met. No influential cases biasing
the model were found (Cook’s Distance values were all
under 1). The seven-predictor model accounted for 51% of
the variance in change in anxiety [F(7, 80) = 12.09, p <
0.001, adjR
2
= 0.47]. Results for e ach item are reported in
Table 2. Only change in general discomfort (SSQ item 1)
and difficulty concentrating (SSQ item 9) made a significant
unique contribution to the change in State Anxiety following
the TSST-G. Interestingly, performing the same analysis to
predict scores of anxiety post TSST-G, instead of increase in
anxiety, reve aled that item #7 became a significant predictor
(β = 0.17, t = 2.00, p = 0.049, sr = 0.15) and item #9
became non-significant (β = 0.18, t = 1.72, p = 0.090, sr
= 0.13).
In summary, Study 2 documented the potential confound of
symptoms c a used by anxiety when assessing unwanted negative
side effects of immersions in VR. A variety of statistical strategies
were used, and two items systematically stand out as potentially
problematic. General discomfort and difficulty concentrating can
very well convey the notion of “cybersickness”, but they are
also very likely to be influenced by anxiety. The item measuring
sweating was also recurrently high lighted as influenced by
anxiety but predicted an increase in anxiety to a lesser extent than
other items.
Frontiers in Psychiatry | www.frontiersin.org 7 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
DISCUSSION
Immersions in VR can lead to unwanted side effects and, as the
applications of this technology grow, the need to monitor their
symptoms is important. The most popular self-administered tool
to assess unwanted negative side effects following immersions
in VR is the SSQ (13). This tool has a very strong track record
and is practical for professionals using VR for mental health
applications, among others. However, simulator sickness (i.e.,
induced by the actual movement of the simulator) and VR-
induced side effects (i.e., induced by visual information without
matching vestibular information and specifically experienced in
VR) are relative ly distinct phenomena (56). As such, using the
SSQ for immersions in VR, as opposed to flight simulators, and
with users from the general population has raised a few questions
in recent years e.g., (57). One of these questions is whether the
symptoms measured by the SSQ measure two distinct latent
dimensions [a Nausea factor and an Oculomotor factor; (33)] or
also additional dimensions such as disorient a tion (
13). A second
question is whether anxiety induced by the immersion may
contaminate symptoms measured by the SSQ in applications in
the trea tment of anxiety and other psychological disorders [e.g.,
chronic pain, (42)]. If this is the case, some symptoms should not
be attributed to VR technology, but as a normal consequence of
the t h erapeutic use of VR, such as in virtuo exposure for anxiety
disorders (58), relaxation in VR, or vestibular rehabilitation
e.g., (59).
Results from Study 1 do not support the three-factor model
proposed by Kennedy et al. (13). Results of confirmatory
factor analyses clearly revealed that, with a sample of civilians
immersed in a wide variety of technologies, the SSQ essentially
measures two distinct latent dimensions: nausea symptoms and
oculomotor symptoms. This finding has practical implications
for scoring the SSQ. According to Kennedy et al. (
13), the total
score is calculated after adding the score of the items from each
factor, summing the three factors, and multiplying the result by
a constant. Since five items (i.e., 1, 5, 8, 9, 11) load on more
than one of the three factors, these items have twice the weight
on the total score than the others and the latent dimensions
of th e SSQ overlap with each other. As mentioned in (33, 35 ),
this scoring method is not intuitive, and many researchers do
not report if they followed it or not when calculating the total
score of the SSQ. Recommendations about the scoring of the
SSQ will be summarized in the Conclusion, but arguments are
strong in favor of considering that the SSQ comprises only
two f actors. At face value, it is surprising to allocate symptoms
of nausea to a Disorientation factor when it is the defining
feature of the Nausea factor, or to allocate blurred vision to
the Disorientation factor when it is also in the Oculomotor
factor. Empirical ana lyses reported previously (
33) and in Study
1 strongly support the relevance of Nause a and Oculomotor
dimensions found by Kennedy et al. (13), but not the model
that includes a Disorientation factor. Differences in applications
and populations may explain the difference in factor structures.
The SSQ was meant to be used with training simulators with
physically fit military personnel, not for immersions in VR with
users from the general population. As pointed out by Lawson et
al. (
3), military personnel may be less likely to experience negative
side effects since they are more likely to be frequently involved in
challenging vehicle motion, in better physical shape, or able to
remain immersed in VR longer despite feeling unwanted effects.
Stanney et al. (
56) also suggested that negative side effects differ
between immersion in VR and in training simulators. The design
of the VE and the task performed are known to have an impact
on the induction of negative side effects (3, 19, 60). Replicat ions
of our results in other centers, with diverse populations and
methodologies, remain warranted.
Results from Study 2 confirm the strong correlation between
scores on the SSQ and anxiety, but most importantly, they show
a significant association between anxiety and many symptoms
measured by the SSQ despite the fact that symptoms were
not induced by an immersion in VR. Only five symptoms did
not significantly correlate with anxiety after a task performed
in vivo: fatigue, headache, eyestrain, dizziness when eyes are
closed, and burping. If one wants items likely to specifically
measure unwanted negative side effects of an immersion in VR,
these five items are the most likely candid a tes. O ther items of
the SSQ that correlate with anxiety are not to be discarded
too rapidly, as various analytical approaches in Study 2 yielded
slightly different c onclusions. Study 2 examined increases on
each item after a stressful task conducted in vivo and, after
controlling for th e number of comparisons, items measuring
general discomfort and difficulty concentrating clearly incre ased
when they should not. These two items were also among those
loading strongly on the anxiety factor constituted by all items
of the State Anxiety scale. A multiple regression supported the
finding while controlling for shared variance. The SSQ item
measuring sweating was correlated with anxiety after the stressful
task conducted in vivo, but not with increase in anxiety. Other
items may seem problematic in some analyses but not in others.
Limitations
The methodologies used in the article are not without limitations.
The sample used to examine the factor structure of the SSQ
is heterogeneous. Participants vary in age, gender, the presence
or absence of anxiety disorders (see Supplementary Material
for analyses performed separately), tasks performed in VR,
hardware, and software. We consider such a variety a strength in
terms of external validity and potential for generalization of the
results. However, t he consequence is t h e impossibility to isolate
specific factors that can influence the findings. As the majority
of the sample is aged between 20 and 50, results may be less
relevant for children, adolescents and older adults. Replications
are therefore required. We encourage researchers to publish
factor analyses on samples that may limit the generalization of
results but would provide information specific to populations,
tasks, and hardware. A systematic approach examining human
factors e.g., (
1, 7, 19) is strongly recommended. Documenting the
factor structure of the SSQ with children and older adults should
also be done.
It is possible to crit icize Study 2 by arguing that testing the
SSQ without an immersion in VR is irrelevant because the SSQ
was meant to be used in the context of immersions in VR or
with simulators. Howe ver, the correlation between anxiety and
Frontiers in Psychiatry | www.frontiersin.org 8 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
unwanted side effects of immersions in VR had already been
documented by several independent groups after immersions in
VR e.g., (35–37, 39–41). Documenting symptoms that should be
induced by immersions in VR occurring in contexts where there
is no VR challenges the construct validity of th e instrument, or
at least some of its items. The ideal study should: (a) be designed
exclusively for th e purpose of documenting the potential overlap
between unwanted side effects of immersions in VR and anxiety;
(b) have participants perform an anxiety induction task such as
the TSST, in vivo, in VR, and in a control condition to isolate
all potential confounds; (d) have a s a mple large enough to benefit
from sufficient statistical power and variance to compare pre/post
residualized changes among the three experimental conditions;
and (e) include participants with and without an anxiety disorder,
or other relevant clinical conditions such as addictions or pain,
for example. The statistical approaches for Study 2 can also
be criticized as somewhat redundant because the analyses all
represent variations on the general lineal model (
31, 32). Indeed,
Study 2 was based on an exploratory approach. Based on the
findings from this article and the methodological suggestions
above, it is our hope that new studies will be designed to test
specific hypotheses emerging from our results.
As summarized below, our recommendation is not to
systematically remove the two items that are identified as
symptoms of anxiety. This can be criticized as a weak
recommendation. Why not remove these items altogether and
avoid potential confusion? These two items may measure lighter
or early symptoms of immersions in VR. Also, just as fever is a
common symptom of many medic al disorders, these symptoms
may target common factors between anxiety and unwanted side
effects of immersions in VR. General discomfort and difficulty
concentrating may be relevant to describe unwanted side effects
of immersions in VR, but they are not defining features. When
it is important to measure pathognomonic symptoms and avoid
false positives, it is better not to include these symptoms. It is
prudent to administer all 16 items of the SSQ until the relevance
of these two symptoms has been characterized and examined in
other contexts.
Conclusion
Considering the findings of these two studies, there is a need
to revise the SSQ when assessing unwanted negative side
effects following an immersion in VR. At the same time, it
is important to avoid confusion in the field and to facilitate
comparisons between studies and samples. We propose the
following recommendations for people interested in measuring
unwanted negative side effects in users of VR from the general
population: (a) administer all 16 original items of the SSQ
from Kennedy et al. (
13), before and after immersions in VR;
(b) report SSQ-Total raw scores by adding the scores from all
items, without giving more weight to some items than others;
(c) conceptualize the SSQ as an instrument with a two-factor
latent structure, with a Nausea factor (items 1, 6, 7, 8, 12, 13,
14, 15, and 16) and an Oculomotor factor (items 2, 3, 4, 5,
9, 10, and 11); and (d) when immersions involve reporting
anxiety or other clinical conditions report, in addition to the
above information, SSQ-Total-Anx, SSQ-Nausea-Anx, and SSQ-
Oculomotor-Anx scores that are scored without items 1 and 9
due to their important overlap with anxiety. We also recommend
that researchers independently replicate our results with different
populations (e.g., people with addictions, schizophrenia, acute
or chronic pain) and continue documenting the relevance of the
item measuring sweating.
The findings about the overlap with some symptoms of
anxiety must raise the awareness of experimenters, clinicians,
and users about the fact that they should not automatically
attribute post-immersion symptoms to VR, particularly if users
are exposed to an anxiety-inducing VE. To ensure public s afety,
unwanted negative side effects should be systematically
monitored before, during, and after immersions when
researchers are developing new VR tools and treatments. A
detailed usage protocol to inform users and reduce the likelihood
of unwanted side effects can be found in Stanney et al. (
61) to
guide professionals using VR in mental health applications.
DATA AVAILABILITY STATEMENT
The raw da ta supporting the conclusions of t h is manuscript will
be made available by the authors, without undue reservation, to
qualified researchers and for audits.
ETHICS STATEMENT
The studies involving human participants were reviewed
and approved by the Comité d’éthique de la recherche de
l’Université du Québec en Outaouais. The patients/participants
provided their written informed consent to participate in
this study.
AUTHOR CONTRIBUTIONS
SB contributed to the literature review, the methodology, data
collection and statistical analyses, and writing and revising
the manuscript. MB contributed to the literature review,
statistical analyses, and writing and revising the manuscript. GR
contributed to the methodology, data collection, and revising
the manuscript. PR contributed to the methodology and data
collection of Study 1 and revising the manuscript. HF, CD-P, CB,
and DF contributed to the methodology and data collec tion of
Study 2 and revising the manuscript. All authors contributed to
the article and approved the submitted version.
FUNDING
This project was supported by grants from the Canada Research
Chairs (#950-231039 and #950-210762), the Natural Science and
Engineering Research Council of Canad a (#50262982 ), and the
Social Sciences and Humanities Research Council of Canada
(#305205).
Frontiers in Psychiatry | www.frontiersin.org 9 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
ACKNOWLEDGMENTS
We wish to thank the undergraduate research assistants and
volunteers who helped with data collection and the IT te am
at the Cyberpsychology Lab of UQO for developing the
virtual environments. This article is a significantly updated
and significantly re v ised version of the manuscript E xploring
new dimensions in the assessment of vir tual reality induced side
effects with SB, GR, and PR as authors that was published
by a journal that is most likely a predatory journal. The
article cannot be found on the supposed journal site or any
other peer-reviewed journal online. The article is not indexed
in databases of published articles. The manuscript cited above
is considered as having never been published in a peer-reviewed
journal. The present a rticle has been rewritten, is based on new
data and meth odology, and constitutes an original contribution.
SUPPLEMENTARY MATERIAL
The Supplementary Material for this article can be found
online at: https://www.frontiersin.org/articles/10.3389/fpsyt.
2021.739742/full#supplementary-material
REFERENCES
1. Cobb SVG, Nichols SC, Ramsey AR, Wilson JR. Virtual reality
induced symptoms and effects. Presence-Teleop Virt. (1999)
8:169–86. doi: 10.11 62 /1 05 47 4 69 95 6 61 52
2. Wilson JR. Virtual environments and ergonomics: needs and opportunities.
Ergonomics. (1997) 40:1057–77. doi: 10.1080/001401397187603
3. Lawson BD, Graeber DA, Mead AM, Muth ER. Signs and symptoms of human
syndromes associated with synthetic experience. In: Handbook of Virtual
Environments: Design, Implementation, and Applications, eds. K. S. Hale, and
K. M. Stanney (Mahwah, New Jersey: Lawrence Erlbaum Associates). (2002).
p. 589–618.
4. Lawson BD. Motion sickness symptomatology and origins. In: Handbook of
Virtual Environments: Design, Implementation, and Applications, eds. K. S.
Hale, and K. M. S tanney, (Boca Raton, FL: CRC Press) (2015). p. 144–163.
5. Nichols S, Patel H. Health and safety implications of virtual
reality: a review of empirical evidence. Appl Ergon. (2002)
33:251–71. doi: 10.101 6/ S0 00 3- 6 87 0(0 2)0 00 20 -0
6. Stanney KM, Kennedy RS. Simulation sickness. In: Human Factors in
Simulation and Training, eds. P. A. Hancock, D. A. Vincenzi, J. A. Wise, and
M. Mouloua (Boca Raton, FL: CRC Press) (2009). p. 117–128.
7. Sharples S, Cobb S, Moody A, Wilson JR. Virtual reality induced
symptoms and effects (VRISE): Comparison of head mounted display
(HMD), desktop and projection display systems. Displays. (2008) 29:58–
69. doi: 10.10 16 /j .displa.2007.09 .00 5
8. Gavgani AM, Nesbitt KV, Blackmore KL, Nalivaiko E. Profiling subjective
symptoms and autonomic changes associated with cybersickness. Auton
Neurosci. (2017) 203:41–5 0. doi: 10.1016/j.autneu.2016.12.004
9. Munafo J, Diedrick M, Stoffregen TA. The virtual reality head-mounted
display Oculus Rift induces motion sickness and is sexist in its effects. Exp
Brain Res. (2017) 235:889–901. doi: 10.1007/s00221-016-4846-7
10. Rangelova S, Motus D, André E. Cybersickness among gamers: an online
survey. In: International Conference on Applied Human Factors and
Ergonomics, ed. T. Ahram (Cham, Switzerland: Springer, Cham) (2019) p.
192–201. doi: 10.10 0 7/9 7 8- 3- 03 0 -2 04 7 6- 1_2 0
11. Yildirim C. Don’t make me sick: investigating the incidence of cybersickness
in commercial virtual reality headsets. Virtual Real. (2019) 24:231–
9. doi: 10.100 7/ s10 0 55 -0 1 9- 00 4 01 -0
12. Caserman P, Garcia-Agundez A, Zerban AG, Göbel S. Cybersickness in
current-generation virtual re ality head-mounted displays: systematic review
and outlook. Virtual Real. (2021) 1–18. doi: 10.1007/s100 55 -0 2 1- 00 5 13 -6
13. Kennedy RS, Lane NE, Berb aum KS, Lilienthal MG. Simulator Sickness
Questionnaire: An enhanced method for quantifying simulator sickness. Int
J Aviat Psychol. (1993) 3:203–20. doi: 10.1207/s15327108ijap0303_3
14. Gallagher M, Ferrè ER. Cybersickness: a multisensory integration perspective.
Multisens Res. (2018) 31:645–74. doi: 10.1163/22134808-20181293
15. Harm DL. Motion sickness neurophysiology, physiological correlates and
treatment. In: Handbook of Virtual Environments: Design, Implement ation,
and Applications, eds. K. S. Hale, and K. M. Stanney (Mahwah, New Jersey:
Lawrence Erlbaum Associates,) (2002). p. 637–661.
16. Kemeny A, Chardonnet JR, Colombet F. Getting Rid of Cybersickness: In
Virtual Reality, Augmented Reality, and Simulators. London: Springer Nature.
(2020). doi: 10.100 7/9 7 8- 3- 0 30 -5 93 4 2- 1
17. Kim J, Luu W, Palmisano S. Multisensory integration and the experience of
scene instability, presence and cybersickness in virtual environments. Comput
Hum Behav. (2020) 113:106484. doi: 10.1016/j.chb.2020.106484
18. Reason JT, Brand JJ. Motion Sickness. Cambridge: Ac ademic Press. (1975).
19. Sharples S, Burnett G, Cobb S. Sickness in virtual reality. In: Advances
in Virtual Reality and Anxiety Disorders, eds. B. K. Wiederhold,
and S. Bouchard (New York, New York: Springer) (2014) p.
35–62. doi: 10.1007 /9 78 - 1- 48 99 - 80 23 -6 _3
20. Treisman M. Motion sickness: an evolutionary hypothesis. Science. (1977)
197:493–5. doi: 10.112 6/ science.301659
21. Riccio GE, Stoffregen TA. An ecological theory of motion
sickness and postural instability. J Environ Psychol. (1991)
3:195–240. doi: 10.120 7/ s15 3 26 96 9 eco0303_2
22. Bonnet CT, Faugloire E, Riley MA, Bardy BG, Stoffregen TA. Motion sickness
preceded by unstable displacements of the center of pressure. Hum Mov Sci.
(2006) 25:800–20. doi: 10.1016/j.humov.2006.03.001
23. Kennedy RS, Drexler J, Kennedy RC. Research in visually induced motion
sickness. Appl Ergon. (2010) 41:494–503. doi: 10.1016/j.apergo.2009.11.006
24. Koch A, Cascorbi I, Westhofen M, Dafotakis M, Klapa S, Kuhtz-Buschbeck
JP. The neurophysiology and treatment of motion sickness. Dtsch Ärztebl Int.
(2018) 115:687–996. doi: 10.3238/arztebl.2018.0687
25. Davis S, Nesbitt K, Nalivaiko E. A systematic review of cybersickness.
In: Proceedings of the 2014 Conference on Interactive Entertainment,
eds. K. Blackmore, K. Nesbitt, and S. P. Smith (New York,
New York: ACM) (2014). p. 1–9. doi: 10.1145/2677758.26
77780
26. Kennedy R, Drexler J, Compton D, Stanney K, Lanham D, Harm D.
Configural scoring of simulator sickness, cybersickness and space adaptation
syndrome: similarities and differences. In: Virtual and Adaptive Environments:
Applications, Implications, and Human Performance Issues, eds. L. J.
Hettinger, and M. Haas (Mahwah, New Jersey: Lawrence Erlbaum Associates
Inc.) (2003). p. 247–278. doi: 10.120 1/9 7 81 41 0 60 88 8 8.ch12
27. Rebenitsch L, Owen C. Review on cybersickness in applications and visual
displays. Virtual Real. (2016) 20:101–25. doi: 10.1007/s10055-016-0285-9
28. Lane NE, Kennedy RS. A new method for quantifying simulator sickness:
development and application of the simulator sickness questionnaire (SSQ)
(Report No. EOTR 88-7). Orlando, FL: Essex Corporation. (1988).
29. Muth ER , Stern RM, Thayer JF, Koch KL. Assessment of the multiple
dimensions of nausea: The Nausea Profile. J Psychosom Res. (1996) 40:511–
20. doi: 10.10 16 /0 02 2- 3 99 9(9 5)0 06 38 -9
30. Ames SL, Wolffsohn JS, McBrien NA. The development of
a symptom questionnaire for assessing virtual reality viewing
using a head-mounted display. Optom Vis Sci. (2005) 82:168–
76. doi: 10.10 97 /0 1.OPX.000 01 56 3 07 .95 08 6.6
31. Nunnally JC. Psychometric Theory, 2nd ed. New York: McGraw-Hill. (1978).
32. Tabachnick BG, Fidell LS. Using Multivariate Statistics, 5th ed. Boston: Allyn
and Bacon/Pearson Education. (2007).
Frontiers in Psychiatry | www.frontiersin.org 10 November 2021 | Volume 12 | Article 739742
Bouchard et al. SSQ Factor Structure for VR
33. Bouchard S, Robillard G, Renaud P. Revising the factor structure of the
simulator sickness questionnaire. Annu Rev CyberTherapy Telemed. (2007)
5:128–37. doi: 10.10 37 /t73 6 06 -0 0 0
34. Schmitt TA, Sass DA, Chappelle W, Thompson W. Selecting the “best” factor
structure and moving measurement validation forward: An illustration. J Pers
Assess. (2018) 100:345–62. doi: 10.1080/00223891.2018.1449116
35. Bouchard S, St-Jacques J, Renaud P, Wiederhold BK. Side effects of immersions
in virtual reality for anxious people. J Cyber Ther Rehabil. (2009) 2:127–37.
36. Diemer J, Lohkamp N, Mühlberger A, Zwanzger P. Fe ar and
physiological arousal during a virtual height challenge—effects in
patients with acrophobia and healthy controls. J Anxiety Disord. (2016)
37:30–9. doi: 10.101 6/ j.janxdis.2015.10.007
37. Ling Y, Brinkman WP, Nefs HT, Qu C, Heynderickx I. Cybersickness and
anxiety in v irtual environments. In: 2011 Joint Virtual Reality Conference, eds.
K. Helin, and M. D’Cruz (Espoo, Finland: VTT Technical Research Centre of
Finland) (2011). p. 80–82.
38. Quintana P, Bouchard S, Serrano Zárate B, Cárdenas-López G. Efectos
secundarios negativos de la inmersión con realidad virtual en poblaciones
clínicas que pade cen ansiedad. Rev de Psicopatol y Psicol Clin. (2014) 19:197–
207. doi: 10.594 4/ rppc.vol.19.num.3.2014 .13 90 1
39. Reger GM, Smolenski D, Edwards-Stewart A, Skopp NA, Rizzo AS, Norr
A. Does virtual reality increase simulator sickness during exposure therapy
for post-traumatic stress disorder? Telemed J E-Health. (2019) 25:859–
61. doi: 10.10 89 /tmj.2 01 8.0 17 5
40. Shiban Y, Diemer J, Brandl S, Zack R, Mühlberger A, Wüst S. Trier
Social Stress Test in vivo and in virtual reality: Dissociation of response
domains. Int J Psychophysiol. (2016) 110:47–55. doi: 10.1016/j.ijpsycho.201
6.10.008
41. Pot-Kolder R, Veling W, Counotte J, Van Der Gaag M. Anxiety partially
mediates cybersickness symptoms in immersive virtual reality environments.
Cyberpsychol Behav Soc Netw. (2018) 21:187–93. doi: 10.1089/cyber.201
7.0082
42. Brown P, Powell W. Pre-exposure cybersickness assessment within a
chronic pain population in virtual reality. Front Virtual Real. (2021)
2:672245. doi: 10.33 8 9/f rvir.2021.672245
43. First M, Spitzer R, Gibbon M, Williams J. Structured Clinical
Interv iew for DSM-IV Axis-I Disorders - Patient Version. New York:
Biometrics Research Department, New York State Psychiatric Institute.
(1996). doi: 10.103 7/t0 78 2 7- 00 0
44. American Psychiatric, Association. D i agnostic and Statistical Manual of
Mental Disorders, 4th ed. – Text Revision. Washington: American Psychiatric
Association Publishing. (2000)
45. American Psychiatric Association. Diagnostic and Statistical Manual of Mental
Disorders, 5th ed. Washington: American Psychiatric Association Publishing.
(2013) doi: 10.117 6 /appi.books.97808 9 04 25 5 96
46. Byrne BM. Structural Equation Modeling with EQS and EQS/WINDOWS:
Basic Concepts, Applications, and Programming. Thousand Oaks: SAGE
Publications. (1994).
47. Arbuckle JL. AMOS 27.0 User’s Guide. Chicago: SPSS Inc. (2020).
48. Burnham KP, Anderson DR. Multimodel inference: understanding
AIC and BIC in model selection. Sociol Metho Res. (2004)
33:261–304. doi: 10.11 77 /0 04 91 2 41 04 2 68 64 4
49. Kirschbaum C, Pirke KM, Hellhammer DH. The trier social stress
test - a tool for investigating psychobiological stress responses in a
laboratory setting. Neuropsychobiology. (1993) 28:76–81. doi: 10.1159/000119
004
50. von Dawans B, Kirschbaum C, Heinrichs M. The Trier Social Stress Test for
Groups (TSST-G): A new research tool for controlled simultaneous social
stress exposure in a group format. Psychoneuroendocrinology. (2011) 36:514–
22. doi: 10.10 16 /j .psyneuen.20 10 .08 .00 4
51. Daudelin-Peltier C, Forget H, Blais C, Deschênes A, Fiset D. The effect of acute
social stress on the recognition of facial expression of emotions. Sci Rep. (2017)
7:1–13. doi: 10.10 38 /s4 1 59 8- 0 17 -0 10 5 3- 3
52. Dickerson SS, Kemeny ME. Acute stressors and cortisol responses: a
theoretical integration and synthesis of laboratory research. Psychol Bull.
(2004) 130:355. doi: 10.1037/0033-2909.130.3.355
53. Gauthier J, Bouchard S. Adaptation canadienne-française de la version révisée
du State-Trait Anxiety Inventory de Spielberger. Can J Behav Sci. (1993)
25:559–78. doi: 10.103 7/ h007 88 8 1
54. Spielberger CD. Manual for the State-Trait Anxiety Inventory, (Form Y)
(Self-Evaluation Questionnaire). Palo Alto: Consulting Psychologist Press.
(1983). doi: 10.103 7/t0 64 9 6- 00 0
55. IBM Corp. IBM SPSS Statistics for Macintosh (Version 27.0). Armonk: IBM
Corp. (2020)
56. Stanney KM, Kennedy RS, Drexler JM. Cybersickness is not simulator
sickness. Proc Hum Factors Ergon Soc Annu Meet. (1997) 41:1138–
42. doi: 10.11 77 /1 07 11 8 13 97 0 41 00 2 92
57. Stone III WB. Psychometric evaluation of the Simulator Sickness Questionnaire
as a measure of cybersickness. [Ames (Iowa)]: Iowa State University. (2017).
58. Bouchard S, Côté S, R ichard DS. Virtual reality applications of
exposure. In: Handbook of Exposure Therapies, eds. D. S. Richard,
and D. Lauterbach (Cambridge, MA : Academic Press) (2006) p.
347–388. doi: 10.1016 /B9 78 -0 1 25 87 42 1 -2 /5 00 17 - X
59. Meldrum D, Herdman S, Vance R, Murray D, Malone K, Duffy D, et al.
Effectiveness of conventional versus virtual reality–based balance exercises
in vestibular rehabilitation f or unilateral peripheral vestibular loss: results
of a randomized controlled trial. Arch Phys Med Rehabil. (2015) 96:1319–
28. doi: 10.10 16 /j .apmr.2015 .02 .032
60. Rebenitsch L, Owen C. Estimating cybersickness from
virtual reality applications. Virtu al Real. (20 21 ) 25:165–
74. doi: 10.10 07 /s1 0 05 5- 0 20 -0 0 44 6- 6
61. Stanney KM, Kennedy RS, Kingdon K. Virtual environment usage protocols.
In Handbook of Virtual Environments: Design, Implementation, and
Applications, eds. K. S. Hale, and K. M. Stanney (Mahwah, New Jersey
Lawrence Erlbaum Associates) (2002). p. 721–730.
Conflict of Interest: SB is the President of, and owns equity in, Cliniques et
Développement In Virtuo, a spin-off from the university that uses virtual re ality
and distributes virtual environments. In addition, GR is the Vice-President
of Corporate Affairs of, and owns equity in, Cliniques et Développement In
Virtuo. The terms of these arrangements have been reviewed and approved
by the Université du Québec en Outaouais in accordance with its conflict of
interest policies.
The remaining authors declare that the research was conducted in the absence of
any commercial or financial relationships that could be construed as a potential
conflict of interest.
Publisher’s Note: All claims expressed in this article are solely those of the authors
and do not necessarily represent those of their affiliated organizations, or those of
the publisher, the editors and the reviewers. Any product that may be evaluated in
this article, or claim that may be made by its manufacturer, is not guaranteed or
endorsed by the publisher.
Copyright © 2021 Bouchard, Berthiaume, Robillard, Forget, Daudelin-Peltier,
Renaud, Blais and Fiset. This is an open-access article distributed under the terms
of the Creative Commons Attribution License (CC BY). The use, distribution or
reproduction i n other forums is permitted, prov id ed the original author(s) and the
copyright owner(s) are credited and that the original publication in this journal
is cited, in accordance with accepted academic practice. No use, distribution or
reproduction is permitted which does not comply with these terms.
Frontiers in Psychiatry | www.frontiersin.org 11 November 2021 | Volume 12 | Article 739742
