Psychometric properties of the brief version of the questionnaire of olfactory disorders in patients with chronic rhinosinusitis

The Questionnaire of Olfactory Disorders‐Negative Statements (QOD‐NS) is a 17‐item instrument measuring olfactory‐specific quality of life (QOL). However, in clinical research patients can be overwhelmed with multiple questionnaires. We recently developed the 7‐item brief QOD‐NS (B‐QOD). Our objective was to evaluate the psychometric properties of the B‐QOD in both the development (D) sample, and in a separate replication (R) sample.

respectively). CFA indicates that the B-QOD single-factor model has good overall fit as measured by the Comparative Fit Index (CFI) and the Standardized Root Mean Squared Residuals (SRMSR) in the D and R samples (CFI = 0.99 and 0.97; SRMSR = 0.035 and 0.053). IPUA shows that the QOD-NS offers no additional predictive benefit of SNOT-22 scores when compared with B-QOD.

INTRODUCTION
Chronic rhinosinusitis (CRS) is a common inflammatory disorder affecting ∼5% to 16% of North American populations. 1 Symptoms of CRS are nasal congestion, discolored nasal drainage, facial pain/pressure, and olfactory dysfunction (OD), with OD estimated to affect between 40% and 80% of CRS patients in certain study populations. 2 OD ranks as the second most important symptom for CRS patients after nasal congestion, 3 and consequently has garnered significant attention in quality of life (QOL) and outcomes research. This has led to the development of instruments that specifically focus on olfactory-specific QOL. [4][5][6] In particular, the questionnaire of olfactory disorders negative statements (QOD-NS) is a 17-item validated instrument in the measurement of olfactory-specific QOL, and also demonstrates robust psychometric properties. 7,8 However, in the clinical and outcomes research setting, patients are frequently inundated with a variety of questionnaires and instruments to complete, and the time required to complete these instruments can be significant. Furthermore, shorter patient-reported outcome metrics (PROMs) have been shown to lead to increased efficiency, reduced patient burden, and greater data quality. 9 For these reasons, we have previously developed a brief version of the QOD-NS (B-QOD). 10 The B-QOD was created to intentionally represent all of the subdomains of the QOD-NS, and the B-QOD demonstrates excellent correlation with the total and subdomain scores of the QOD-NS and consists of only seven items which are a subset of the QOD-NS. 10 Our present objective was to evaluate the psychometric properties of the B-QOD in the CRS population using the initial development sample and also in a separate replication sample. In the remainder, we refer to these samples as the development sample (D) and the replication sample (R), respectively.

Sample populations
The purpose of these analyses was to evaluate the psychometric properties of the B-QOD using the initial D sample (enrollment years: 2011-2016; n = 203) and in the R sample (enrollment years: 2016-2019; n = 281). Study enrollment originated from two observational, prospective research investigations of human subjects funded by the National Institute on Deafness and Other Communication Disorders (NIDCD, Bethesda, MD, USA). Study participants were recruited at varying times from heterogeneous patient populations presenting to academic, rhinology centers located in North America: Oregon Health & Science University (Portland, OR, USA), the Medical University of South Carolina (MUSC, Charleston, SC, USA), Stanford University (Palo Alto, CA, USA), the University of Utah (Salt Lake City, UT, USA), the University of Colorado (Aurora, CO, USA), the University of Calgary (Calgary, Alberta, Canada), and the University of Virginia (Charlottesville, VA, USA). The Institutional Review Board at each performance site provide ethical review and oversight of this minimal risk study. Table 1 provides the demographic and disease-specific characteristics of these cohorts.
Symptomatic, adult study participants received a confirmed diagnosis of symptomatic and medically recalcitrant CRS, with nasal polyposis (CRSwNP) or without nasal polyposis (CRSsNP), from a fellowship-trained rhinologist following criteria established by current practice guidelines. 11 Participants presented with symptoms including, but not limited to, nasal congestion, mucopurulent drainage, facial pain/pressure, and olfactory impairment. All patients provided written, informed consent to ensure voluntary participation for this minimal risk study.

Exclusion criteria
Study participants were excluded from final cohort selection if they failed to complete QOD-NS sample survey evaluations at the time of study enrollment. Additionally, any study participants with comorbid primary ciliary dyskinesia or cystic fibrosis (CF) were excluded due to possible disparity in disease presentation and/or treatment approach.

Analytic strategy and biostatistics
First, we performed exploratory factor analyses using the B-QOD data from the two samples in order to examine the number of underlying constructs that the B-QOD might be representing, with the hypothesis that the B-QOD represents a single factor. We present standardized factor loadings, which estimate the correlation between the underlying factor and the items; residual variances estimate the variance in item scores not attributable to the underlying factor. An item is deemed a "salient" indicator if the standardized factor loading for that item exceeds |0.32| because the factor "explains" >10% of the variance in the item; items with loadings greater than 0.71, 0.63, and 0.55 are considered "excellent," "very good," and "good," respectively. 12 We then checked the internal consistency and reliability of the estimates achieved by the B-QOD.
To do this, we used Cronbach's alpha. Nunnally and Bernstein 13 provide guidelines for judging the adequacy of reliability estimates stating that estimates exceeding 0.70 are considered adequate for use in research and estimates exceeding 0.80 are considered good. Interested readers desiring to learn more about psychometric analysis and the statistics involved could consult Nunnally and Bernstein's 13 and McDonald's 14 books on the topic. Next we explored the degree of possible information loss between the B-QOD and the QOD-NS. Because the B-QOD is based on seven of the 17 items in the QOD-NS it is possible that in using fewer than half of the original items, the B-QOD may omit important information from overall scale scores using all 17 items. To evaluate the degree of information loss, we correlated scores between the full 17-item measure with the 7-item B-QOD.
After these steps, a confirmatory factor analysis (CFA) model is needed to evaluate the single-factor hypothesis directly. For this end, we used the CFA model for ordinal indicators in Mplus software. 15 We then utilized the comparative fit index (CFI), which indicates good fit when values exceed 0.95; 16  values are <0.05. We also calculated standardized factor loadings from the confirmatory analysis on both samples. Finally, we performed an analysis of incremental predictive utility of the QOD-NS compared to the B-QOD scores for a common QOL measure in the CRS literature, the 22-item Sino-Nasal Outcome Test (SNOT-22). 17 First, we descriptively computed and compared the correlations between QOD-NS and B-QOD scores with SNOT-22 total scale scores in the D and R to test whether QOD-NS scores have additional incremental predictive utility in the prediction of SNOT-22 total scores, a hierarchical linear regression approach was used. In the first step, SNOT-22 total scores were regressed on B-QOD scores; in the second step, QOD-NS scores were added as a predictor to the model. The statistic of interest is the change in R 2 from step 1 to step 2.

Exploratory factor analyses
The two exploratory factor analyses (EFAs) yielded similar results about the underlying structure of the B-QOD. Globally, both EFAs exhibited a single, dominant dimension of variability for the underlying factor structure with only a single eigenvalue greater than one (sample D 1st eigenvalue = 4.17; sample R 1st eigenvalue = 3.57). Table 2 provides the standardized factor loadings from the EFAs for the two samples. The standardized factor loadings are substantively similar in size across the two samples. As would be expected, the standardized factor loadings tend to be a larger in D, which illustrates the importance of evaluating psychometric properties on an independent sample over and above the development sample. Despite this slight attenuation, the standardized factor loadings in sample R are all sizable and far exceed the threshold for a salient loading (i.e., a loading of 0.30 or greater).

Internal consistency reliability estimates
Given that the EFAs strongly support the single-factor structure of the B-QOD, we next turn to how reliably these items measure that single latent dimension. Table 3 provides the Cronbach's alpha estimates for the two samples. The reliability estimates for the B-QOD far exceed the criterion value considered good in both samples. Table 4 provides these correlations for the two samples between QOD-NS and B-QOD scores. These correlations should be considered very large and suggest that little information is lost when using the B-QOD rather than all 17 of the QOD-NS items.

CFA of the B-QOD items
The fit statistics for the models, along with the CFI, SRMR, and RMSEA statistics in the two samples appear in Table 5. Overall, these results suggest that the single-factor hypothesis for B-QOD is a useful approximation to the data. Given the adequate level of fit of the model, we now evaluated the standardized factor loadings from the CFA models in both samples. These are presented in Table 6. These loadings are all large in size indicating that each item strongly relates to the underlying single factor.

DISCUSSION
We have previously reported on our development of the B-QOD, 10 which is a subset of seven items from the original QOD-NS. Our initial report confirmed that the B-QOD purposefully included all of the subdomains of the original QOD-NS, and B-QOD scores have good correlation with total QOD-NS scores as well as the subdomains scores that are captured by the QOD-NS. 7,10 Furthermore, the QOD-NS has been previously validated in CRS patients, and this validation included CRS patients along the entire spectrum of olfactory ability. 18 As such, we maintain a similar approach to the creation and validation of the B-QOD in CRS patients ranging from normosmics to anosmics. However, given the substantial reduction in the number of items from the QOD-NS to the B-QOD, it is possible that the shortened instrument may not have all of the desired psychometric properties. Psychometric properties refer to the validity and reliability of a measurement tool. 13 A valid tool is one that measures what is says it is going to measure, whereas a reliable tool consistently measures the same construct. Our analysis has shown that the B-QOD is both valid and reliable. The B-QOD has good internal validity, as shown by the exploratory and confirmatory factor analyses demonstrating a single unified construct (Tables 2, 5, and 6). Furthermore, our results show that there is no loss of information when comparing the B-QOD to the original QOD-NS (Table 4). The external validity of the B-QOD is confirmed by our analysis demonstrating good correlations with SNOT-22 scores, and the fact that the original QOD-NS does not add any predictive utility to SNOT-22 scores when compared to the B-QOD (Table 7). Furthermore, the Cronbach's alpha analysis confirms the internal consistency and reliability of the B-QOD (Table 3). It is important to note that the results hold true both in the original cohort used to develop the B-QOD, but also notably in an entirely separate replication cohort. The consistent findings between the performance of the B-QOD in two independent cohorts add further evidence that this instrument is valid and reliable and should be considered for use.
The B-QOD is likely most useful for situations where there is little time for patient to completely questionnaire (i.e., a busy clinical setting) or where patients may be overburdened by numerous surveys or questionnaires (i.e., prospective clinical research settings). In reality, the majority of patients or research subjects are likely encompassed by these two scenarios, and so we expected that the B-QOD could be useful for a wide variety of future projects or investigations. However, some investigators may desire the greater granularity that the original QOD-NS allows, particularly the ability to study the different subdomains or factors captured by the QOD-NS. 7 This study has several limitations. First, the participants in both D and R were all CRS patients, and it is possible that the instruments psychometric properties may be different in different populations. Also, the participants had high associated disease severity and were assessed through tertiary practices, and therefore these results may not be externally generalizable to all CRS patients. Further validation and study of our shortened version in other populations with different etiologies for olfactory dysfunction (i.e., again, postinfectious, posttraumatic, etc.) of the will be necessary to further confirm its clinical utility and validity.

CONCLUSION
The results of this psychometric investigation indicate that data from B-QOD capture a structurally coherent single latent dimension, that the B-QOD reliably measures this dimension, that little information is lost when using the seven items of the B-QOD relative to using all 17 items of the QOD-NS, and that the B-QOD has excellent external predictive utility when compared to the QOD-NS. In settings where olfactory-specific QOL is of interest but survey burden is a concern, or where rapid completion of PROMs is of interest, the B-QOD should be considered for use due to its few items and excellent psychometric properties.