Detecting and Differentiating the Direction of Change and Intervention Effects in Randomized Trials

Published In

Journal of Applied Psychology

Document Type

Citation

Publication Date

2018

Abstract

Calls continue for randomized interventions in organizational settings. In many cases, however, practical constraints require researchers to use 2-wave randomized pretest–posttest control group designs. We discuss the importance of randomized trials for theory development with a focus on analytic options for 2-wave designs. Our discussion has implications for both designing studies and interpreting results. We review 23 published work and organizational health psychology intervention studies and find that a majority of studies featured a statistical model known to have low statistical power relative to other options. Furthermore, a majority of studies invoked terminology implying the direction of change without providing explicit statistical tests. To improve research practice, we detail statistical power differences in 3 commonly used statistical models and emphasize the distinction between (a) intervention effects and (b) the size and direction of change over time. We encourage researchers to provide inferential evidence for both types of information and show that only 1 of the 3 reviewed models provides information on the direction of change over time, but at a potential expense for statistical power to detect intervention effects. A reanalysis of data from a published work–family workplace intervention illustrates these nuances and supports recommendations for research practice. We conclude by providing recommendations.

DOI

10.1037/apl0000251

Persistent Identifier

http://archives.pdx.edu/ds/psu/24727

Share

COinS