A Measurement System for Science and Engineering Research Center Performance Evaluation
Abstract
This research provides performance metrics for cooperative research centers that enhance translational research formed by the partnership of government, industry and academia. Centers are part of complex ecosystems that vary greatly in the type of science conducted, organizational structures and expected outcomes. The ability to realize their objectives depends on transparent measurement systems to assist in decision making in research translation.
A generalizable, hierarchical decision model that uses both quantitative and qualitative metrics is developed based upon program goals. Mission-oriented metrics are used to compare the effectiveness of the cooperative research centers through case studies.
The US National Science Foundation (NSF) industry university cooperative research center (IUCRC) program is the domain of organizational effectiveness because of its longevity, clear organizational structure, repeated use and availability of data. Not unlike a franchise business model, the program has been replicated numerous times gaining recognition as one of the most successful federally funded collaborative research center (CRC) programs. Understanding IUCRCs is important because they are a key US policy lever for enhancing translational research. While the program model is somewhat unique, the research project begins to close the gap for comparing CRCs by introducing a generalizable model and method into the literature stream.
Through a literature review, program objectives, goals, and outputs are linked together to construct a four-level hierarchical decision model (HDM). At level 1, the purpose of the HDM is to determine the degree to which a center meets the mission of the NSF IUCRC program by evaluating a holistic set of metrics. Level 2 specifies three program objectives of industry-relevant research, the promotion of students and knowledge and technology transfer. Six goals populate level 3 and seventeen measurable outputs, characterized by desirability functions, fill level 4. A structured model development process shows how experts validate the content and construct of the model using these linked concepts.
A subjective data collection approach is discussed showing how collection, analysis and quantification of expert pair-wise-comparison data is used to establish weights for each of the decision criteria. Several methods are discussed showing how inconsistency and disagreement are measured and analyzed until acceptable levels are reached.
Through six developed case studies, actual center data are used to illustrate how the model calculates a score and how criterion-related validity is conducted with experts. First, the Wood-Based Composites (WBC) IUCRC uses the validated model construct to illustrate how a performance measurement score is calculated. Results are discussed that show how the WBC could obtain a significant performance increase by re-configuring project teams to include multi-disciplinary researchers and encouraging students to select center research projects towards completion of dissertations or theses.
Populating metrics with actual data from five (5) more IUCRCs establishes baseline performance scores for a total of six case examples. These case studies are used to compare results, evaluate the impact of expert disagreement and conduct criterion-related validity. Comparative analysis demonstrates the ability of the model to efficiently ascertain criteria that are relatively more important towards each centers’ performance score. Applying this information, specific performance improvement recommendations for each center are presented.
Upon review, experts generally agreed with the results. Criterion-related validity discusses how the performance measurement scoring system can be used for comparative analysis among science and engineering focused research centers. Dendrograms highlight where experts disagree and provide a method for further disagreement analysis. Judgment quantification values for different expert clusters are substituted into the model one-at-a-time (OAT) providing a method to analyze how changes in decisions based on these disagreements impact the results of the model’s output.
This research project contributes to the field by introducing a generalizable model and measurement system that compares performance of NSF supported science and engineering focused research centers. Funding these centers is expensive. Understanding where to shift resources can be a powerful decision-support tool for center directors. Transparency among experts regarding disagreement within the ecosystem about the decision criteria can help policy makers understand how to clarify objectives and analyze the impact of policy changes.