Applied Measurement for Evaluation

Description: Successful evaluation depends on our ability to generate evidence attesting to the feasibility, relevance and/or effectiveness of the interventions, services, or products we study. While theory guides our designs and how we organize our work, it is measurement that provides the evidence we use in making judgments about the quality of what we evaluate. Measurement, whether it results from self-report survey, interview/focus groups, observation, document review, or administrative data must be systematic, replicable, interpretable, reliable, and valid. While hard sciences such as physics and engineering have advanced precise and accurate measurement (i.e., weigh, length, mass, volume), the measurement used in evaluation studies is often imprecise and characterized by considerable error. The quality of the inferences made in evaluation studies is directly related to the quality of the measurement on which we base our judgments. Judgments attesting to the ineffective interventions may be flawed - the reflection of measures that are imprecise and not sensitive to the characteristics we chose to evaluate. Evaluation attempts to compensate for imprecise measurement with increasingly sophisticated statistical procedures to manipulate data. The emphasis on statistical analysis all too often obscures the important characteristics of the measures we choose. This class content will cover these topics:  

• Assessing measurement precision: examining the precision of measures in relationship to the degree of accuracy that is needed for what is being evaluated. Issues to be addressed include: measurement/item bias, the sensitivity of measures in terms of developmental and cultural issues, scientific soundness (reliability, validity, error, etc.), and the ability of the measure to detect change over time.

•  Quantification: Measurement is essentially assigning numbers to what is observed (direct and inferential). Decisions about how we quantify observations and the implications these decisions have for using the data resulting from the measures, as well as for the objectivity and certainty we bring to the judgment made in our evaluations will be examined. This section of the course will focus on the quality of response options, coding categories - Do response options/coding categories segment the respondent sample in meaningful and useful ways?

• Issues and Considerations - using existing measures versus developing your own measures: What to look for and how to assess whether existing measures are suitable for your evaluation project will be examined. Issues associated with the development and use of new measures will be addressed in terms of how to establish sound psychometric properties, and what cautionary statements should accompanying interpretation and evaluation findings using these new measures.

• Criteria for choosing measures: assessing the adequacy of measures in terms of the characteristics of measurement - choosing measures that fit your evaluation theory and evaluation focus (exploration, needs assessment, level of implementation, process, impact and outcome). Measurement feasibility, practicability and relevance will be examined. Various measurement techniques will be examined in terms of precision and adequacy, as well as the implications of using screening, broad-range, and peaked tests.

• Error - influences on measurement precision: The characteristics of various measurement techniques, assessment conditions (setting, respondent interest, etc.), and evaluator characteristics will be addressed.

Participants will be provided with a copy of the text: Measurement Theory in Action (Case Studies and Exercises) by Shulz, K.S. and D.J. Whitney (Sage, 2004).

Instructor: Dr. Ann Doucette is Senior Research Scientist, Center for Health Services Research and Policy, The George Washington University Medical Center,Washington, DC.  She has broad experience in the management, analysis, and evaluation of intervention programs, including the development of accountability and outcomes monitoring systems for programs that cut across individual and system levels; and in research methodology, data collection, psychometric and measurement techniques, evaluation research, and applied statistical analysis, including both quantitative and qualitative approaches.  

She has worked with foundations, public schools, mental health services, universities, and community groups; with young children and families; in social policy, juvenile justice, urban and minority education, morality and ethics, vocational education, conflict resolution, and more. She developed several assessment measures using Item Response Theory to generate more precision with briefer, less burdensome assessment instrumentation that lends itself to computer-adaptive applications and real-time data usage. Currently, she is collaborating on the development of a comprehensive, integrated measurement system that assesses both treatment process indicators as well as service intervention outcomes for children and adolescents. Among her other responsibilities, she is co-chair of the Outcomes Roundtable for Children (supported by the Substance Abuse and Mental Health Services Administration) and serves on the Executive Committee and Methods Workgroup of the Forum on Performance Measures for Behavioral Healthcare and Related Service Systems. She received her doctoral training at Columbia University.

Certificates: CEP IB.b or CAEP IIB.b; and CQEM III.d

Fee: $795