Linking Evaluation Questions |
Description: Statistics are a mainstay in the toolkit of program and policy evaluators. Human memory being what it is, however, even evaluators with reasonable statistical training, over the years, often forget the basics. And the basics aren't always enough. If evaluators are going to make sensible use of consultants, communicate effectively with funders, and understand others' evaluation reports, then they often need at least a conceptual understanding of relatively complex, recently developed statistical techniques. The purposes of this course are: to link common evaluation questions with appropriate statistical procedures; to offer a strong conceptual grounding in several important statistical procedures; and to describe how to interpret the results from the statistics in ways that are principled and will be persuasive to intended audiences. The general format for the class will be to start with the evaluation question(s) and then discuss the choice and interpretation of the most-suited statistical test(s). Emphasis will be on creating a basic understanding of what statistical procedures do, of when to use them, and why, and then on how to learn more from the data. Little attention is given to equations or computer programs, with the emphasis instead being on conceptual understanding and practical choices.* Within a framework of common evaluation questions, statistical procedures and principled data inquiry will be explored. (A) More fundamental topics to be covered include (1) basic data quality checks and basic exploratory data analysis procedures, (2) basic descriptive statistics, (3) the core functions of inferential statistics (why we use them), (4) common inferential statistics, including t-tests, the correlation coefficient, and chi square, and (5) the fundamentals of regression analysis. (B) For certain types of evaluation questions, more complex statistical techniques need to be considered. More complex techniques to be discussed (again, at a conceptual level) include (1) structural equation modeling, (2) multi-level modeling, and (3) cluster analysis and other classification techniques. (C) Examples of methods for learning from data, i.e., for "snooping" with validity, for making new discoveries principled, and for more persuasive reporting of findings will include (1) planned and unplanned tests of moderation, (2) graphical methods for unequal treatment effects, (3) use of previously-discussed techniques such as clustering, (4) identifying and describing converging patterns of evidence, and (5) iterating between findings and explanations. Each participant will receive a set of readings and current support materials. Prerequisites: Familiarity with basic statistics. *This course is not the equivalent of nor a substitute for Dr. Poister's Applied Statistics for Evaluators course (offered at previous Institute programs). Similarly, the course is not the equivalent of nor a substitute for Dr. Mark's previously taught course on Understanding Data that was offered at several previous Institute programs. This course looks at a longer chain (from evaluation question to choice of statistical procedure to interpretation), and will spend less time on the interpretation side than was possible when an entire two days of time was devoted to the topic of interpretation. Instructor: Dr. Melvin M. Mark is Professor of Psychology and Senior Scientist at the Institute for Policy Research and Evaluation at the Pennsylvania State University and is President-Elect of AEA and Past Editor of the American Journal of Evaluation. He has conducted federally funded evaluations in the areas of prevention programs for at-risk youth, federal personnel policies, and industrial modernization, and has been involved in evaluations of state and local programs. An award-winning teacher, he has published numerous papers and chapters on the theory and design of evaluations. He is co-editor of Social Science and Social Policy; > Multiple Methods in Program Evaluation; Realist Evaluation: An Emerging Theory in Support of Practice and a forthcoming Handbook of Evaluation (with Ian Shaw and Jennifer Greene). He is co-author (with Gary Henry and George Julnes) of Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass 2000).
Certificates: CEP, IB.e; or CAEP, IIB.e; and CQEM, III.g. Fee: $1150 |
