Decision Making

High quality evaluation forms are the corner stone of effective university or college course evaluation, but additional system features turn data into information. IASystem™ supports both pedagogical and programmatic decision making by suggesting “rules of thumb” based on institution-specific reliability estimates.

“Rules of Thumb”

Item reliability plays a key role in decision making “rules of thumb” for both pedagogical and programmatic decision making. The higher the reliability, the more confident we can be that average ratings reflect student opinions about a class rather than random error, and the smaller the difference between ratings required for statistical significance. IASystem™ uses Spearman-Brown corrections to develop decision making “rules of thumb” for each institution based on two years of course evaluation results.

Pedagogical Decision Making

IASystem™ instructional improvement items are designed to support pedagogical changes to individual courses. For these items, IASystem™ computes inter-rater correlation coefficients and determines the minimum class size required to achieve adequate (r ≥ .7) reliability for decision making. The following sample graph is based on analysis of two-years of course evaluations at the University of Washington and shows that a minimum of seven to ten students per class is required for single-class decision making.

Inter-Rater Reliability by Class Size at the University of Washington

Programmatic Decision Making

IASystem™ “global” items are designed to inform programmatic decision making relating to academic program review and accreditation, and to inform decisions around faculty promotion and tenure. Because of the importance of these decisions, IASystem™ combines the four global items into a single index for added measurement stability, and recommends basing decisions on data from combined classes rather than only a single class. For these analyses, IASystem™ computes inter-class correlation coefficients and determines the minimum number of classes required to achieve adequate (r ≥ .7) reliability. The following sample graph is based on analysis of two-years of course evaluations at the University of Washington and shows that a minimum of five classes is required for programmatic decision making.

Inter-Class Reliability by Number of Classes at the University of Washington