Skip to main content

Understanding Student Ratings

This introduction to the student ratings tool is to help faculty and administrators capitalize on its features to promote student learning and evaluate faculty progress in achieving this aim.

Watch this video for detailed instructions on how to understand student ratings. Additionally, there are links to certain video segments with their associated topics provided below the video.

The context for the video is found at the bottom of this page.

Student Ratings Presentation

An index to the contents of the video recording of the presentation and discussion

0:00 Brent Webb provides background on the work of the task force (opens in a new tab)

1:35 Issues around previous instrument that gave rise to new survey (opens in a new tab)

3:11 Survey overview (Burlinggame) (opens in a new tab)

4:42 7-year timeline overview (opens in a new tab)

6:04 Task force charge (opens in a new tab)

7:51 How the old system guided formation of the new (opens in a new tab)

9:43 Issues addressed by the task force I (opens in a new tab)

12:33 Issues addressed by the task force II (opens in a new tab)

13:02 Sample” instructor” item from the New Instrument (opens in a new tab)

14:17 Sample “AIMs” item from the New Instrument (opens in a new tab)

14:51 Improvements in Reporting (opens in a new tab)

15:18 Sample semester report for a class (opens in a new tab)

18:15 “AIMs items” discussed (opens in a new tab)

19:55 Questions (opens in a new tab)

19:58 What does the composite score “cover”? (opens in a new tab)

20:47 Why does the form on my device look different from the one on today’s screen? (opens in a new tab)

21:58 What does the composite score for course “cover”? (opens in a new tab)

23:28 What does the composite score represent? (opens in a new tab)

24:44 How do we deal with the reliability band; is there any guidance on that? (opens in a new tab)

26:45 How will the data (and uncertainty band considerations) be used in the CFS process? (opens in a new tab)

28:35 How do we “frame” and “sell” this to our students correctly? (opens in a new tab)

29:25 Why not make survey participation required of students? (opens in a new tab)

31:01 You have bands on the “section and department”, but not on “course, college, and university” and I wonder why. (opens in a new tab)

32:54 Comment: “I’m still worried about the response rate, because… 90% participation is needed for the data to be really reliable.” (opens in a new tab)

34:36 Two slides to provide insight into how “trends” might appear (opens in a new tab)

32:54 (this is partly in response to the comment and expression of concern). (opens in a new tab)

34:54 Multi-year report: Positive Change (opens in a new tab)

35:14 Multi-year report: Negative Change (opens in a new tab)

35:40 Questions addressed by W-2015 pilot (Reese) (opens in a new tab)

  • Can there be a single reliable composite score?
  • How does the new composite score relate to the old global instructor item?

40:53 Summary of what the pilot looked like (participation and response rate) (opens in a new tab)

41:50 Can 5-items be summed into a composite? (slide) (opens in a new tab)

42:02 What is the estimated reliability of the composite? (slide) (opens in a new tab)

43:07 Relationship of Items/Composite to old global (opens in a new tab)

43:18 Comment from Brent Webb about course size (opens in a new tab)

44:14 Comment about reliability improving with aggregation over time (opens in a new tab)

45:04 Response pattern affects shape of uncertainty band (opens in a new tab)

46:28 Summary recommendations (future) (opens in a new tab)

47:20 Questions (opens in a new tab)

47:26 How do you handle a class that has a large enrollment, but perhaps 10 separate lab sections? How do we aggregate the sections and improve certainty? (opens in a new tab)

49:00 How do we deal with the bias resulting from any one student commenting over and over? (opens in a new tab)

51:42 When will faculty receive training on this? (opens in a new tab)

53:19 How do we deal with pre-CFS faculty who have spent 4 or 5 years on the old system, and now we’re switching to the new one? (opens in a new tab)

55:00 What to do in a small department when there’s only one person to use as a comparison? (opens in a new tab)

56:54 How can we compare the old “single point” rating to the new “range of values”? (opens in a new tab)

59:33 How are we to view the uncertainty band; what are it’s components? (opens in a new tab)

1:00:43 What do you do about uncertainty bands in a department where there’s a lot of heterogeneity (odd duck courses)? (opens in a new tab)

1:03:47 Will there eventually be a “trend” component built into the instrument? (opens in a new tab)

1:05:09 Can you assure us that students comments in the new instrument will be better than those we received in the past? (opens in a new tab)

1:08:10 What do you do about an uncertainty band for an instructor when only one person teaches the course? (opens in a new tab)

1:08:56 Comment/caution about how the new instrument is presented (opens in a new tab)

1:10:44 Question about how the grades component of the new survey is derived (opens in a new tab)

1:12:23 Have you done any analysis on the individuals who are more likely to make comments? (opens in a new tab)

1:12:56 What are the sources of variability in the uncertainty band? (opens in a new tab)

1:13:39 Is there going to be a summary sheet for all the courses in our department? (opens in a new tab)

1:14:27 Comment: it would be helpful to be able to summarize by core courses versus service courses (opens in a new tab)

1:15:20 Have you checked on the relationship between GPA (or ACT score) and student ratings? (opens in a new tab)

1:16:14 Do you have any sense for what kind of bias occurs with a low response rate? (opens in a new tab)

1:18:16 What do we do about inappropriate or even profane student comments? (opens in a new tab)

1:20:47 Most of our classes are small – what is that magic number where reliability becomes no longer a problem? (opens in a new tab)

1:23:04 What are the quality concerns about the other approaches to teacher evaluations, such as peer review? (opens in a new tab)

1:25:21 Brent Webb provides concluding comments (opens in a new tab)

1:31:52 End (opens in a new tab)

On September 18, 2015, department chairs were given a preview of the new student ratings tool. Two members of the Student Ratings Task Force, Professors Gary Burlingame (Psychology) and Shane Reese (Statistics), as well as members of the Academic Vice President’s Council, presented an overview of the development process, discussed the new instrument, and answered questions. The recording has been edited to reduce the nearly two-hour meeting to a reasonable length for viewing.