Feedback Design

Most experts who had experienced feedback could extract important information from graphical displays. The study could not confirm why many experts did not actively incorporate feedback, but it treats feedback design as needing further empirical investigation. From the second round onward, feedback included each participant’s prior response, aggregated group judgments, and summarized open-text arguments. Participants were informed when judgments differed significantly by disciplinary background. Only about half of participants considered statistical results and summarized open-text arguments when responding in later rounds.