Survey Comments: Just How Much Do They Matter?

One of the most common pain points we hear from prospective clients centers around comment data. Employees take time to provide honest feedback yet practitioners are often overwhelmed by the volume of data and burden in finding the most useful suggestions, and this potentially valuable feedback often is underutilized.

To combat this problem, we embarked on a multi-year research study that covered all aspects of open-ended comments, which included characterizing what makes for a useful comment, identifying who writes them, understanding the role of nudge interventions, and comparing usefulness across all items.

These studies led directly to the creation of our machine learning tool, the OV VOICE. The OV VOICE – which stands for value-optimized, intelligent comment extractor – applies our research findings at scale, assigning a 0-100 Usefulness Quotient (UQ) score to each comment, identifying which comments are most useful. This allows users to identify the individual comments most likely to drive change across large, multi-language datasets. In other words, organizations can use the OV VOICE to instantly sort and filter through a limitless number of open-ended comments to easily identify the most useful ones across a range of topics, business units, or other metrics.

Recently, we turned our research focus on the validation of the UQ. While we understand usefulness well, we wanted to learn if – and how – providing useful comments to managers leads to better action planning and improvement.

To determine this, we looked at how team scores change from year-to-year based upon the quality of feedback provided.

Our research clearly shows a relationship between useful feedback within a team and the belief that survey results will lead to meaningful change in the second year. This suggests that teams who provided useful feedback are accustomed to seeing their feedback lead to change.

Aside from the relationships between UQ and confidence in survey-based change, we also looked at how groups with higher UQ change differently. Here, across both open-ended questions, groups with higher UQ (75+) improved on confidence that the survey will lead to change more than their counterparts with less useful feedback.

This begins to suggest that it isn’t simply a relationship between optimism in change and high UQ, but providing high UQ comments leads to more noticeable improvements.

Next, we investigated whether it is providing any comments, or just specifically useful comments, that helps managers create greater levels of change post-survey.

This first graph (below) shows a comparison of how groups that received comments as well as those who did not receive any comments change from year one and year two on the survey. While there are some small differences, they aren’t significant, suggesting that simply providing comments does not make a major difference in the information managers have to create and sustain action plans.

By comparison, this next graph (below) shows the change in survey scores based upon the usefulness of comments received. The groups that received highly useful comments (75+) improved at dramatically different rates compared to all other groups.

Altogether, this validation study illustrates the importance of soliciting and finding useful comments. Clearly, groups that receive useful feedback have additional context and suggestions to jump start action planning, regardless of topic. The UQ can’t determine which comments have the ‘right’ suggestions for improvement, but it can highlight well-thought-out conversation starters. By using the OV VOICE, whether in reports or dynamic reporting portals, usefulness can be identified and prioritized for greater change impact – the ultimate goal of any survey.

Want to learn how the OV VOICE can help your organization solicit, find, and act on open-ended comments? Contact Victoria Hendrickson directly.

OrgVitality specializes in organizational development initiatives including strategic employee surveys, engagement surveys, lifecycle and pulse surveys, organizational network analysis, action planning, 360 assessments, linkage analysis, and so much more. Discover why leading global corporations seek out our expert advice, rely on our powerful technology solutions, and depend on our project teams. Contact us today for a free consultation and demo.

‌ ‌