Small changes to how questionnaires are designed can significantly improve respondents’ attention, recall and consistency, new survey experiments have shown, highlighting the limits of relying on artificial intelligence to generate research questions without accounting for human behaviour.
The experiments, conducted by consumer insights specialist Erwin Chang, found substantial differences in results when respondents were given time to process information before answering questions, when important instructions were visually emphasised and when lengthy survey tasks included brief breaks.
One experiment involving a fictional ready-to-drink vanilla coffee product found that 89 per cent of respondents correctly identified the product category when they viewed its packaging before receiving evaluation questions. Accuracy dropped to 72 per cent when the same packaging and questions appeared together. Brand recall similarly fell from 92 per cent to 75 per cent.
Brandspur Brand News reports that the findings are particularly relevant as generative AI increasingly makes it possible for researchers and businesses to produce detailed questionnaires rapidly. While the technology can help construct logically organised surveys, the experiments suggest that questionnaire quality also depends heavily on understanding how people actually read, remember and make decisions.
Another test demonstrated the importance of making critical instructions immediately noticeable. Only 11 per cent of participants followed a longer instruction asking them to select a particular response, compared with 42 per cent when the key instruction was shortened and visually emphasised.
Familiarity with response formats also influenced preferences. Among three approaches tested, 52.4 per cent of respondents preferred star ratings, compared with 29 per cent for conventional checkbox ratings and 18.6 per cent for semantic differential scales.
Survey fatigue produced another measurable effect. Participants completing a lengthy evaluation exercise were tested using questions repeated later in the questionnaire. When respondents received a brief five-second break during the exercise, consistency on one repeated question increased from 78 per cent to 86 per cent. On another, it rose from 88 per cent to 94 per cent.
The research also raised questions about traditional methods for detecting questionable survey responses. In a behavioural-validation exercise, 1,479 people were screened and 679 failed to meet the study’s validation criteria, leaving 800 qualified participants.
Traditional open-ended quality checks flagged approximately one per cent of the screened sample, while conventional control questions identified about eight per cent. A broader behavioural approach, which considered fictitious product selections and whether overall responses were realistically plausible, flagged approximately 46 per cent.
The figure does not mean those respondents were necessarily dishonest. Instead, their overall answers failed to satisfy the behavioural criteria established for the study, reinforcing the argument that researchers should examine response patterns rather than relying on isolated mistakes.
Artificial intelligence could also change how researchers use open-ended questions. Large volumes of written responses that once required extensive manual coding can increasingly be organised and analysed with AI, potentially allowing surveys to explore consumers’ experiences in greater depth.
Rather than asking respondents to summarise broad experiences from memory, the research suggests that questionnaires can guide people through specific moments by asking about details such as where an event occurred, who was present, what happened and how they felt.
The findings point to a wider shift in survey research: generating questions may be becoming easier with AI, but obtaining reliable answers still depends on designing those questions around the way humans pay attention, become tired, recall experiences and make decisions.
