1.2 - Designing self-reports
Learn to choose and write open questions, closed questions, Likert items and semantic differential scales, and evaluate whether a self-report records the intended experience fairly.
Open and closed questions
Self-reports ask people to describe their own experiences, thoughts or behaviour. They can reveal experiences an observer cannot directly see, but an answer remains a report rather than independent proof of what happened.
An open question allows the respondent to answer in their own words. For example, “What made your first week at a new club enjoyable or difficult?” can reveal unexpected experiences and reasons. Such qualitative detail helps the researcher understand what enjoyment means to that respondent. Responses take longer to give and analyse; if they are later coded, clear coding rules are needed. A long written response may disadvantage people with limited literacy or confidence in the language.
A closed question restricts the answer to specified categories, values or options. “How many club sessions did you attend in the last seven days: 0, 1, 2, 3, or 4 or more?” provides easily compared frequency categories. “Did you attend a session yesterday: yes/no?” is also closed. Neither is automatically a rating scale. Closed questions are quicker to score, but may force an answer that does not capture the person's experience. Include all relevant options, avoid overlapping categories, and use other, not applicable or prefer not to answer where these are needed. Do not treat no response as no attendance.
An open response is not inherently accurate, and a numerical answer is not inherently objective. People can forget, misunderstand or give a socially desirable answer in either format. The choice depends on whether the aim needs detailed reasons, comparable counts, or both.
Likert agreement ratings
A Likert item presents a statement and asks how strongly the respondent agrees or disagrees. A set of such items can form a Likert scale, while one statement is often called a Likert rating item. It is not simply any question with numbers from 1 to 10.
For example, the statement “I feel welcome at this club” can use the ordered responses below. Ask the respondent to select one response, and state the reference period, such as the last four weeks, if it matters to the aim.
| Response | Recorded code |
|---|---|
| Strongly disagree | 1 |
| Disagree | 2 |
| Neither agree nor disagree | 3 |
| Agree | 4 |
| Strongly agree | 5 |
These codes preserve the order of agreement. They do not establish that the psychological distance from 1 to 2 equals the distance from 4 to 5. A higher code indicates greater agreement with this particular statement, not automatically better mental health or more social skill. A neutral midpoint allows neutrality, but it is not the same as not applicable or do not understand; provide those separately if relevant.
An agreement format is compact and consistent across respondents, but people may agree regardless of content (acquiescence) or choose a favourable answer (social desirability). An item such as “I enjoy the friendly and well-run club” is double-barrelled: someone may enjoy it while thinking it is poorly run. Separate the ideas. Reverse-worded items can sometimes check habitual agreement, but confusing negatives can create measurement error and any scoring reversal must be defined.
Semantic differential ratings
A semantic differential scale asks the respondent to rate something between two opposite descriptive terms. It measures the connotative meaning or evaluation of the specified object, rather than agreement with a statement.
For example, “Rate the atmosphere at this club during the last four weeks” might use a seven-position scale with unpleasant at one end and pleasant at the other:
| Unpleasant | Midpoint | Pleasant | ||||
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
The respondent marks one position. A rating of 6 is nearer pleasant than unpleasant. The endpoints must be genuine opposites on the intended dimension, such as welcoming–unwelcoming. Pleasant–busy is unsuitable as one bipolar dimension because a place can be both pleasant and busy. The object and period being rated should remain consistent across respondents.
Both semantic differential and Likert formats are closed ratings with ordered responses, allowing compact comparison. Their prompts differ: a semantic differential offers opposite descriptive anchors; a Likert item asks for agreement with a statement. A bare instruction “rate your mood from 1 to 10” without these anchors is not enough to identify either correctly. Semantic differential responses may also be affected by interpretation of adjectives, cultural or language differences, and a tendency to avoid the endpoints. Ordered codes do not automatically guarantee equal psychological intervals.
Improving and evaluating self-report questions
A clear question specifies one idea, uses neutral language, and gives a reference period the respondent can reasonably remember. “How often do you waste time online?” implies disapproval and leaves the time period vague. “On how many of the last seven days did you use social media after 10 pm?” is more neutral and defines the period and behaviour. It measures reported late use, which is not the same construct as wasting time.
Avoid technical vocabulary, assumptions and leading questions such as “Don't you agree that the club has improved your confidence?” A respondent may not think it has improved at all. Use balanced response options and a route for people to whom the item does not apply. Keep administration consistent: a reassuring neutral introduction and the same instructions reduce differences in how the question is presented. Pilot with people similar to the intended respondents to discover confusing words or missing options before the main study.
Private or anonymous completion can reduce pressure to give socially acceptable answers, especially when a teacher or club leader would otherwise see named responses. It does not guarantee honesty or accurate memory. Explain how data will be used, let people decline questions where appropriate, and protect confidentiality. Questions about sensitive experiences need particular care; an online form is not automatically anonymous if it stores names or account identifiers.
Open questions can deepen validity by revealing how respondents understood an experience, but coding them consistently may be difficult. Closed questions can improve scoring consistency and efficiency, but overly narrow options can undermine validity. Repeating a clear question consistently supports reliability; if the person's experience genuinely changes, a changed answer is not automatically an unreliable measure. Select and evaluate the question format in relation to the aim, rather than claim that one format is always best.