How to design a questionnaire

Surveys are methods to collect information regarding knowledge, opinion, behaviour or facts from a population. They provide us with qualitative or semi-quantitative data, which in turn can be used for further research, administration or policy making. Surveys are economical to conduct. They are efficient, as a large population over a wide region can be covered in a short period of time. Also, they are non-interventional and thus have minimal ethical concerns. However, it must be remembered that surveys are often limited by low response rates and selection bias. Moreover, the quality of a survey is greatly dependent on the design of the questionnaire used. This editorial briefly outlines the process of development of a questionnaire in the context of the three survey-based studies published in this issue of the journal.[1,2,3]

A questionnaire appears to be just a simple list of questions to the naive. However, the language of the questions, the type of questions used, the order in which they are arranged and many other details, all impact the results of the survey. Therefore, it is essential that the questionnaire is designed meticulously and is validated before use. It should be ensured throughout the development process that the focus does not deviate from the research question. The design of the questionnaire should also be tailored to suit the medium of the survey (phone, email, web, postal, one-on-one interview, etc.).

A review of the relevant literature, followed by discussion with experts on the subject, is required before the questions or items are formulated. A brief introduction clearly stating the aims of the survey and providing information about the investigators makes a good beginning. The introduction should also appeal to and attract responders. Arrangement of the questions in a logical and structured sequence, with general questions preceding the specific, is recommended to get better responses. Controversial items may be presented towards the end of the questionnaire. Routine questions about the respondent such as age, gender, place of work and other demographics may also be placed here.

A few simple rules and common sense help in formulating good questions. The questions should be appropriate, so that the responses answer the research question. The questions should be worded carefully to be intelligible and have maximum clarity. They should be unambiguous so that the meaning conveyed to the respondent is the same as that intended by the investigator. They should be unbiased and not favour one response over the other.[4] Both positively and negatively worded questions can be included to limit response bias. Double barrelled questions (ones that ask two questions) and those with double negatives are not recommended. Leading questions should be avoided.

Closed-ended questions in which the participants choose one or more of the predetermined responses are easier and faster to answer but provide limited information. The list of responses for these questions should be exhaustive to include all reasonable alternatives, and also be mutually exclusive.[5] Open-ended questions requiring the respondents to answer in their own words take more time but result in a greater depth of information. They may also be more difficult to analyse. A mix of closed- and open-ended questions with closed-ended ones at the beginning is likely to keep the respondent interested and attract a high response rate.

Once a questionnaire is developed, it needs to be piloted/validated by a panel of experts before it is used in the real world. The two main dimensions tested are validity and reliability. Validity assesses whether the questionnaire measures what it is intended to measure, while reliability evaluates its ability to provide consistent results. Components of validity include face, content, criterion and construct validity among others. Face validity is the degree to which the questionnaire 'appears' to measure what it is expected to measure, in the opinion of experts and the respondents. It is usually measured informally and not quantitatively. Content validity tests whether the current questionnaire covers all the relevant items needed to answer the research question. The experts rate the questionnaire items for readability, clarity and comprehensiveness using dichotomous, Likert or some such scale, and an item-rated content validity index (I-CVI) is calculated. Scale-level CVI (S-CVI) is calculated from I-CVI and denotes the level of agreement between the experts. Criterion validity is the degree of correlation between the current test and a predetermined standard. However, for some questionnaires, a standard instrument may not be available. Construct validity is the degree to which the test actually measures what it claims to measure. It is the most valuable component of validity but is the most difficult to measure and deserves an article of its own. Reliability is usually tested over time (test–retest reliability or repeatability), across items (internal consistency) and across different researchers (inter-rater reliability). Test–retest reliability is assessed by the correlation between the scores measured at different times, while internal consistency is often evaluated using Cronbach's alpha, a measure derived from pair-wise correlations between various items. Inter-rater reliability assesses the degree of agreement between different raters usually utilising kappa statistics.[6] The questionnaire needs to be modified based on this validation. The revised questionnaire must be reviewed at this point to confirm that it still addresses the objectives of the survey.

There is no consensus on the size of a validation study. Some recommend a sample size of 100, while others recommend the size as a multiple (ranging from 5 to 30) of the number of items. The general advice is to take as large a sample size as possible.[7]

If a questionnaire already exists in another language, it can be used to avoid the exhaustive process of developing a new one. However, translation, cross-cultural adaptation and validation in the target population are needed. The translation should be conceptual rather than literal. The process requires forward translation from the source to the target language preferably by two independent translators who have the target language as their mother tongue. The next step, backward translation from the target to the source language, is again performed independently, preferably by two other translators.[7] The translations are then reviewed by a committee consisting of the investigators and the translators to resolve the discrepancies. Multiple iterations of this process may be required to get an accurate version of the questionnaire. Adaptation ensures that the question is culturally relevant in the target population and language while maintaining the conceptual meaning of the original question. For instance, while translating Medical Outcomes Study Short-Form Health Survey (MOS SF36) in Hindi, the example of 'vigorous activities such as lifting heavy objects' was changed to 'lifting a bucket of water' for better understanding by the target population.[8]

In their study published in this issue of IJA, Panjiar et al. surveyed the knowledge, attitude and practice of preoperative fasting guidelines among Indian anaesthesiologists online.[1] The questionnaire is well structured with items arranged in three sections. However, the exclusive use of closed-ended questions restricted the information that could be gathered. The questionnaire was subjected only to content validation. There was a selection bias as only those attending a conference and using Internet were included. Also, the response rate was low.

In the second survey published in this issue, Kar et al. translated the English version of MOS SF36 questionnaire to Telugu and used it to assess the quality of life in Indian post-thoracotomy patients.[2] The authors followed the due process of forward and backward translation and ensured conceptual equivalence of the questions, but did not validate the translated questionnaire before using it for the survey.

A study on design and validation of a questionnaire on preoperative fasting practices is also published in this issue.[3] It describes the development and content validation of the questionnaire, albeit incompletely. Authors and readers should realise that a comprehensive description of the process of design, development and validation of a new survey instrument is not only desirable but invaluable for the eventual publication of the survey itself. Survey instruments have a much longer life than the surveys for which they were originally designed and are cited far more often.

India is a vast country with multiple languages and cultures. Despite their limitations, we need studies such as the ones in this issue to understand our achievements and shortcomings, and to improve our practices. In future, more rigorous methods should be used when designing and conducting a survey. Also, the development of the questionnaire should be described in detail, if necessary in an appendix, so that the results can be interpreted appropriately.