Ten Commandments for creating a survey


Charlton Heston - the ten commandments

Thanks to computer software and the web it is now easier than ever to create a survey. The technical side is so easy that people forget that there is more to it than just adding some questions to a form.

I was fortunate to attend a lecture by Marguerrite Cox, a biostatistician from Duke Clinical Research Institute in the USA, and I thought it might be useful to share her 10 Commandments.

(I have changed the examples to be more relevant to a tech audience)

Before you even start with the questions you need to:

  • Decide on the survey’s goals,

  • Identify the target population

  • Determine the sample size

  • Select a survey mode eg internet, phone, in-person et

Ten Commandments

1. Thou Shalt Not Be Vague

  • Know exactly what you want to ask and be specific and add a reference frame.
  • “How often do you install x per year?” instead of “how often you install x?”

2. Thou Shalt Not Make Unto Thee Complicated Questions

  • Be clear and simple
  • Avoid jargon
  • Avoid complex sentences

3. Thou Shalt Not Ask Double-barrelled Questions

  • Example: “How satisfied are you with the software’s support and documentation?”
  • Break these questions down into multiple questions:
  • “How satisfied are you with your software’s support?”
  • “How satisfied are you with your software’s documentation?”

4. Thou Shalt Avoid Leading Questions

  • Example: “Do you believe the documentation is good enough?”

5. Thou Shalt Ask as Few Questions as Possible

  • Trade-off between getting all the information you need and reducing the amount of non-responses

6. Honour Thy Order of Thy Questions

  • Questions should flow logically
  • Questions at the beginning of the survey may influence answers to later questions
  • Easy, general questions should be at the beginning
  • Sensitive questions should be placed at the end of the survey
  • If the questions are similar, random ordering can be helpful

7. Thou Shalt Anticipate All Possible Answer Choices

  • Example: “How would you describe your programming skills (check all that apply)?”:
  • novice, new to this language, still learning, moderate, experienced, practically invented the language, Other (specify), Prefer not to answer.

8. If Thou Uses an Ordinal Scale, Thou Shalt Use a Balanced Scale with 5-7 Choices and a Clear Middle Category

  • Ordinal data: categorical data where there is a clear and consistent ordering
  • Examples:
  • Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree 
  • Very good, good, fair, poor, very poor
  • Never, rarely, sometimes, most of the time, always, No chance
  • very little chance, unsure, some chance, very good chance
  • Just as the order of the questions is important, the order of the answer choices is important
  • Primacy Effect: respondents pick the first choice (paper and internet surveys)
  • Recency Effect: respondents pick the choice most recently heard (in-person and phone surveys)

9. Thou Shalt Not Have Overlapping Categories

  • How many web sites do you manage?”: 1-10, 10-20, 20-50, 50-75, Over 75
  • What if they manage 20? Some of your respondents may check 10-20, while some may check 20-50. Some may be confused and not respond.

10. Thou Shalt Do a Trial Run of Thy Survey

  • This will help you identify issues with questions and/or responses

A bonus – not so much a commandment but a universal truth!

everybody lies