Surveys are one of the most common research methods used by businesses to collect valuable data from both current and potential consumers. You can use a survey to:
- Research your target audience and collect data related to demographics and behaviour, gather feedback from your customer base enabling you to improve your product or service
- Determine whether there is a gap in the market before launching a new product or service.
In today’s digital world there are a plethora of platforms that will make the creation and dissemination of a survey seem like child’s play, but don’t let this fool you. There is an art to designing an effective survey that can be relied upon for generating credible data. In fact, many surveys are so poorly designed that the data they generate is frankly, meaningless.
If you are using survey data as the basis in which to make business decisions, you want to be assured of statistical accuracy, or at least be aware of the common mistakes that can corrupt or distort your data.
In the following section, we will provide you with an overview on how to design an effective survey including understanding the differences between qualitative and quantitative based approaches and the type of questions to ask to ensure your data is credible. We will include details on how to frame survey questions appropriately to avoid ambiguity and confirmation bias, as well as an introduction to survey sample size
Know what you want to investigate
To avoid unnecessary leg work it’s important to clearly define your research goals and objectives. Be specific about what you are trying to achieve and narrow the focus of your questions accordingly. Remember to keep it short and sweet, the more questions you include the less likely people are to respond. Aim for no more than five questions if possible. Be ruthless, be brutal. Constantly ask yourself is this information needed for my research? Do I really need to include this question?
Evaluate whether a qualitative or quantitative approach is best
Quantitative data is information that can be expressed in numerical format and is easily measured. How much? How often? How many? Are all examples of typical survey questions that can be expressed in a quantitative fashion. Because it is numerically based, analysing quantitative data is a relatively straightforward process.
Qualitative data includes information extracted from open ended questions that often take narrative form. For this reason, it is much more subjective, interpretive and exploratory. Qualitative data includes asking people about their thoughts, feelings and observations. These forms of questions are typically used when a researcher is attempting to understand people’s attitudes and motivations behind a certain behaviour or action.
To understand qualitative data, we need to properly contextualise it. This involves the process of conducting secondary research, meaning you need to supplement your survey results (your primary research) with information obtained from journal articles, previous studies and newspaper articles. This will enable you to paint a comprehensive picture and understand your survey respondents in a nuanced way. It’s important when conducting any sort of research to not immediately take information at face value. When evaluating qualitative data you may need to dig beyond the surface to understand your respondents’ true thoughts and feelings.
Asking open ended questions can provide researchers with rich, in depth data about their target audience. However, because qualitative data is less static and rigid than quantitative data it can be difficult to precisely measure and analyse. Qualitative data analysis typically involves a process commonly referred to as coding. We will explore this in some detail in our free e-book available for download here.
Types of Survey Questions to avoid
The leading question
When designing a survey, it’s vital to give careful consideration to the way in which you frame questions. In their haste to draw certain conclusions, inexperienced researchers often frame their survey questions inappropriately. One of the most common examples of this is asking leading questions.
Leading questions are loaded with a set of assumptions that hint at the answer the researcher is looking for and prompts the respondent to answer in a particular way. An example of a leading question may be;
Tell me about a problem you had with your teacher?
The nature of this question presumes that the respondent has had a combative relationship with their teacher and is probing them to provide examples of conflict in the context of a relationship that may otherwise have been quite amicable.
An example of a non-leading question would be;
Tell me about your relationship with your teacher?
If your survey questions are leading in this way it may aggravate your respondents, not to mention it compromises the integrity of your research. This can be considered a classic example of what is commonly referred to as confirmation bias. This is the tendency of researchers to cherry pick information and ideas that confirm what they already believe or want to be true, whilst neglecting or discounting information that suggests otherwise.
One of the easiest ways to guard against confirmation bias and ensure the legitimacy of your research is to get an outsider’s opinion. The higher the personal stakes and the more you have invested in a particular research outcome, the more likely you are to allow confirmation bias to seep in.
It’s not a coincidence that large companies often outsource research to an independent agency. Bringing in an objective third party to review your survey questions, one with no connection to your business, service or product, will reduce the likelihood that your research will be tainted with confirmation bias and will greatly improve the credibility of your research.
The Ambiguous Question
One of, is not the most common mistake people make when designing a survey is asking ambiguous questions. These are questions which are poorly defined and can be interpreted in a multitude of different ways. An example of an ambiguous question would be asking participants: “Do you believe the media reinforces negative stereotypes of people with mental illness?”
The language in this question is vague and imprecise. It’s not clear whether the term media refers to depictions in film and television, news networks, reality television social media, radio, podcasts or talk shows. The term media is so broad that its use in this context is highly inappropriate. Placing all these quite distinct mediums in the one category will not produce insightful nuanced data on which to base credible conclusions. Furthermore, it is equally unclear what is meant by the term negative stereotype or mental illness for that matter.
Public survey data in particular is often highly criticised as being too ambiguous and therefore leading to an incomplete or inaccurate understanding of people’s actual attitudes.
Even a question that may seem relatively straight forward like; “Do you consider our software better than other companies?” is too vague to generate meaningful data. Better how? In what way?
Ambiguous questions can be considered a form of confirmation bias as it leaves room for researchers to guess respondents’ true feelings or default to whatever answer suits them.
It is important to craft your survey questions with thought and precision. The meaning of a particular word or phrase may mean something to you and something quite different to someone else. Gut Check recommends replacing the word “you” in surveys with clearly defined phrases such as “you personally,” “you, yourself,” or “you or someone else in your home”.
Consider adding an “other” field to multiple choice survey questions to capture survey-takers true feelings. Yes, the feedback will be qualitative, but it will be more accurate.
The Confusing Question
You must use words and language that are familiar to your target audience. No-one can answer a question they don’t understand. You may know the difference between bites, bytes, poststructuralism and postmodernism but do your respondents? Don’t use acronyms, industry specific jargon or terminology when writing your survey questions. Make sure the vernacular you’re using is consistent with that used by your target population.
Further Reading
One of the most important things to remember when conducting a survey is sample size. This is used as a measure to determine the statistical accuracy of your results. Too small a sample size and your data will be deemed unreliable, too large and you’ve wasted time and energy. If you need a hand setting up a survey and then to understand and use the data, get in touch and let us assist you.