Does having validated employee survey questions matter?

People who are new to culture surveys invariably ask the textbook question: “Are these questions validated?”

It is a fair question, of course. People want to know that surveys have been well constructed and the questions or statements have been vetted and stress tested. And I can assure you that, yes, we use questions that are validated in the classic sense. They have been drawn from research, tested against large and diverse populations, and have been shown to measure the thing that they're meant to measure. Our Chief Scientist has also written about what exactly validation and significance mean in employee surveys. 

The question always makes me smile, though, because people sometimes ask it without really thinking through what it means. If I’m feeling cheeky, I’ll respond: “Valid for what?” Validation is about ensuring from a scientific point of view that a question can be used in a certain context, so I’m curious to know if the client has thought through how they will use the responses in the context of their organization.

I was working with a client who was frustrated because the consultant she had been collaborating with was suggesting one question and we were suggesting another. We both had a Ph.D. on our side to back up the validity of our questions and the client didn’t know which to choose.

I said to her, "At the end of the day, what matters is which question makes sense to you? This is your business, if you don't understand the question don't let anybody tell you that you need to ask it. You're the one who is going to have to do something with the results, you pick the question that you think makes sense to you.”

For best results, start with the validated questions, and then take them and make them your own. Learn from how and why they were written the way they were, then mold them to fit your organization.

Fundamentally, if you're not going to do something with the answer don't ask it.

There is such a thing as a valid but useless question

Some questions have been validated, but serve little purpose. My favorite useless validated question is one of the Gallup Q12: “I have a good friend at work.”

I know Gallup has done lots of research to show that if people have a good friend at work then they're more likely to be engaged, and that's nice. But what are you meant to do with that? Should you go and buy someone a dog? It's weird and crosses a lot of different boundaries. I get the intent, but I think it’s more designed to get commentary than to create a useful outcome.

There are also unvalidated yet useful questions

Conversely, unvalidated questions can be useful if phrased correctly. Where there are no validated questions that correspond with what you're interested in, there will often be a validated form to assist you to phrase a question.

For example, every company's values are different, so not every question about every value has been asked and validated. But there are models you can use like “I believe at our company value X is used on a day-to-day basis” or “Here is the experience we want you to have. Is it occurring?”

One question we ask in our internal surveys that other organizations don't is, “I am motivated by our desire to be a culture first company.” That very much goes to the heart of the company we are.

Should I ask that?

Checking that there is scientific validity behind survey questions is savvy, but you have to go beyond that. Survey design is not about a psychologist in an ivory tower dispensing validated questions; it's about thinking about the experience you want your people to have and then measuring if it's occurring. You should use the questions that help you answer that.

If there is one take-away from this article, it’s this: if you don't understand the question and what you’re going to do with the answer, don't ask it.

Ask the right questions – every time

Access 30+ science-backed and customizable survey templates with Culture Amp

Learn more