The challenge of multi-country research

Blog Jori van de Spijker – Practice Lead Brand & Comms

‘The 24 best countries to work in’, ‘The Top 10 best countries to live in’. The internet is filled with international research where countries are compared with each other. Same goes for our international clients, where global teams map out the performance of different markets. With good research it’s possible. But there are a few pitfalls you have to watch out for, to prevent you from drawing completely wrong conclusions.

Cultural differences

A while ago I got in touch with a client that wanted to open a store in the Netherlands. They collected NPS-scores, and wanted to compare them with the stores they already had in other countries. Everyone with experience in multi-country research can predict what happens next: stores in Brazil, China and the US score high, whilst stores in Japan, Germany and the Netherlands score low. Is this because of the stores? No, not necessarily. Firstly, it’s because of cultural differences. Brazilians often always react enthusiastic, and score high on questionnaires. And well… us Dutch people think an 8 is an excellent grade, even when we are really content. For Dutch, German or Japanese marketers it’s not really fair, they always come out worse.

Where are these differences coming from?

Social norms
There isn’t just one reason. Firstly, the social norms are different. In the Netherlands we have a Calvinistic background. ‘Just act normal, then you’ll be crazy enough’. Not over the top. In for example the US, this is completely different. Over there, the ‘American Dream’ is the belief and it’s custom to express your satisfaction in ‘great’ or ‘awesome’.

Point system
In the Netherlands we grow up with report figures of 0-10, with a 7 or 8 being a good score. In the US where it’s A + to E, it’s very different. Americans tend to think in terms of extremes. A 9 or 10 is incredibly high by Dutch standards, while for Americans it isn’t that crazy. An A is much easier to give. Especially with an NPS question, this has a direct impact.

Hierarchy
Next up we have the hierarchy in a country. To what extent do you look up to others, how badly do you want to please other people? In countries such as India, people are inclined to say yes very quickly, to agree with something. Whilst we – and you can for instance also find this in the familiar culture dimensions of Hofstede – have a much lower power distance. In the Netherlands an intern would dare to knock on the CEO’s door with a good idea. Being critical, is seen as something positive. In other countries it’s known to be rude to contradict people and respondents are therefore more inclined to agree with a presented statement.

is it therefore not possible to conduct international research?

But what does this mean, is international research not possible as a result of this? Of course, it is. But it stands and falls on a good set-up of the study where you take the differences between countries into account. The best solution is to register behaviour, instead of questioning it. At DVJ we use a lot of such techniques, like measuring reaction time, the moment of zapping away or online eye-tracking. Therewith preventing people from reacting socially desirable, because you’re measuring actual behaviour.

Another option is letting respondents choose. Instead of asking to what extent someone agrees with a statement, you can ask them which of a set of statements is the most applicable. In China for example it’s not uncommon that 90%+ of the respondents agrees with all statements. By letting them make a choice, you learn what they actually find most applicable.

Finally, it’s important to work with country specific benchmarks. When we for example test a commercial, we always compare the scores with our database for commercials from that specific country. By placing it – in each country – in the context of what is a common score for that country, it is possible to map out the relative performance and compare this between countries. It’s important here not to fall back on a regional benchmark (e.g. the “Asia benchmark”). Differences between for example Japan and China are so huge that an Asia benchmark is meaningless.

Top 10

Today’s automation makes it easier for people to conduct their own surveys. But not without danger. There is no substantive expertise to do this correctly. Which not only leads to bad research, but even worse: incorrect conclusions. So next time you’re wanting to compare the performance of different markets, take a moment to think about the challenges. But more importantly, solve it with a smart and well-thought-out research design.