Masala and Meat, and Professional Integrity

Years ago, an ex-ambassador turned consultant for UNDP from a developed country remarked that I was been too critical of a number of survey reports in Myanmar. Well, my language might had been crude by diplomatic standards, but for statistical reports I believe more meat is better than more masala (spices) as my older friend, an architect, used to say. Think about a cv (coefficient of variation) of more than 300% reported for an indicator in a major nation-wide statistical survey. That means masala is three times, say, the weight of the meat. What kind of curry is that?

Obviously, in the definition cv = standard error of an indicator divided by the indicator, I'm equating the numerator with masala and the denominator as meat. Thus this formula measures for each piece of meat in your curry how much masala goes with it. Now you could go on debating about the right (optimal as the experts would say) combination of masala and meat, but survey statisticians used a working rule, a rule of thumb, like cv of more than 20% is suspect. US Census case studies say: (i) High reliability: CVs less than 15%, (ii) Medium Reliability: CVs between 15‐30% ‐ be careful (iii) Low Reliability: CVs over 30% ‐ use with extreme caution.

But having found that your cv is 300% what would you do? Tick one:

□       Sweep it underneath the carpet

□       Report as it is

I dare say that in the old days the first one will be ticked automatically. But now you'll think. Besides, survey statisticians noted that the order of response categories is important. So the order of response items in the above question may have to be reverse to make it fair, I don't know.

Back to the question; in my personal opinion, you should report as it is, and note somewhere in the report about the possible cause(s), and caution the reader on proper use.

Anyway, in our case the report came out intact with the cv of 300+ percent as it was found means the institution, the professionals, and the officers responsible merits great respect for their professional integrity and the report for its statistical integrity. And I would say that it is the kind of thing that makes people trust a public institution for—sincerity, regardless of occasional shortcomings and some irregular performances.