Confessions of a dumb blogger - IV: reproducible science

As young civil servant recruits, one of us talked about the Communist Manifesto at least once. It was such a long time I couldn't remember who introduced that topic and as usual we weren't that serious. In those days we must have been pretending to be the intelligentsia of the times hotly debating politics, arts, poverty, and science over a few cups of tea offered, almost every afternoon, by an artist friend.

It's forty-plus years since I've heard about another manifesto - A manifesto for reproducible science (NATURE HUMAN BEHAVIOUR 1, 0021 (2017) | DOI: 10.1038/s41562-016-0021). This time without company, peering hard at my laptop screen while sipping through a cup of instant white coffee brought back from Viet Nam by a family friend. This time I was looking through the search results on reproducible research and open science and this time too I was no smarter than the last time I was being exposed to “big” ideas.

Anyway, true to my philosophy of easier done than said, I would skim the related topics starting with Wikipedia or whatever catches my eyes first. When I got a little bit of a feeling of what the field looks like I would start looking for what I could do hands-on. I would say that this is the second step of the celebrated dictum for acquiring knowledge in our Myanmar language—suu-tuu-pyuu. To quote myself from an earlier post:

Learning Machine-Learning (or for that matter Data Mining, or Statistics, or …) is obviously easier said than done. Yet we may approach any kind of learning through a three-step process: စု- တု-ပြု(suu-tuu-pyuu) or accumulate-imitate-create, as we Myanmars used to say. Turning our conventional wisdom upside down, I would now suggest that at least for the accumulation step, it could be easier done than said!

I won't fumble with my own words to say something about open science and reproducible research because that would be quite useless. But I think I could say that what reproducible research, open science, and the like are doing is essentially asking the travelers before us to leave their tracks and stick their notes for a journey we have not done before and for “Which to discover we must travel too” as Rubaiyat puts it:

But shouldn't there be exceptions to “Not one returns to tell us of the Road ” relating to less sublime aspects of our lives? Within the vulgarities of our lives, or more importantly in science, to be really helpful should we not resist our selfish habit to share the fish only for the pot, but hide the spot where we cast our nets (ဟင်းစားဘဲပေးမယ် ကွန်ချက်တော့မပြဘူး)? And I know at least this example from Christopher Gandrud:

He went as far as to let you reproduce the book from its source code! Truly a technological “frying the carp with its own fat”(ငါးကြင်းဆီနဲ့ငါးကြင်းကြော်). The book carries this message:

2. Reproduce this book

This book practices what it preaches. It can be reproduced. I wrote the

book using the programs and methods that I describe. Full documentation

and source files can be found at the book’s GitHub repository. Feel free to

read and even use (within reason and with attribution, of course) the book’s

source code. You can find it at: https://GitHub.com/christophergandrud/

Rep-Res-Book.

If you can't go as far as Gandrud you could try to imitate him in a small way (I'm veering away from Gandrud because I haven't read him yet). Here's a new facility called R Notebooks added to RStudio in October 2016. The following excerpt from “Why I love R Notebooks” by R Views neatly summarizes what I want to know about the R Notebook and how I could use it:

So I tried to do data science with R Notebook, in order to (i) tell you about the details of creating the parallel coordinates plots shown in my post “Playing with microdata”, (ii) show you the output produced by executing the chunks of code written for that purpose, and (iii) share all those chunks of code in a single, reproducible document.

It went well and the resulting HTML file could be opened in any browser from where you could download the Rmd file:

However, posting of this HTML output to Blogger is not straightforward. On this problem Bart Rogiers, as recently as June of this year, wrote that you need to be an HTML/JavaScript/CSS expert to be able to publish HTML document as a Blogger post without destroying the formatting. His workaround consisted of this workflow: (i) Create your R script (ii) Compile your notebook (iii) Get self-contained html body code (without images) (iv) Modify image urls (v) Further modify HTML if necessary (vi) Publish.

Luckily, Kyle's workaround that I used for my previous post “Confessions of dumb blogger - II” still works, but not as good as Rogiers would like. All I need to know is how to, (i) take the Rmd file from the R Notebook creation process, and (ii) run the following code chunk:

# create "clean" HTML from Rmd file

library(knitr)

library(markdown)

knit("parCoord_notebook.Rmd") # produces a .md file

markdownToHTML("parCoord_notebook.md", "parCoord_notebook.html",         fragment.only=TRUE) # produces clean .html

The HTML file so produced couldn't be exactly like the R Notebook HTML file shown in the picture above, but comes complete with narrative, codes and images (plots), but without the heading of the post, namely, “R Notebook version of my parallel coordinates plots”, and without the button on the page that allows you to download the Rmd file. This Blogger-HTML version of my Notebook will be published as my next post. The only formatting I've done to this file before posting is to open it in Notepad++ and use its facility to remove blank lines.

The Rmd file is what makes the R Notebook fully reproducible. When you run it in the RStudio, the complete R Notebook is produced. RStudio provides services for you to share documents publicly or privately. You may as well use email or some cloud service to share them.

I've uploaded the full HTML file of the Notebook (parCoord_notebook.nb.html) to the Google Drive. You could download it from here.

Zawgyi's beard, butterfly gamon, or black bat flower

Recently I've had the chance to see one of our nieces call for help on the Facebook to identify a flower via my wife's page. I at once recognized it through its shape to be what we knew as butterfly gamon which my late big sister grew a long time ago.

I googled for ဂမုန်းလိပ်ပြာ and found the following page on Wikipedia in Myanmar language:

Searching with the image of the flower on Google I found out its scientific name to be Tacca chantrieri. Looking up, I found Wikipedia's entry as:

Comparing the Myanmar version with the English version of the description of this flower I was unhappy because the Myanmar version seems to be relying too much on folklore and falls short on science as if the author(s) were entirely unaware of the English version. They could at least have given its scientific name in the Myanmar version, I thought.

This reminds me of the way modern day researchers criticized the math genius Ramanujan when he wrote about squaring the circle (Arndt and Haenel 2001, Pi Unleashed, p. 58).

What I would like to say is that including elements of Myanmar folklore in an Wikipedia article certainly makes it colorful and interesting. But they have to be pointed out as such. For example, instead of writing like

One who cares for the Zawgyi gamon is likely to win in lottery or Its leaves should not be cut off. If done, quarrels between husband and wife are likely to happen or A tea-cupful of liquid extract obtained by grinding its leaves taken for about ten days cure the coughing up of blood (consumption),

quotation marks could be placed around them. Or a phrase like “Many believe that ...” could be added to make it more explicit that we are dealing with folklore.

The Wikipedia's philosophy that an article like Zawgyi Beard Gamon which ranks as a Stub will have contributions to expand and improve it as it called for and hoped, didn't materialize for this particular case and maybe for many more. I guess that is because we Myanmars were so late in getting interested in, and involved with, Wikipedia or other sites and services on the Internet, except of course, the immensely popular Facebook.

If we Myanmars were not interested, wouldn't any non-Myanmars be? We don't know. But if they were interested, most of them may try for a Google translation, for example, to make sense out of an article like “Zawgyi Beard Gamon” in Myanmar language. This is what they would get now:

What you see is a translation where the original is not at all recognizable. It is distorted and looks funny. But in reality, it isn't a laughing matter at all. Yes, Google Translation has problems but it may not be entirely Google's fault because it is successful with other languages.

Google Translate first added Myanmar language in December 2014. According to the official Google Translate Blog:

• Myanmar (Burmese, မြန်မာစာ) is the official language of Myanmar with 33 million native speakers. Myanmar language has been in the works for a long time as it's a challenging language for automatic translation, both from language structure and font encoding perspectives. While our system understands different Myanmar inputs, we encourage the use of open standards and therefore only output Myanmar translations in Unicode. ...

We’re just getting started with these new languages and have a long way to go. You can help us by suggesting your corrections using "Improve this translation" functionality on Translate and contributing to Translate Community.

Well, Google Translate is using 'neural machine translation engine - Google Neural Machine Translation (GNMT) - which translates "whole sentences at a time", rather than just piece by piece'. This sounded to me like they are doing something very advanced and very good.

While the technology of the translation engine is way beyond our heads, it is not hard to understand that it needs data to use in its process of grinding out translations . For that Google Translate seems to need at least a collection of the same text in a pair of languages (Myanmar and English versions, for example) of more than 150-200 million words, and another collection of more than a billion words each for Myanmar and English separately.

So it seems we could improve translations from Myanmar to other languages (not only with Google's GNMT, but possibly with other approaches) generally by making available a wider range of material to work on. That means making documents in Myanmar language in digital format widely available and in big volume - the bigger the better. That also means making sure they are in Unicode format. Why? Because, it seems so obvious.

Confessions of a dumb blogger- III

Instead of the “clean” HTML as in my last post, what if I knit the .Rmd document to get the standard output? I tried that and then opened the result in Chrome browser. I got a beautifully formatted page.

Can blogger handle this HTML file?

To test that I opened this file in Chrome and viewed and copied its page source. In Blogger I pasted the source code into the HTML sheet. The preview showed the above post, but it didn't appear right. If you know HTML programming, you may be able to edit it, but I guess there will be quite some work to do!

First, let's look at the page source in Chrome. For the benefit of myself and my fellow dummies I exported the page source to pdf format in letter page size. It was a staggering 122 pages with only the last three pages contain the information we want to display.

So, Kyle was right: “the HTML file that R produces is probably going to be swarmed with hundreds (or, gasp, thousands) of lines of disgusting Javascript”.

On the other hand, Kyle was talking about rendering RMarkdown files to HTML for blogging. I guess that the judgment “disgusting” is only relative, and if only we were to be working for reproducible research these would turn into lines of “lovely” Javascript. But that's another matter.

For our purpose if we could remove the superfluous matter from this source file manually, we might get the slim HTML source we want to get. So I tried using Notepad++ for editing the HTML page.

Here's what you see when the HTML file is opened in Notepad++.

Here I could try deleting what looks like superfluous matter (remember I have zero knowledge of HTML) so that I would get “clean” HTML. Then I could open it in Chrome (or other browser) via the “Run” menu. If it is not right I could undelete and try again.

And then through trial and error I finally ended up with exactly the same “clean” HTML as with Kyle's approach (see my last post). Well, that's not surprising at all because Kyle's approach is just the programmatic way of doing what I've done manually, with the added benefit of having no room for error!

Confessions of a dumb blogger- II

I was perfectly happy with my make do solution of Open Office-StackEdit-Gimp trio for publishing to Blogger. However, I did strayed a little bit into—to choose the biggest name for a related subject—reproducible research. As noted in my last post, replacing pictures of R code in my posts with code that could be copied and run in R has been my significant achievement only very recently.

Without really understanding what is involved in reproducible research, I like this claim that “reproducibility was essential to pass wisdom on to the next generation”. For me, I should rephrase it as reproducibility is essential to pass the sense of healthy curiosity to my fellow dummies in the context of my blogs by way of allowing “the readers of my paper to verify and reproduce my computational experiments”. Obviously, the word post has to be substituted for “paper”.

So it is natural that I would peep into the world of reproducible research which the CRAN Task View says:

“The goal of reproducible research is to tie specific instructions to data analysis and experimental data so that scholarship can be recreated, better understood and verified. Packages in R for this purpose can be split into groups for: literate programming, package reproducibility, code/data formatting tools, format convertors, and object caching. … The primary way that R facilitates reproducible research is using a document that is a combination of content and data analysis code.”

I wasn't too serious or ambitious with reproducible research. It was way above me. If I could grab some technique from there for packaging content and data analysis code to be able to publish to Blogger in a tidy way, I would be more than happy and I guess R Markdown may be the right package for that.

With the last quote (from Kyle Wurtz, September 21, 2014) of my last post, the author warned of HTML produced by RStudio would be too sophisticated to be handled by your blogging site. He then showed a work around:

It’s quite easy to get around this by using the knitr and markdown packages to manually to render your markdown file to HTML. The first step is to use knitr to turn the .Rmd into a regular .md markdown file. The second step is to use the markdownToHTML function from the markdown package to render the .md to an HTML file. The key is in the second step – the default for fragment.only is FALSE, which embeds all that nasty Javascript into the HTML file. But setting this to TRUE produces an HTML file without the header and body tags, CSS, and Javascript.

I followed Kyle's advice and it worked as good as using StackEdit. The difference is that I need to do a bit more work. Again I start with my post written in Open Office Writer, complete with narrative, R code, and pictures. I assume you know just a little bit of working with RStudio (like me), so:

1. I created a new project in RStudio and created a new Rmarkdown document with the default option to output an HTML document. For that purpose I already have my example post with the title “One Thousand Decimal Digits of Pi” prepared with Open Office Writer. My idea is to present my R code for creating 1000 decimal digits of Pi using a Spigot algorithm.
2. In the R Markdown pane of RStudio, I copied and pasted the first part of the narrative:

3. The block of R code is pasted between line-18 and line-84.
4. The last line of narrative is pasted at line-85.

5. The R Markdown file is saved as spigotPI.Rmd.
6. As advised by Kyle I run these four lines on the console:
   library(knitr)

library(markdown)

knit("spigotPI.Rmd") # produces a .md file

markdownToHTML("spigotPI.md", "spigotPI_1.html", fragment.only=TRUE) # produces clean .html

5. I previewed the resulting "spigotPI_1.html" file in Blogger. It came out well with narrative, code, and output of 1000 pi digits—except for the name of the first author—on the first line of my example blog post shown below.
   
   

One Thousand Decimal Digits of Pi

“According to Jörg Arndt and Christoph Haenel, thirty-nine digits are sufficient to perform most cosmological calculations, because that is the accuracy necessary to calculate the volume of the known universe with a precision of one atom.” (Pi - Wikipedia, the free encyclopedia)

For everyday use pi=3.1416 would be quite sufficient. Being human, crunching out record breaking number of digits, in trillions now-a-days, is the full time job of some serious scientists as well as amateurs. Now with a laptop and some limited knowledge, anyone can produce a respectable number of pi digits way beyond the need for measuring the dimensions of the universe!
The R script below was an improvement over my attempt at generating 1000 decimal digits of pi using my laptop with i5 processor, 8GB RAM, and a method of calculation known as a spigot algorithm. The output of the original script was shown in my post Tea-shop PI- I of September 10, 2014.

    # Using Gosper's page's formula from
    # "The world of Pi - Spigot algorithm.pdf" by Rabinowitz et al.
    # Faster than orginal spigot (42.96 secs), here (17.34 secs)

    # (1) Initialization
    ## n is the desired number of decimal digits
    n <- 1002
    len <- ceiling(0.9*n)
    p <- 0
    nines <- 0
    ## create array and initialize, subscript 1 to len
    a <- array(dim=len)
    a <- (function(x){ 3+5*x })(0:(len-1))
    # (2) iteration
    for (j in 1:n) {
       ## multiply with new base 10
       a <- a*10

       ## normalize
       for (i in len:2){

         q <- a[i]%/% ((4+3*(i-2))*(4+3*(i-2)+1)*3)
         r <- a[i]%% ((4+3*(i-2))*(4+3*(i-2)+1)*3)

         a[i] <- r
         a[i-1] <- a[i-1]+ (q*(1+i-2)*(2*(i-2)+1))
       } # end for i

       ## calculate next provisional digit of pi
       q <- a[1]%/%10
       #p <- q
       a[1] <- a[1]%%10 

       ## correct the old provisional digits

       if (q == 9){
                nines <- nines + 1  # no digits ouput
          } else if (q == 10){
                p <- p + 1
            write(p,file="Spigot2-PI-digits.txt",append=TRUE)
                if (nines > 0){
                  for (k in 1:nines){
                write(0,file="Spigot2-PI-digits.txt",append=TRUE)
                  } # end for
                }
                p <- 0
                nines <- 0
           } else {
            write(p,file="Spigot2-PI-digits.txt",append=TRUE)
                if (nines > 0){
                  for (k in 1:nines){
                write(9,file="Spigot2-PI-digits.txt",append=TRUE)
                  } # end for
                }
                p <- q
                nines <- 0
          } #end if
    } # end for j

    # sink()
    # rm(list=ls())
    x <- scan("Spigot2-PI-digits.txt")
    write(x,file="",ncolumns = 50, sep="")

## 03141592653589793238462643383279502884197169399375
## 10582097494459230781640628620899862803482534211706
## 79821480865132823066470938446095505822317253594081
## 28481117450284102701938521105559644622948954930381
## 96442881097566593344612847564823378678316527120190
## 91456485669234603486104543266482133936072602491412
## 73724587006606315588174881520920962829254091715364
## 36789259036001133053054882046652138414695194151160
## 94330572703657595919530921861173819326117931051185
## 48074462379962749567351885752724891227938183011949
## 12983367336244065664308602139494639522473719070217
## 98609437027705392171762931767523846748184676694051
## 32000568127145263560827785771342757789609173637178
## 72146844090122495343014654958537105079227968925892
## 35420199561121290219608640344181598136297747713099
## 60518707211349999998372978049951059731732816096318
## 59502445945534690830264252230825334468503526193118
## 81710100031378387528865875332083814206171776691473
## 03598253490428755468731159562863882353787593751957
## 78185778053217122680661300192787661119590921642019
## 89

    # end spigot

Conclusion: If this dummy can do pi digits, so can you!

Confessions of a dumb blogger

I was dumb enough not to know how to get R codes into my posts right until in the last few of them.

From the beginning, I wrote my posts in MS Word and later in Open Office Writer. I created my pictures in GIMP. Then I copy and paste the text of the post on to the “Compose” sheet for my new post in Blogger. I never bothered to learn HTML and so I didn't select the other option, that is, “HTML”. The text was OK. Next, I upload my pictures and place them in my post, and it was OK too.

But when I paste my text containing R code, it went funny. For example

would come out on the blog as

So, what I did was make a picture of the R code by first converting the text of the code in Open Office Writer to pdf format, and then converting it into a picture with GIMP. It worked, but the resolution of the code wasn't that good. Also, the downside of it was that you couldn't copy the code from my post and run it on the R console directly or via a script file.

Well, that isn't my problem, I must have thought, until recently. Then, I tried inserting R code that was more than a page long to my post in the usual way via an image. But the result turned out to be ugly!

So I tried processing my post first by converting my draft directly to HTML in Open Office Writer. But it didn't work. Next I tried using Notepad++ for conversion to HTML and it didn't work either. May be because I didn't know them well enough and may be I should visit them again later.

Now I can see that the obvious solution will be to type my post out on the “Compose” sheet of Blogger. But it seemed like the dumbest thing to do, even for a dummy like me. Since the text of my posts which have been composed with Open Office Writer could be pasted without problem on to the compose sheet of Blogger, the next step in my preferred workflow would be to correctly add my R code by copy and paste from my R scripts, and finally to upload and paste my pictures. As I've shown earlier, I would need to format my codes in some way so that they would be displayed correctly in my post.

So I looked around for some solution and luckily found a number of alternatives. Generally I would need to look for some kind of markdown language since we were told that the Markdown language was created with the goal of enabling people "to write using an easy-to-read, easy-to-write plain text format, and optionally convert it to structurally valid XHTML(or HTML)". That led me to StackEdit, though after a good deal of false leads and frustrations.

To start with, like a lot of other guys, I thought that if I could convert my post into Google docs format I would be able copy and paste it to Blogger. But in reality it was not to be.

Fortunately I found that I could just copy the content from my Open Office Writer document and paste it directly to Blogger! However, R codes and data tables will not come out correctly and images will not come out at all. Anyway, I wanted to confirm if it is really impossible to copy and paste from Google docs to Blogger and found that it is indeed impossible (in a post from Blogger Help Forum here). Besides, a number of alternatives were suggested and there I found mmplex's suggestion for StackEdit attractive as “a free online markdown editor that allows directly posting to blogger. You can edit offline also and synchronize your documents with drive or dropbox. It even works pretty well in a tablet. Notice that it's far more powerful than docs if you post math formulas or code snippets.”

I added StackEdit application from the chrome web store:

(1) Now let's write my toy example in Open Office Writer:

(2) Then I copy and paste it in Blogger compose sheet:

(3) Looks fine, but … preview gives:

(4) So I open the StackEdit application and paste my two lines of R code.

(5) Then I click the hash button in the navigation bar and choose to save it to disk as HTML.

What I need to do now is to insert my two lines of R code formatted as HTML into my unfinished blogger post previewed in step (3) above. To finish it -

(6) I open the saved HTML file of R code in Chrome, view page source, copy it and replace the incorrectly formatted lines of R code in the HTML sheet in Blogger

and finally get it right!

About the line “Written with StackEdit.” added by the StackEdit application, the answer by benweet of October 19, 2013 said:

“I'm the creator of StackEdit, which is designed to write Markdown documents and blog posts, not to write questions/answers on Stack Exchange. As mentioned in StackEdit's settings, this footer is to encourage users to support the project by adding a backlink in their documents/posts. It's there by default but it's fully removable/customizable.

Since it's a free application and it has no budget for advertising, I'm opposed to removing it from the application. However I will add few words about this issue in the settings or in the welcome document.”

Well, I chose to delete that line before I was aware of benweet's reply, and still do so. I hope he won't mind.

As for this post, you'll have noticed that I said nothing about formatting and other goodies of StackEdit. That's because I still need to learn … and as for you, stray further if you like. Have fun, and do experiment!

Still one thing. Why was such a fuss that Open Office Writer could convert my post to HTML with ease, but wouldn't come out correctly in Blogger?

This remark refers to HTML produced by RStudio but it could as well be applicable to HTML produced by Open Office Writer, or some other application:

“You might think that you’d simply be able to open up the source HTML in a text editor, copy the source code, and paste it into your blog editor as HTML. But the HTML file that R produces is probably going to be swarmed with hundreds (or, gasp, thousands) of lines of disgusting Javascript. This almost certainly won’t be supported by your blogging site, and you’ll end up with a blank page.”

Playing with microdata IV: response rates

The World Bank Group Country Survey FY 2014 for Myanmar which has been the subject of my last three posts has a low overall response rate of just 26%. Clearly, users may have to look beyond the survey data to make up their minds if the data reflect the opinions of the respective stakeholder groups.

That led me to look for the response rate situation of the opinion surveys in general. This paper, Understanding non response rates: insights from 600,000 opinion surveys by Germ´an Reyes, June 2016, seems to be a good one to look into. Contrary to my expectations, the paper was based on opinion surveys from Latin America, Spain, and more recently from the Caribbean and United States rather than a more global coverage. The data came from two databases arising from two complementary public opinion surveys:

The first one is conducted by Latinobar´ometro in 18 Latin American countries and Spain since 1995, interviewing about 1,200 individuals per country each year (with the exception of 2012 and 2014) about individual socioeconomic background, and preferences towards social and political issues. The second database comes from the Latin American Public Opinion Project (LAPOP). In addition to the countries covered by Latinobar´ometro, this database includes information from the Caribbean and United states, although the time span covered is shorter since the survey started in 2004, and the survey wasn’t conducted in 2011 nor 2013. LAPOP also interviews about 1,200 individuals per country each year about political issues. In both cases samples are representative at the national level for the voting-age population.

I was looking for response rates, that is, the percentage of respondents who responded from among the persons selected for interviewing. Alternatively put, I was looking for the opposite of those who didn't respond (the unit non-response). The paper however targeted at rates of non-response by each questions or group of questions (item non-response). In estimating the item non-response rates, denominator used in this paper was the total number of questions asked to an individual respondent and the numerator was the number of questions he/she chose not to answer. This ratio was then used as the dependent variable in regression analysis.

To my dismay I couldn't find any data on unit non-response rates in the paper and so I had to set a more modest objective of comparing unit non-response rates for World Bank Group Country Surveys from other countries in the ASEAN. Anyway, the figure-A2 shows that the public policies category had the lowest non-response rates compared to democracy, economy and socioeconomic categories. My guess is that it is so because we stand to lose or gain directly from public policies and our instinct, naturally, would be to let our voices heard on these matters.

If so why would we have a remarkably low response rate of 26% (a unit non-response rate of 74%)?

Some response rates for the World Bank Group Country Surveys for the ASEAN were:

Country	Number responded	Response rate	Year
Indonesia	316	29%	2015
Lao PDR	532	52%	2012
Lao PDR	164	45%	2015
Malaysia	258	30%	2016
Myanmar	173	26%	2014
Philippines	352	16%	2016
Vietnam	632	43%	2017

I am a bit more familiar with household surveys and I guess such big non-response rates would have been big problems there. Cited by Massey and Tourangeau response rate of less than 80% is officially considered to deserve serious attention in statistical surveys in the US:

The Office of Management and Budget’s Standards and Guidelines for Statistical Surveys, issued in 2006, recommend that investigators carry out studies like these to estimate the level of nonresponse bias whenever the response rate for a survey falls below 80 percent (http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/statpolicy/standards_stat_surveys.pdf).

Before looking at the list of response rates for some ASEAN countries above, I thought that countries like Indonesia, Malaysia, and Philippines with more mature statistical systems and greater exposure of the public to sample surveys/opinion polls would have considerably better response rates than us. But in reality, even for the Malaysia with a higher response rate, 70% of stakeholder failed to respond! Perhaps opinion survey people have some tricks for living comfortably with such problems. Perhaps we could console ourselves that we were not that bad as far as response rates go in the ASEAN.

But can't we get a better response rate and what is hindering that? For me, I would've thought a little and write down my three development priorities straight. But think again, those stakeholders, those experts, were not answering that question only. The overall objective of the World Bank Group Country Survey in Myanmar was for “gaining a better understanding of how stakeholders in Myanmar perceive the Bank Group”. And for that matter they were to answer questions relating to:

Their overall attitudes toward the World Bank Group in Myanmar;

Overall impressions of the World Bank Group’s operations, knowledge work and activities, and communication and information sharing in Myanmar;

Perceptions of the World Bank Group’s future role in Myanmar.

I don't know what the stakeholders would have thought. But for a dumb guy like me, those questions look formidable. Despite the assurance of confidentiality, what if I couldn't come up with answers that were not intelligent enough? Like me, those stakeholders who didn't respond were worried that their reputations would be at stake in answering those questions and so they chose to play safe, I imagined.

For some time I've been laboring with this feeling of hopelessness for all our peoples trying to make a peaceful and simple living and playing our part with dignity in our planet, because there have been so many distractions within and without. I don't really know if I've cheated myself being convinced that there are all-purpose solutions to these, and slept well thereafter. Seems like we need to get more educated (get more ပညာ) to face all internal issues and look for effective “small state survival strategies” for external affairs. I guess getting more educated would mean getting better informed, getting more knowledge, and getting wiser. I'm not smart enough to say what underpins “small state survival”, but I could mention successful examples like Switzerland and Singapore, among others.

One glossary of Pali terms gives ပညာ as:

paññā:Discernment; insight; wisdom; intelligence; common sense; ingenuity. One of the ten perfections (pāramīs).

In the context of this post, the respondents don't need intelligence or ingenuity to outsmart the data collectors; they could just use their common sense and be strong enough to speak out, I would say. What is there to be afraid? After-all, we have to be masters of our own fate.

Playing with microdata III: parallel sets etc

This sample of parallel sets plots gives us colorful crisp images with clear messages to match. Their favorite theme of illustration seemed to be the casualties of the Titanic disaster and as I can see parallel sets spares you the effort to create a vivid mental image of the tragic true story. This is from Kosara_BeautifulVis_2010.pdf .

My attempt at visualization of development priorities by stakeholder groups which has been the theme of my last two posts, now using parallel sets, wasn't so clear:

This may be because every stakeholder respondent has the choice of up to three development priorities out of the given thirty-five, making the result too complex for clear visualization. Justin in Games of Thrones Parallel Sets Data Visualization, gives his rule of thumb:

What can we do with our complex data? May be we could follow the lead of Markus (from my last post):

His idea is to simplify the data by taking the top 10 distinct three-response sets by number of occurrences. For this we first need to convert the dataframe based on individual respondents to grouped dataframe with distinct response triplets (r1-r2-r3) and their frequencies (n), like this:

    r1 r2 r3 n

 42 11 19 20 5

 65 19 20 21 5

  9  4  5 11 3

 61 11 20 21 3

121 18 20 34 3

 10  3  9 11 2

 23  4  5 20 2

 26  3  9 20 2

 28  2 13 20 2

 33  4 14 20 2

This resulted in the parallel sets plot:

So far so good. But the problem is that the top-10 response categories just covered 29 respondents out of 170 (17%), and biggest n is 5 only! Also, going down the rows of the dataframe is useless because we'll be getting down to frequencies of 1.

This is the script for producing my two parallel sets plots:

        # parallel sets plot using ggparallel package
        # myint thann, July 28, 2017

        # import data
        library(foreign)
        x <- read.dta("myanmar_cs_fy14_datafile_with_dk_.dta")

        # ---- extract and manipulate data
        # (1) extract data on development priorities + stakeholder groups       
        xa2g1r <- x[,c(1,3:37,447)]                         
        # (2) convert factors to integer
        xa2g1r[,2:36] <- lapply(xa2g1r[,2:36], as.integer)      
        # (3) convert NAs to 0 in development priorities
        xa2g1r[,2:36][is.na(xa2g1r[,2:36])] <- 0 
        # (4) convert 2 to 0 in development priorities
        xa2g1r[,2:36][xa2g1r[,2:36]==2] <- 0
        # (5) remove persons with no response in development priorities
        xa2g1r <- xa2g1r[rowSums(xa2g1r[,2:36])>0,]         
        # ---- end extract and manipulate data

        # (6) create response data
        xa2g1r$r1 <- 0
        xa2g1r$r2 <- 0
        xa2g1r$r3 <- 0

        # (7) create first, second, and third responses
        for (j in 1:170) {
            k = 38
            for(i in 2:36){
                if (xa2g1r[j,i]==1) {
                    xa2g1r[j,k] <- i
                    k=k+1
                }
                if (k > 40) break
            }
        }

        ###  run parallel sets plot
        #  (8) run parallel sets plot for all responses 
        library(ggparallel)
        ggparallel(vars = list("r1",  "r2",  "r3"), xa2g1r, method="angle")+
                         coord_flip()
        #  save plot
        ggsave("perset_all.png", width = 5, height = 5)

        #  run parallel sets plot for top ten most frequent responses 
        #  (9) create grouped data frame with number of respondents 
        #        for each distinct set of three responses
        xa2g1r_G <- xa2g1r[,38:40]
        xa2g1r_G <- aggregate(list(n=rep(1,nrow(xa2g1r_G))), xa2g1r_G, length)
        xa2g1r_G10 <- xa2g1r_G[order(xa2g1r_G$n, decreasing=TRUE),][1:10,]
        #  (10) run parallel sets plot for top ten responses
        ggparallel(list('r1', 'r2', 'r3'), xa2g1r_G10, weight = 'n', order = 0)
        #  save plot
        ggsave("perset_top10.png", width = 5, height = 5)

Notes:

• In step (7) of my script above, I used loops to create data for regular dataframe with r1, r2, and r3. Attempting to avoid loops, I needed 10 steps to arrive at the equivalent dataframe used in my last two posts, poor me. You could certainly do that better.
• Markus's code for producing the grouped data frame used the dplyr package:

I used one of the three answers to Find how many times duplicated rows repeat in R data frame in stackoverflow by thelatemail that uses the aggregate function (see step-(9) in my R script above). Kudos to the versatility of R and the power of Q/A sites.