Friday, September 4, 2020

Adding syllable-rhymes level hyperlinks to Myanmar Dictionary


After the Myanmar Language Spelling book, my great find was the Myanmar to Myanmar Language dictionary called “ခရီးဆောင်မြန်မာအဘိဓါန်”(Traveler’s Myanmar Dictionary) in PDF format. It is the updated (1999) version of the Myanmar Dictionary published in 1991. I was lucky to own the five volume Myanmar Dictionary published 1978-1980 by the Myanmar Language Commission and didn’t know the 1991 version. My version contained the valuable front matter on the characteristics of the Myanmar Language and tips on the use of the Dictionary, as well a substantial amount of illustrations in beautiful line drawings. I don’t know about the intermediate version, this Traveler’s version include only the (i) guide to pronunciation, (ii) list of abbreviations, and (iii) alphabetical order, from among the characteristics of the Myanmar language and doesn’t include any illustrations at all.
The electronic Traveler’s dictionary circulating principally on the Facebook is a scanned version with clickable alphabet list that would take you to the first page of entries for a given alphabet and a back-key to take you back to the alphabet list on every page.
There is an application derived from this Traveler’s Dictionary both for Android and iPhones called the DMNL Myanmar Dictionary. This application uses the SQLite database and is a nice addition to Myanmar dictionaries. However not all information available in the Traveler’s dictionary, and the entries were limited to the head word and its definition(s), pronunciation, and grammatical information only. Alphabetical order is very complicated in our language and navigation through a dictionary is a nightmare even for native speakers.
As for the technology to make the navigation through the text fast and trouble-free, I guess the first choice would be the database approach, only if all the information in the printed dictionary is available!
Next, if all information could be converted to text, a PDF version would be fine with its default search-ability (though with some limitations for the Myanmar language). OCR on the currently available PDF file seems impossible because of the quite poor resolution.
So the only available choice for me to try to improve on the electronic Traveler’s Dictionary would be to extend what is available now. That is, to make the navigation to pages easier by extending hyperlinks to the basic syllable rhymes level.

Workflow

  1. Import the pages 1-48 of the dictionary as layers into GIMP. This corresponds to all the front matter and dictionary text body for the first consonant “က”. Add text on the cover page. Export as PDF pages.
  2. Merge the pages in PDFsam.
  3. Open the file in LibreOffice Draw and add hyperlinks to the syllable rhymes level. Export as PDF and is done.
For this post I’ll skip the first two steps and deal only with the third and last.

Open merged PDF in LibreOffice

Open the merged PDF in LibreOffice and it is opened in the Draw module:  

Duplicate page-7 and 8 and move to end


Go to page-7 by clicking in the Navigator window and open the Page menu. Click Duplicate Page.

This page is duplicated as page-8 (highlighted). Open the View menu and click Page Pane.

The current page (page-8) is seen selected in the (1) Page Pane. (2) Open Page menu and click Move and then select Move to End. Now the duplicate page is moved to end and becomes page-49. Do this to duplicate and move the original page-8 also.

Go to page-7 to add hyperlink for the first consonant “က”. (1) Open Shapes, select Rectangle, Rounded (2) Draw a rectangle and fill with orange color. You can do this by opening Format menu, clicking Area and selecting the orange color. Then select Text Box from the Insert menu and place it over the red rectange and type “က”. Its size can be increased or decreased using the Format/Text menu.

Adding text on the margins

This is independent of other components and could be done at any time. The syllabes corresponding to those on page-49 and 50 at typed-in at appropriate places enclosed in red boxes. They are meant to help the reader identify syllables if more than one exist on a particular page. Once the spelling book has been saved as a PDF file you could search these syllables with “Find” and get to the desired page.

Change the cover

Page-1 shows the original cover.

You can replace it with your own by first clicking View/Master at page-1, and a blank space appeared. Here click Page/Set Background Image and select your image.

Now the new image is displayed. Here you can see some margin left at the bottom. To eliminate it click Properties then Margin and select None.

Saving the file

Select Export As from the File menu to saved the file as PDF.
The output of this exercise “GPLO_MyanDict-[က].pdf” is shared here.

Friday, August 28, 2020

Checking the “Find” text search for PDF document


In my last post I noted that I couldn’t find the word “ကြွက်”(mouse in our language) even though it was clearly there in the spelling book. I was wondering if that was due to Google wrongly converting that word when it ran the OCR with Google Docs. Then I had some doubts on LibreOffice’s conversion of the text file output from Google to PDF. However, I noticed that when Google OCR went wrong, it showed up in the text output. For example, the word “ကစော့ပင်” from the original Spelling Book came out as “ကရော့ပင်”. After second thought, I suspected that difficulties might due to the application of text search function developed for the English language to an entirely different kind of language like ours.
In this exercise, I would use regex (Regular expression) to accurately search text in the Myanmar Spelling Book to find out if the OCR operation, PDF conversion, or the text search functions were at fault. The workflow would be, (i) create a text string containing the word(s) to be searched, and run search with regex, (i) import the spelling book PDF file into R, (iii) split the text of the spelling book to group the terms as they appear in the original book, (iv) do text search, (v) convert the spelling book text into a corpus using the Quanteda package, (v) kwic analysis with Quanteda.

Import the spelling book data

library(readtext)
rt_pdf <- readtext("C:/DATA/othSOFTWARE/searchable-PDF/mySPELL-book/LR-ocrPDF/MyanSpellBook(OCR)-ver0.pdf")

Search the verifying words “ကြိမ်” and “ကြ ိမ်”

y <- strsplit(corp_MyaSpell, "\n")
z <- unlist(y)
utf8::utf8_print(grep("ကြိမ်", z, value = TRUE))
MyanSpellBook(OCR)-ver0.pdf218                                                    
"ကျောင်းခေါ်ကြိမ် ကျောင်းသင်္ခမ်း ကျောင်းအမ"                                               
MyanSpellBook(OCR)-ver0.pdf366                                                    
"ကြိမ်ဖန်များစွာ"                                                                     
MyanSpellBook(OCR)-ver0.pdf370                                                    
"ကြိမ် (တစ်၊ နှစ်) ကြိမ်ကြိုက် ကြိမ်ခြင်း ကြိမ်ခြမ်းတံစဉ်း ကြိမ်ဒဏ် ကြိမ်ပိုက်တော ကြိမ်ပင် ကြိမ်ပျစ်ပင် ကြိမ်ဖန်များ…"
MyanSpellBook(OCR)-ver0.pdf371                                                    
"ကြိမ်လုံး ကြိမ်လုံး"                                                                     
MyanSpellBook(OCR)-ver0.pdf372                                                    
"ကြိမ်းမောင်း ကြိမ်းဝါး"                                                               
MyanSpellBook(OCR)-ver0.pdf2540                                                   
"ဖန်-(ဝါ- ) ဖန်(ကြိမ်-) ဖန်ခါးပင် ဖန်ခုန်ကစား ဖန်ခွက် ဖန်ဆင်း ဖန်တလီဖန်တလာ ဖန်တီဖန်တွတ် ဖန်တီး ဖန်တွတ်တွတ် ဖန်…"
MyanSpellBook(OCR)-ver0.pdf3081                                                   
"မျောက် မျောက်(သတ္တဝါ) မျောက်ကြိမ်ပင် မျောက်ခါးပတ်ပင် မျက်ခုံညင်းပင် မျောက်ချောပင် မျောက်အိုးဝါး မျေ…"
MyanSpellBook(OCR)-ver0.pdf3141                                                   
"မြင်းကောင်းခွာလိပ် မြင်းကျား မြင်းကြိမ် မြင်းခံ မြင်းခုံ(ထိုင်ခုံရှည်) မြင်းခုံတိုင် မြင်းစခုံနီပင် မြင်းဇောင်း …" 
MyanSpellBook(OCR)-ver0.pdf3397                                                   
"ရနယ်(အခြင် ~) ရဗနှီးဗန်း) ရမထာကြိမ်"                                                    
MyanSpellBook(OCR)-ver0.pdf3954                                                   
"ဝက်ကြိမ်ပင် ဝက်ကြွပ်ပင် ဝက်ခုတ်လယ် ဝက်ချေးပနဲပင် ဝက်ချုပ်နာ ဝက်ခြံ(လှည်း ~) ဝက်ခြံတင်းတိပ် ဝက်ပုတ် ဝက်ဖရုံပင်"     
MyanSpellBook(OCR)-ver0.pdf4281                                                   
"အကြတ်နေ့ (သင်္ကြန် ~) အကြိတ်အနယ် အကြပ်တပ်သား အကြပ်အဆင့် အကြမ်းဖျင်း အကြိမ်ကြိမ်အဖန်ဖန် အကွင်းအခြည်"      
utf8::utf8_print(grep("ကြ ိမ်", z, value = TRUE))
character(0)

Create corpus

library(quanteda)
corp_MyaSpell <- corpus(rt_pdf)
summary(corp_MyaSpell)

Corpus consisting of 1 document, showing 1 document:

                        Text Types Tokens Sentences
 MyanSpellBook(OCR)-ver0.pdf 12262  24707       292

Run Key Words in Context (kwic)

spelltoks <- tokens(corp_MyaSpell, "fastestword")
utf8::utf8_print(sapply(data.frame(kwic(spelltoks, "ကြိမ်",window = 2, valuetype = "regex")),paste0))
      docname                       from    to     
 [1,] "MyanSpellBook(OCR)-ver0.pdf" "747"   "747"  
 [2,] "MyanSpellBook(OCR)-ver0.pdf" "1108"  "1108" 
 [3,] "MyanSpellBook(OCR)-ver0.pdf" "1124"  "1124" 
 [4,] "MyanSpellBook(OCR)-ver0.pdf" "1127"  "1127" 
 [5,] "MyanSpellBook(OCR)-ver0.pdf" "1128"  "1128" 
 [6,] "MyanSpellBook(OCR)-ver0.pdf" "1129"  "1129" 
 [7,] "MyanSpellBook(OCR)-ver0.pdf" "1130"  "1130" 
 [8,] "MyanSpellBook(OCR)-ver0.pdf" "1131"  "1131" 
 [9,] "MyanSpellBook(OCR)-ver0.pdf" "1132"  "1132" 
[10,] "MyanSpellBook(OCR)-ver0.pdf" "1133"  "1133" 
[11,] "MyanSpellBook(OCR)-ver0.pdf" "1134"  "1134" 
[12,] "MyanSpellBook(OCR)-ver0.pdf" "1135"  "1135" 
[13,] "MyanSpellBook(OCR)-ver0.pdf" "1136"  "1136" 
[14,] "MyanSpellBook(OCR)-ver0.pdf" "6425"  "6425" 
[15,] "MyanSpellBook(OCR)-ver0.pdf" "7532"  "7532" 
[16,] "MyanSpellBook(OCR)-ver0.pdf" "7736"  "7736" 
[17,] "MyanSpellBook(OCR)-ver0.pdf" "8370"  "8370" 
[18,] "MyanSpellBook(OCR)-ver0.pdf" "9780"  "9780" 
[19,] "MyanSpellBook(OCR)-ver0.pdf" "10589" "10589"
      pre                                          
 [1,] "ကျောင်း-(ထိန်း-) ကျောင်း-(ဝမ်း)"                 
 [2,] "ကြပ်ထုပ်ထိုး ကြပ်ပူတိုက်"                             
 [3,] "ကြမ်းပိုးကောင် ကြမ်းပိုးတေလေ"                      
 [4,] "(တစ်၊ နှစ်)"                                   
 [5,] "နှစ်) ကြိမ်ကြိုက်"                                 
 [6,] "ကြိမ်ကြိုက် ကြိမ်ခြင်း"                             
 [7,] "ကြိမ်ခြင်း ကြိမ်ခြမ်းတံစဉ်း"                        
 [8,] "ကြိမ်ခြမ်းတံစဉ်း ကြိမ်ဒဏ်"                          
 [9,] "ကြိမ်ဒဏ် ကြိမ်ပိုက်တော"                             
[10,] "ကြိမ်ပိုက်တော ကြိမ်ပင်"                             
[11,] "ကြိမ်ပင် ကြိမ်ပျစ်ပင်"                             
[12,] "ကြိမ်ပျစ်ပင် ကြိမ်ဖန်များစွာ\nကြိမ်လုံး"                
[13,] "ကြိမ်ဖန်များစွာ\nကြိမ်လုံး ကြိမ်လုံး\nကြိမ်းမောင်း"        
[14,] "(ဖုဿရထား)\nဖန်-(ဝါ- )"                        
[15,] "မျက်ဝန်း\n\nမျောက် မျောက်(သတ္တဝါ)"               
[16,] "မြင်း(သတ္တဝါ)\nမြောင်း\nမြင်းကောင်းခွာလိပ် မြင်းကျား"
[17,] "~) ရဗနှီးဗန်း)"                                
[18,] "ဝံသာနု\nဝံ့-(ရဲ ~)"                              
[19,] "အကြပ်အဆင့် အကြမ်းဖျင်း"                          
      keyword                                             
 [1,] "ကျောင်းကန်\nကျဉ်နီကျဉ်နက်\nကျောင်းခေါ်ကြိမ်"                 
 [2,] "ကြပ်မတ်\nကြမ်\nကြမ္မာငင်\nကြိမ်ဖန်များစွာ\nကြမ်း\nကြမ်း(အခင်း)"
 [3,] "ကြမ်းပြင်\nကြိမ်"                                      
 [4,] "ကြိမ်ကြိုက်"                                            
 [5,] "ကြိမ်ခြင်း"                                           
 [6,] "ကြိမ်ခြမ်းတံစဉ်း"                                       
 [7,] "ကြိမ်ဒဏ်"                                             
 [8,] "ကြိမ်ပိုက်တော"                                          
 [9,] "ကြိမ်ပင်"                                             
[10,] "ကြိမ်ပျစ်ပင်"                                          
[11,] "ကြိမ်ဖန်များစွာ\nကြိမ်လုံး"                                
[12,] "ကြိမ်လုံး\nကြိမ်းမောင်း"                                  
[13,] "ကြိမ်းဝါး\nကြုံ"                                       
[14,] "ဖန်(ကြိမ်-)"                                          
[15,] "မျောက်ကြိမ်ပင်"                                        
[16,] "မြင်းကြိမ်"                                           
[17,] "ရမထာကြိမ်\nရမင်းပင်\nရမန်ခံ"                             
[18,] "ဝံ့စား\nဝက်\nဝက်ကြိမ်ပင်"                                 
[19,] "အကြိမ်ကြိမ်အဖန်ဖန်"                                      
      post                                  pattern
 [1,] "ကျောင်းသင်္ခမ်း ကျောင်းအမ\nကျိုင်း"          "ကြိမ်"  
 [2,] "ကြမ်းကြားလေကြား ကြမ်းကြော်"             "ကြိမ်"  
 [3,] "(တစ်၊ နှစ်)"                            "ကြိမ်"  
 [4,] "ကြိမ်ခြင်း ကြိမ်ခြမ်းတံစဉ်း"                 "ကြိမ်"  
 [5,] "ကြိမ်ခြမ်းတံစဉ်း ကြိမ်ဒဏ်"                   "ကြိမ်"  
 [6,] "ကြိမ်ဒဏ် ကြိမ်ပိုက်တော"                      "ကြိမ်"  
 [7,] "ကြိမ်ပိုက်တော ကြိမ်ပင်"                      "ကြိမ်"  
 [8,] "ကြိမ်ပင် ကြိမ်ပျစ်ပင်"                      "ကြိမ်"  
 [9,] "ကြိမ်ပျစ်ပင် ကြိမ်ဖန်များစွာ\nကြိမ်လုံး"         "ကြိမ်"  
[10,] "ကြိမ်ဖန်များစွာ\nကြိမ်လုံး ကြိမ်လုံး\nကြိမ်းမောင်း" "ကြိမ်"  
[11,] "ကြိမ်လုံး\nကြိမ်းမောင်း ကြိမ်းဝါး\nကြုံ"        "ကြိမ်"  
[12,] "ကြိမ်းဝါး\nကြုံ -(ပိန်-)"                  "ကြိမ်"  
[13,] "-(ပိန်-) ကြုံ"                           "ကြိမ်"  
[14,] "ဖန်ခါးပင် ဖန်ခုန်ကစား"                    "ကြိမ်"  
[15,] "မျောက်ခါးပတ်ပင် မျက်ခုံညင်းပင်"              "ကြိမ်"  
[16,] "မြင်းခံ မြင်းခုံ(ထိုင်ခုံရှည်)"                  "ကြိမ်"  
[17,] "ရမယ်ရှာ ရလဒ်(သင်္ချာသုံး)"                  "ကြိမ်"  
[18,] "ဝက်ကြွပ်ပင် ဝက်ခုတ်လယ်"                      "ကြိမ်"  
[19,] "အကွင်းအခြည်\nအကွပ်အထိန်း အကျွေးအမွေး"         "ကြိမ်"  
utf8::utf8_print(sapply(data.frame(kwic(spelltoks, "ကြွက်",window = 2, valuetype = "regex")),paste0))
      docname                       from   to    
 [1,] "MyanSpellBook(OCR)-ver0.pdf" "523"  "523" 
 [2,] "MyanSpellBook(OCR)-ver0.pdf" "1287" "1287"
 [3,] "MyanSpellBook(OCR)-ver0.pdf" "1288" "1288"
 [4,] "MyanSpellBook(OCR)-ver0.pdf" "1289" "1289"
 [5,] "MyanSpellBook(OCR)-ver0.pdf" "1290" "1290"
 [6,] "MyanSpellBook(OCR)-ver0.pdf" "1291" "1291"
 [7,] "MyanSpellBook(OCR)-ver0.pdf" "1292" "1292"
 [8,] "MyanSpellBook(OCR)-ver0.pdf" "1293" "1293"
 [9,] "MyanSpellBook(OCR)-ver0.pdf" "1294" "1294"
[10,] "MyanSpellBook(OCR)-ver0.pdf" "1295" "1295"
[11,] "MyanSpellBook(OCR)-ver0.pdf" "7160" "7160"
      pre                                       keyword         
 [1,] "ကယ်ချွတ် ကယ်ပေါက်"                            "ကယ်လောင်းကြွက်\nကိုယ်"
 [2,] "ကြွေးဆပ်\nကြွေးဆယ်မြိချ\nကြွေးဆယ်မြီချ ကြွေးမြီချ" "ကြွေးရင့်\nကြွက်"   
 [3,] "ကြွေးမြီချ ကြွေးရင့်\nကြွက်"                    "ကြွက်စုတ်"         
 [4,] "ကြွေးရင့်\nကြွက် ကြွက်စုတ်"                       "ကြွက်တက်"         
 [5,] "ကြွက်စုတ် ကြွက်တက်"                             "ကြွက်နားပေါင်းပင်" 
 [6,] "ကြွက်တက် ကြွက်နားပေါင်းပင်"                     "ကြွက်နို့"          
 [7,] "ကြွက်နားပေါင်းပင် ကြွက်နို့"                      "ကြွက်မြီးတန်း"     
 [8,] "ကြွက်နို့ ကြွက်မြီးတန်း"                          "ကြွက်လျှောက်"      
 [9,] "ကြွက်မြီးတန်း ကြွက်လျှောက်"                      "ကြွက်ဝမ်းဖြူ"      
[10,] "ကြွက်လျှောက် ကြွက်ဝမ်းဖြူ"                       "ကြွက်သား"        
[11,] "မီ ခံအုတ်"                                   "မီးခိုးကြွက်လျှောက်"  
      post                            pattern
 [1,] "ကိုယ်ကျိုးနည်း ကိုယ်ကျင့်တရား"            "ကြွက်"  
 [2,] "ကြွက်စုတ် ကြွက်တက်"                   "ကြွက်"  
 [3,] "ကြွက်တက် ကြွက်နားပေါင်းပင်"           "ကြွက်"  
 [4,] "ကြွက်နားပေါင်းပင် ကြွက်နို့"            "ကြွက်"  
 [5,] "ကြွက်နို့ ကြွက်မြီးတန်း"                "ကြွက်"  
 [6,] "ကြွက်မြီးတန်း ကြွက်လျှောက်"            "ကြွက်"  
 [7,] "ကြွက်လျှောက် ကြွက်ဝမ်းဖြူ"             "ကြွက်"  
 [8,] "ကြွက်ဝမ်းဖြူ ကြွက်သား"               "ကြွက်"  
 [9,] "ကြွက်သား (ခြေသလုံး-)\nခခယမ်းယမ်း"    "ကြွက်"  
[10,] "(ခြေသလုံး-)\nခခယမ်းယမ်း ခခွေးအက္ခရာ" "ကြွက်"  
[11,] "မီးခိုးတိတ် မီးခတ်ကျောက်"              "ကြွက်"  

Conclusion

Without a custom software for searching Myanmar text, text search using ready-made software based on the English language would have some inherent limitations. From the result of my exercise, I think, I will have to conclude that the cause of the difficulties I am trying to find out comes from such limitations.
Nevertheless, such ready-made tools are basically usable as-is with our language, I guess. With some ingenuity you may be able to devise some workarounds where a straight approach is not available. For example, if you can’t search for “ကြိမ်” you could search for “ကြ” and pick what you are looking for from the results. Another example is from an old book of 1847 (The Laws of Menoo) available on Archive.org. It contained pages interleaved in Myanmar and English. This book is a searchable PDF, but you could only search in English. Obviously if you could reach a particular English page through text search you get to the Myanmar language page. For example if you are looking for “သင်္ဂဟဝတ္ထု” and could guess that it begins with “Theng” you could search with Acrobat Reader and get 28 hits, out of which 4 would be what you are looking for.

Tuesday, August 25, 2020

Making Myanmar Spelling Book searchable- II


As I hinted at end of my previous post, the neat idea was to download the result of OCR with Google Docs as Plain Text(.txt) instead of (.odt) as I used to do.
Also, I don’t need to go into a fancy process like slicing up the image in GIMP to get single column text images. The Google Docs OCR could handle two columns well! That was what I thought. Yet when I tried out that idea, I found that instead of finding the entire right-column text to be positioned neatly below the entire left-column, they were not always that way. So I had to get back to working with the sliced left and right columns as mentioned in my previous post!
So my final work flow is:
  1. Extract pages 1-285 of the text (using PDFsam Basic software).
  2. Open the resulting pdf file in GIMP using the Open as Layers… option in the File menu.Now I have all 285 pages of pdf opened as graphic layers.
  3. Images layers were cut into left-column and right-column images using the Slice Using Guides option in the Image menu of GIMP.
  4. By installing the Export Layers plug-in for GIMP, I now have additional options in the File menu. I select the Export Layers option and export the layers as 570 separate pdf files.
  5. Merge the resulting pdf files to 8 pdf files (using PDFsam Basic software).
  6. Open Google Drive; import the pdf files; open each in Google Docs. This converted the text-image to editable text. Each is downloaded as a Plain Text(.txt) file.
  7. Text files were opened in LibreOffice Writer and font changed to Pyidaungsu (Myanmar Unicode font). All were then merged into a single odt file and converted to a a pdf file with the Writer.
The following shows the first page of this spelling book pdf file with one error marked:
The OCR process seems to be quite good, but I can’t recommend the output for use without some hard editing to eliminate OCR errors. I had tested the text search-ability and it doesn’t seem to have serious problems. However, when I tried to find “ကြွက်”(mouse in our language), that can’t be found. When I manually located that word in the file, then copy and pasted it on console I can see that there was a space in the string. See it at the first two lines in the box below. The next two lines shows the correct string.
Then when I input the string with the space, the search worked! Seems like there’s some problem with Google OCR because it was reading from a fully formed and clear image of that word. However, when I tried running the same search on my Android cell phone there wasn’t any problem!
> "ကြ ွက်"
[1] "က\u103c \u103dက\u103a"
> "ကြွက်" 
[1] "က\u103c\u103dက\u103a"
As for the looks, the output is completely different from the original pages. But the new version need not look like the original (a sample page is shown below) so long as it serves our purpose. For that matter, all we need to do would be to make sure that OCR errors are identified completely and corrected.