The Choices of a Simplification System

There are many choices to make when building an LS system. In my experience there are three big decisions to take: the target audience; the source documents; and the mode of presentation. Let's look at each of these in detail.

Target Audience

Firstly, you need a well defined group to aim your simplification at.  This group should have a clear style of language, documents written specifically for the group may be useful here.  They should all require a similar form of simplification, otherwise you will be writing several simplification systems.  The group shouldn't be too narrowly defined (e.g. Deaf children in the age range 8-11 with a below average reading age), as this will make it difficult to find test subjects.  It also shouldn't be too broadly defined, otherwise different simplification needs may be present.

Once you have a group to simplify documents for, you're ready to consider the next step.

Source Material

You must decide what type of text to simplify.  It's easy to assume that text is text and you can just build a general model, but in fact different genres have their own peculiarities and jargon.  Consider the difference between a set of news articles, Wikipedia entries and social media posts.  Each will be significantly different in composition than the last.  Of course, the text genre should be one which the target audience wants you to simplify!  That's why this step comes after selecting the target group.  It's also important at this point to check that nobody else has tried to simplify the type of documents that you're working with.

Mode of Presentation

There are, roughly speaking, three ways of presenting the simplifications to the end user.  Which one you choose depends upon factors such as the text itself, the requirements of the user and the reason behind the simplification. Each has advantages and disadvantages as outlined below:

Fully Automated

Substitutions are applied directly to the text where possible.  The user never sees the simplifications being made and so does not need to know that they are reading a simplified document.

Advantages:

• User is presented with a simple document
• Requires minimum work from author / user
• Can be performed on the fly - e.g. whilst browsing the web / reading e-books / etc.

Disadvantages:

• Errors in the simplification process cannot be recovered
• Simplification may alter the meaning away from what the original author intended
• Some words may not be simplified - leaving difficult terms to be seen by the user

Author Aid

The simplifications are presented to the author of the document, who chooses when and how to apply the simplifications.  In this case, the simplification acts similarly to a spell checker.

Advantages:

• Author can make simplifications which retain their original intention
• No chance of grammar mistakes - as the author has the final say of which words to use

Disadvantages:

• Work must be simplified before being published
• No guarantee that the author will apply the suggested simplifications

On Demand

The user has the option to look up simplifications if they find a word too difficult.  These simplifications are typically presented somewhere nearby on the screen.  For example, if a word is clicked on, the simplification may appear in a pop up box or in a dedicated area outside of the body of text.

Advantages:

• User gets to see the original text
• Helps language learning / recovery as the user can decide when they require simplification

Disadvantages:

•  User may struggle if a text or portion of a text has a high density of difficult words
• The user may be distracted by the simplifications, which divert their attention away from the text

LREC 2014 - Post Blog

Looking back on LREC, I think I went with a real apprehension. My only previous conference experience was ACL 2013 in Sofia.  I had a great time and so my feelings towards LREC were a mixture of excitement for it being similar, but also a worry that it might not be as good.  I'm glad to report that it lived up to and even exceeded my expectations.  The sessions were interesting;  the posters engaging;  the networking events were even fun;  the attendees were approachable and people even engaged with my talk, asking questions and talking to me afterwards.

It was great to meet many of the people who I have previously referenced.  Many of those who wrote the papers in the lexical simplification list were in attendance.  I made it my business to approach them and introduce myself.  It's a great experience to meet people who have written such great papers and have made a contribution to the field.

For those that might find it useful, I've collected two lists below.  The first is a list of papers which had something to do with readability / simplification.  The second is a list of papers which I found interesting for one reason or another.  The simplification papers will be making their way to the lexical simplification list soon.

Improvements to Dependency Parsing Using Automatic Simplification of Data. T Jelínek PDF

Measuring Readability of Polish Texts: Baseline Experiments. B Broda, B Nitoń, W Gruszczyński and M Ogrodniczuk PDF

Can Numerical Expressions Be Simpler? Implementation and Demonstration of a Numerical Simplification System for Spanish. S Bautista and H Saggion PDF

Text Readability and Word Distribution in Japanese. Satoshi Sato PDF

And finally, my paper:
Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline. M Shardlow PDF

Some other papers that I found interesting were:

Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. M Sabou, K Bontcheva, L Derczynski and A Scharl PDF

Identification of Multiword Expressions in the brWaC. P Anick M Verhagen and J Pustejovsky PDF

Creating a Massively Parallel Bible Corpus. T Mayer and M Cysouw PDF

LREC 2014

I write this post from my hotel in Reykjavik.  Ready to get started at LREC tomorrow.  I'm looking forward to another conference experience.  I don't think this city could be more contrasting to my previous conference in Sofia, Bulgaria.  I'm also particularly excited as I am presenting in the main conference this year.  It is a really great opportunity to share my research and I'm looking forward to that.

I haven't completed my presentation yet.  But when I do, the slides will be up here. I am presenting on Thursday morning, so still time to put the finishing touches to my slides. I will be presenting on an error study of the lexical simplification pipeline which I produced a short while back.  It's an interesting little side experiment which developed into some useful results.  Effectively I've been able to show that the areas in the field which are least addressed are the areas which cause the most problems.  A game changer - if people take notice.  I'm really looking forward to seeing how the work is received.  I imagine there will be a mixture of views on the work.

In terms of my PhD project, this work will feed directly into the first chapter of my thesis.  It allows me to motivate work on mitigating errors throughout the lexical simplification pipeline.  I have recently come to what I think is a final draft of my thesis structure.  But I shall save that for another post.

Accepted Papers

It has been a good start to 2014 so far with 2 papers accepted. 

The first is a conference paper at a large conference called LREC 2014.  This will take place in Reykjavik, Iceland at the end of May this year.  The paper details some experiments that I performed last year, looking at the distribution of errors in the lexical simplification pipeline.  The hope is that if I can show the types of errors and whee they're coming from then I can motivate my research into mitigating these errors.  It will certainly make a nice introduction in my thesis.  The paper has been accepted for a main track oral presentation.  This means that I'll be presenting alongside established researchers in the main event of the conference.  It's certainly a mixture of very exciting and somewhat terrifying!

The second paper is published in a special issue of an open access journal (open access means that the publisher allows anybody to read it for free).  I was unsure at first of whether to publish with them but a number of factors led me to doing so.  The danger of publishing with open access journals, particularly the relatively unknown, is that the quality of the reviewing and the impact of the journal can be very low.  In the worst case you can end up paying thousands of pounds for somebody to essentially publish your paper on their blog.  The paper I have submitted is a survey paper.  I wrote it almost two years ago and have been updating it ever since.  It is effectively the background chapter in my thesis.  I originally attempted to submit it to a few prestigious journals, but it was rejected for a few different reasons.  Survey papers from relatively unknown authors are tough to get published.  So, off the back of rejections and updates it seemed sensible to submit to a less prestigious journal.  I saw the call for the special issue and thought I'd write up my paper for it. The organisation behind the journal is fairly new (2010), but they do seem to tick the right sorts of boxes.  They have an ISSN, they produce print versions of the journal, they don't charge an extortionate fee, the past issues seem well populated with sensible research, etc.  The survey paper is well suited to this type of publication as it will appeal to a broad audience and therefore will benefit from open access publishing.


Links (and maybe some more information) will be put up when I have the pdfs.

XKCD and simplification.

I have been an avid reader of the webcomic xkcd since my days as an undergrad.  If you've never heard of it, I would recommend you check it out, some of them are laugh-out-loud funny.  There are several comics that stick out as having a simplification theme.  I'm going to use this post to look at those comics through the lens of automatic simplification.  I'll try to explain what we can do with the current technology and what we just plain can't.

Simple

Particle accelerators are complex beasts.  I can empathise with the character who read so much simple Wikipedia that he can only talk in that way now.  One of the techniques we use in simplification is language modelling.  A mathematical model of sentences is trained which can be used to score new sentences to say how likely they are to have been produced by a language.  So for example:  "I went to the bank" might recieve a higher score than "I to the banking place did go".  As the latter sentence is poorly written.  An interesting factor of language models is that the scores they give rely heavily on the sentences which are used to train them.  So if you train a model using the text of the English Wikipedia you are likely to get very difficult to understand language.  If you train a model using the text of Simple Wikipedia, you are likely to get very simple sounding language, just like the second character in this comic.  A great paper which explains this further (without the xkcd references) is Kauchak (2013) (see the lexical simplification list).

Up Goer Five

This next one is too long to put in this post - but it's worth a read.  The full comic is here: Up Goer Five (Or, right click on the image and open it in a new tab to view).

The comic presents a simplified blueprint of the Saturn V rocket.  The translator has been restricted to only the thousand most common words in the English language.  There is some question as to where the statistics for the 'thousand most common words' came from.  If it were taken from NASA technical rocket manuals then there may have needed to be little change!  We'll assume that it was taken from some comprehensive resource.  The best way of determining this with currently available resources would be to use the top ranked words in the Google Web1T corpus (Google counted a trillion words and recorded how often each one occurred.)

The style of translation in this comic is phenomenally difficult to achieve, even for humans.  You can try it for yourself at The Up-Goer Five text editor.  Most words have been substituted for simpler phrases, or explanations.  Some technical terms rely on outside knowledge - which actually has the effect of making the sentence more difficult to understand.  For example, one label reads: "This is full of that stuff they burned in lights before houses had power".  This is referring to kerosene, which is highly understandable if you know of kerosene but inapproachable if not.

It would be an interesting experiment to determine the lowest number of words required to be able to produce this kind of simplification without having to draw on inferred knowledge (such as the type of fuel lights once burned).  My guess is that you would need 10 - 20,000 before this became a reality.  It would be difficult to automatically produce text at this level of simplicity.  Explaining a concept requires a really deep understanding and background knowledge, which is difficult to emulate with a machine.

Winter

The above comic touches on an excellent point.  If the words we use are understandable, does it matter if they're not the correct words? Previously, I have written about lexical complexity noting that many factors affect how difficult we find a word.  The big factor that is played on here is context.  For example the term 'handcoats' in the second panel is understandable (as gloves) because we know from the first panel that 'the sky is cold'.  Handcoats is a word that you've probably never seen before, and out of context it would be difficult to get the meaning.  This highlights the importance of selecting words which fit the context of a sentence. If the correct context is chosen and a simple word fitting that context is used, then the understandability of a sentence will be dramatically increased.

Word Sense Disambiguation

Some words have more than one meaning.  The brain seems to have an innate ability to work out what a sentence means.  Take the following two sentences:

"I tied my boat to the bank"

"I put my money in the bank"

In the first sentence the you probably imagine somebody tying their boat to the side of a river, yet in the second sentence you imagine somebody investing their money with a financial institution.  That string of four characters: 'b a n k' has completely changed meaning.

Word sense disambiguation (WSD) is a well researched task in computational linguistics with an important application to lexical simplification.  The majority of previous research splits roughly into three categories:

• Supervised: Using labelled data, a system builds a classifier which can recognise the different senses of a word, from a variety of features in the words surrounding it.
• Unsupervised: With unlabelled data, a system learns the different senses of a word.  Classification of new data makes use of the previously learned senses.
• Knowledge Based: A large knowledge resource such as WordNet provides information about the words which can be used during disambiguation.

 WSD is vital to the task of lexical simplification.  Consider simplifying a sentence from the previous example. If you look up the word 'bank' in a thesaurus you will have a list of synonyms that looks something like the following:

Bank:
Financial Institution; Treasury; Safe;
Edge; Beach; Riverside;

If a system does not employ WSD, then there is no method of telling which of the synonyms are correct for the context.  We do not wish to say "I tied my boat to the treasury", or "I put my money in the riverside".  These examples are at best farcical and at worst nonsensical.  WSD is paramount to selecting the correct set of synonyms.

I will not venture to a full explanation of WSD as applied to lexical simplification.  Suffice to say that there are four papers which I have so far identified as addressing the matter.  These can be found in the lexical simplification list.

• Can Spanish be simpler? LexSiS: Lexical simplification for Spanish. Bott et al. 2012
• Wordnet-based lexical simplification of a document. Thomas and Anderson 2012
• Putting it simply: a context-aware approach to lexical simplification. Biran et al. 2011
• Lexical simplification. De Belder et al. 2010

3rd Year

On Wednesday (4/9/2013) I successfully completed my end of second year interview.  This means that I am now officially a third year PhD student.  I am now at the dead halfway point of my PhD, having completed 24 months with 24 more remaining.  It has been a long road getting here and there is still a long way to go.  Below is a brief analysis of the achievements in my PhD so far and the goals yet to come.

Completed So Far:

• Literature Review:  This was the first thing I did as a PhD student.  Reading took up most of the first six months of my research.  I consumed, refined and categorised as much of the relevant literature as I could find.  I am attempting to publish this as a survey paper currently, since the only available text simplification survey is a technical report from 2008
• Lexical Simplification Errors: I recently undertook a pilot study looking at the errors thrown up by the lexical simplification pipeline.  I'm looking to publish this in an upcoming conference, so won't say too much about the results here and now.
• Complex Word Identification: This was the first element of the lexical simplification pipeline that I studied.  I built a corpus of sentences, each with one word marked as complex for the purpose of evaluating current methods of identification.  This work was published in 2 separate workshop papers at ACL 2013.
• Substitution Generation: Once we have identified a complex word, we must generate a set of substitutions for it.  However, those words which are complex are also those which are least likely to be found in a thesaurus, complicating the task.  To address this I spent considerable efforts learning simplifications from massive corpora with some success.  This work is also currently being written up for publication.

Still to come:

• Word Sense Disambiguation: The next step in the pipeline is to apply some word sense disambiguation.  This has been done before, so I will be looking at the best ways to apply it and hopefully making a novel contribution here.  I am just starting out on this phase of research and am currently immersed in the WSD literature, trying to get my head round the myriad techniques that already exist there.
• Synonym Ranking: I have looked into the best way to rank synonyms according to their complexity before at the start of my project.  The small amounts of work that I did back then did not discover anything radical, but did help me to better understand the structure of a lexical simplification system.  When I revisit this area it will be with the hope of making some significant contribution.  I was really interested in the work David Kauchak presented at ACL 2013 and will be interested to explore what more can be done in this area.
• User Evaluation: Finally, I will spend some time exploring the effects of each of the modules I have developed on individual users.  It is of paramount importance to evaluate text simplification in the context of the users it is aimed at and to this end I will be focussing my research on a specific user group.  Although which group is as yet undecided.
• Thesis: This will undoubtedly take a significant portion of my final year.  The chapter titles will hopefully be the bullet points you see listed above.

So there you have it.  Although it appears that I have done a lot so far, it still feels like I have a real mountain to climb.  There are significant hurdles and vast amounts of reading, researching and writing ahead.  I look forward to the challenges that the next two years of my PhD will bring.