An Excellent Experimental Framework.

Recently I have been watching through a lecture series on Deep Learning for NLP.  It's a topic that I have long been interested in, and I'm learning lots as I go along.  It has been a welcome relief from thesis writing.

In the first 15 minutes of the fifth lecture, Richard Socher outlines the steps that his students should take for their class projects. I wanted to reproduce them here as I think they are really useful for anybody who is performing experiments in Machine Learning with Natural Language Processing. Although the steps are aimed at a class project in a university course, I think they are applicable to anybody starting out in NLP, and helpful as a reminder to all who are established.  I had to figure these steps out myself as I went along, so it is very encouraging to see them being taught to students.  The eight steps (with my notes) are as follows. Or, if you'd prefer, scroll to the bottom of the page and watch the first fifteen minutes of the video there.

Step 1 - Define Task

Before you make a start you need to know what your task is going to be.  Read the literature. Read the literature's background literature. Email researchers who you find interesting and ask about their work. Skype them if they'll let you.  Make sure you know clearly what your task is.

Step 2 - Define Dataset

There are a lot of ready made datasets that you can go and grab.  You'll already know what these are if you've read the literature suitably.  If there is no suitable dataset for your task, then you will need to build one.  This is a whole other area and can lead to an entire paper in itself.

Step 3 - Define Your Metric

How are you going to evaluate your results?  Is it the right way? Often your dataset will naturally lead you to one evaluation metric.  If not, look at what other people are using. Try to understand a metric before using it.  This will speed up your analysis later.

Step 4 - Split Your Dataset

Training, Validation, Testing. Make sure you have these three partitions.  If you're feeling really confident, don't look at the testing data until you absolutely have to, then run your algorithm as few times as possible. Once should be enough.  The training set is the meat and bones of your algorithm's learning and the validation set is for fiddling with parameters.  But be careful, the more you fiddle with parameters, the more likely you become to over fit to your validation set.

Step 5 - Establish a Baseline

What would be a sensible baseline?  Random decisions? Common sense judgements?  Maximum class? A competitor's algorithm?  Think carefully about this.  How will your model be different from the baseline?

Step 6 - Implement an Existing (Neural Net) Model

Neural Net is in brackets because it is specific to the course in the video.  Go find an interesting ML model and apply it.  I have started to use WEKA for all my ML experiments and I find it really efficient.

Step 7 - Always Be Close to Your Data

Go look at your data.  If you're tagging something look at what's been tagged.  If you're generating something read the resultant text.  Analyse the data. Make metadata. Analyse the metadata.  Repeat. Where do things go wrong?  If you have a pipeline, are some modules better than others? Are errors propagating through?

Step 8 - Try Different Models

Play around a bit.  Maybe another model will perform much better or worse than the first one you try.  If you analyse the differences in performance, you might start to realise something interesting about your task.  I would recommend setting up a framework for your experiments, so as you can quickly change things around and re-run with different models / parameters.

Step 9 - Do Something New

 OK, this isn't a step in the video per se. But, it is the extension of the methodology.  Now that you have tried something, you can see where it was good and where it was bad.  Now is the time to figure out how to make it better.  Different models, different features, different ways of using features, are all good places to start.


The video is below and also accessible at this link:
https://www.youtube.com/watch?v=I2TfdXfSOfc

If you are interested in the topic then the course syllabus with all videos, and lecture notes is here:
http://cs224d.stanford.edu/syllabus.html

Randomising lines a very large file with java

I came across an interesting problem today.  I have some nice results from counting data and I wanted to see if the same results would appear if I randomised the underlying dataset.  The problem?  The dataset is a 17 Gigabyte file.

All the solutions I could find online required the file to be read into memory at some point.  Obviously, with my 8GB of RAM these were not acceptable solutions.  I needed a solution which would allow one line to be in memory at once and then to be discarded.

I reasoned that if I wrote the lines of the file into separate files, I could create some randomisation.  I also realised that the more files there were, the greater the randomisation.

Below is the java code I wrote, as well as a bash wrapper script. It takes a file and a numeric argument denoting how many files to write into.  It then  assigns each line at random to one of the files until it runs out of lines.  These files can then be concatenated together in  a post-processing step.  I think it's quite a neat solution.  I've commented the code for readability, so hopefully it will be reusable.  Of course this is not true randomisation as some ordering is preserved, however it should work for most purposes.

For my 17GB file it took 22 minutes to run, writing to 1000 files.  Needless to say that most of that time was taken up by I/O.

import java.io.PrintWriter;
import java.io.FileWriter;
import java.io.FileReader;
import java.io.BufferedReader;
import java.util.Random;

public class RandomiseLines
{
  public static void main(String [] args) throws Exception
  {
    if(args.length != 2)
    {
      System.out.println("Usage: java RandomiseLines <file> <No of Output Files>");
      System.exit(-1);
    }

    //the number of separate files to place lines into.
    final int FILENUM = Integer.parseInt(args[1]);

    //initialise the random number generator.
    final long SEED = 1;
    Random generator = new Random(SEED);

    //if seed isn't required, comment above and use:
    /*
      Random generator = new Random;
    */

    //initialise the file writers
    PrintWriter [] writers = new PrintWriter[FILENUM];
    for (int i = 0; i < FILENUM; i++)
     writers[i] = new PrintWriter(new FileWriter("out." + i + ".txt"));

    //read in the file
    int key;
    String line;
    BufferedReader in = new BufferedReader(new FileReader(args[0]));
    while((line = in.readLine()) != null)
    {

      //generate a random number between 0 and FILENUM - 1
      key = (int)Math.floor(FILENUM*generator.nextDouble());

      //write the line to the chosen file;
      writers[key].println(line);
    }//while

    //close IO
    in.close();
    for(int i = 0; i < FILENUM; i++)
     writers[i].close();
   
  }//main
}//class

The following shell script can be used as a wrapper to the programme.

#!/bin/bash

FileNum=10;

java RandomiseLines $1 $FileNum

echo "" > randomised.txt

for i in `seq 0 $(($FileNum -1))`; do
 cat out.$i.txt >> randomised.txt;
 rm out.$i.txt
done

exit