Tutorial: Machine learning of Iris data using Pybrain neural network

Neural networks are very interesting and worth exploring to solve machine learning problems. It is fortunately incredibly easy to model a neural network. Today we will deploy create a neural network to solve a classification problem using Iris data. Iris is a popular dataset to test and model various machine learning algorithms. It contains 3 sets of flower types (target variable) and 4 dimensions feature values.

What is needed?

Python, Numpy, Matplotlib, Scipy, Scikit-learn, Pybrain

Goal

We will aim to get the classification accuracy of our neural network atleast 95% or more.

Analyze Iris Dataset

Before we delve into the machine learning, lets quickly analyze the data provided so that we can understand more about the structure of data. Lets pull the iris input and output, using the scikit-learn library.

from sklearn import datasets iris = datasets.load_iris() X, y = iris.data, iris.target

Here X (capital) contains 4d array of input data ()  and y(small) contains target class (1d).

Attribute Information:

1. sepal length in cm

2. sepal width in cm

3. petal length in cm

4. petal width in cm

Target class:

1 — Iris Setosa

2 — Iris Versicolour

3 — Iris Virginica

From below graph, we plot and clearly see the 3 classes.

Getting Started

Lets get started with machine learning using PyBrain.  We will design a 4 input, 3 output, 3 hidden layer, feed forward neural network with backpropagation trainer. We will use SoftMaxLayer function on output neurons., Sigmoidal on hidden neurons.

FNN with Backpropagation

Why 4 inputs and 3 hidden layers, 3 output?

We feed 4 dimensional input array to 4 input neuron (1 input to each neuron). For the start we compute 3 hidden neurons using a rule of thumb Total (input neurons + output neurons / 2)  which is 3 but later we will play around. Finally because we use SofmaxLayer activation function (0,1) we have to have a separate output neuron per class output.

First we will load all the pybrain modules

>>> from pybrain.utilities import percentError >>> from pybrain.tools.shortcuts import buildNetwork >>> from pybrain.supervised.trainers import BackpropTrainer >>> from pybrain.structure.modules import SoftmaxLayer >>> import numpy as np >>> import matplotlib.pyplot as pl

then we will setup the data for the neural network. Here we specify the dimensions of the input and target data which is 4,1 and the number of classes as 3.

ds = ClassificationDataSet(4, 1, nb_classes=3) for i in range(len(X)): ds.addSample(X[i],y[i])

Now check the dataset ds['input']

>>> ds['input'] array([[ 5.1, 3.5, 1.4, 0.2], [ 4.9, 3. , 1.4, 0.2], [ 4.7, 3.2, 1.3, 0.2], [ 4.6, 3.1, 1.5, 0.2], [ 5. , 3.6, 1.4, 0.2], ....  then>>> ds['target'] array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], ...

Lets check the length of the input and target in the ds.

>>> len(ds), len(ds['input']), len(ds['target']) (150, 150, 150)

Looks like the data we fed into neural network is intact!!! Thats good!!!

To move on further, lets split the dataset t0 Training data, Testing data and Validation data in the ratio 60/20/20. Pybrain has a builtin function to split the dataset.

>>> trndata,partdata = ds.splitWithProportion (0.60) >>> len(partdata) 60 >>> len(trndata) 90 >>> tstdata,validata = partdata.splitWithProportion(0.50) >>> len(tstdata) 30 >>> len(validata) 30

As you can see about, out of the total 150, training data is 90 rows, test and validation has 30. To test the final performance of the neural network, we run the validation data only once.

For neural network classification, it is highly advisable to encode classes with one output neuron per class. This is because of the we use the Softmaxlayer activation function (outputs 0/1) in output neuron which requires one neuron per output class. Let do that

>>> trndata._convertToOneOfMany() >>> tstdata._convertToOneOfMany() >>> validata._convertToOneOfMany()

After this if you see the target values, it is encoded like this which are suitable for outputs produced by 3 neurons..

>>> trndata['target'] array([[1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0],

The original 1d unencoded target values are stored in ‘class’ created by the function to preserve the values.

>>> trndata['class'] array([[ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.],  Lets double check the dimensions again, to make sure everything is ok, before we model the neural network.>>> trndata.indim, trndata.outdim, tstdata.indim, tstdata.outdim (4, 3, 4, 3)

All is good with the data and lets move on to setting up the neural network.

Creating a Neural Network

Lets setup 3 layer FNN neural network with 4 inputs, 3 hidden neurons, and 3 outputs. We do this using the pybrain code.

>>> net = buildNetwork(4,3,3, outclass=SoftmaxLayer)

and we setup the trainer for backpropagation algorithm

>>> trainer = BackpropTrainer(net,dataset=trndata,momentum=0.1,verbose=True,weightdecay=0.01)

Lets train the neural network for just 50 epochs (cycles) and we will plot the training error.

>>> trnerr,valerr = trainer.trainUntilConvergence(dataset=trndata,maxEpochs=50) >>> pl.plot(trnerr,'b',valerr,'r')

In this plot, we clearly see the training error going down to converge to the minimum. Thats good, we are moving in the right direction then

We train the model with the training data for about 500 iterations.

>>> t.trainOnDataset(trndata,500) Total error: 0.144073198342 Total error: 0.125821975525 Total error: 0.116370069949 Total error: 0.112602692528 Total error: 0.111153167746 Total error: 0.110051076529 Total error: 0.10958277085 Total error: 0.108547901802 Total error: 0.107476439267 .................  To check how many times the trainer was trained...>>> trainer.totalepochs 101

We never trained the model with the test data and the data is unseen by the model. Lets see how it goes..Lets evaluate on the test data and compare the prediction. we will compute on the percentage error in training on datasets (train and test).

We evaluate the error rate on test data

>>> out = n.activateOnDataset(tstdata).argmax(axis=1) >>> percentError( out, tstdata['class'] ) 3.3333333333333335

This means that we have 96.77% score on test data. ! At this point we have no way of knowing whether the model is overfitting or underfitting or just memorizing.

At this point, the model only learnt the training data, tested on the test datset.

How to Predict and Score?

To predict how well our model works when seeing new dataset, rather than memorising, we run our neural network model to predict on the final validation set.

Here is how to make the model do prediction. This is a raw 3 neuron output.

np.array([net.activate(x) for x, _ in tstdata])

or this way

>>> out = net.activateOnDataset(tstdata) >>> out array([[ 0.03930682, 0.58523366, 0.37545952], [ 0.03912971, 0.58569724, 0.37517305], [ 0.03922704, 0.58756942, 0.37320354], [ 0.04229602, 0.57161333, 0.38609065], [ 0.04197473, 0.573551 , 0.38447427], [ 0.04107508, 0.57824981, 0.38067511], ...  then we will shape the neuron output to output class, so that highest output activation gives the class [0,1,2].>>> out = out.argmax(axis=1) >>> out array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2])

We have one final piece of data to test, which is validation data. We run only once on the validation set. This way our model learns how to generalize and not to memorize.

>>> output = np.array([net.activate(x) for x, _ in validata]) >>> output = output.argmax(axis=1) >>> output array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) >>> percentError( output, validata['class'] ) 3.3333333333333335

We have 96.7777% score on the validation data. Nice!!!

Lets try to push that score up.

We will again remodel the neural network with 5 hidden neurons instead of 3 and see how it goes!!!

>>> fnn = buildNetwork(4,5,3, outclass=SoftmaxLayer) >>> tr = BackpropTrainer(n,dataset=trndata,momentum=0.1,verbose=True,weightdecay=0.01)

and we train the Backpropagation trainer for 1000 cycles on the training dataset.

>>> tr.trainOnDataset(trndata,1000)

The model is given unseen test data and validation data and we will compute the score..

>>> out = fnn.activateOnDataset(tstdata).argmax(axis=1) >>> percentError( out, tstdata['class'] ) 0.0 >>> out = fnn.activateOnDataset(validata).argmax(axis=1) >>> percentError( out, validata['class'] ) 3.3333333333333335

Wow! 100% for test datset and 96.77% on validation dataset (overfitting). To make a robust model, play around with various configurations and see which one is the best.

Congratulations for modeling your first neural network!!!!!