Jim Susinno 2002 Machine Learning Final Project - Face Detector using Neural Nets Requires directories full of .pgm portable greymap images. Optionally can include another directory of negative training examples. Tags are used like the -L and -I tags to gcc. -nr is used to pass in a number of negative random training examples. -nb is used to pass in a number of negative black training examples. <> means no space between args, e.g. "-P../data" [] means theres a space, i.e. "-layers 0" usage: ./glfacerec -P -N -T -t -nb -nr -layers [int] [int] [int]... the -layers tag consumes the args after it - how many is specified by the first int arg, #of hidden layers. default is 1. The next n in [0,4] integers indicate the sizes of the layers in the network. For instance, for a 2 hidden layer network with 20 nodes in the first layer and 5 in the second, we would add: -layers 2 20 5 to the command line. you may specify up to 4 hidden layers. want more, hack the source. some idioms: ./glfacerec -P../orl_faces/s1 -N../negatives/ -nr5 ./glfacerec -P../chevrons -nr5 ./glfacerec -P../data/positive -N../data/negative -T../data/test ./glfacerec -P../newdata/positive -N../newdata/negative -T../newdata/test_positive -t../newdata/test_negative ./glfacerec -P../newdata/positive -N../newdata/negative -T../newdata/test_positive -t../newdata/test_negative -layers 2 20 5 A note on matrix representation: --------------------------------- The dimensions of the matrices should be the product of the 2 layers of nodes on either side of the weights....In other words, a 1 layer network with 12 inputs and 2 outputs will have a matrix of dimension 12*2==24. A 2-layer network with 12 input node, 6 hidden and 2 output nodes will have 2 matrices of dimension 12*6 and 6*2. We add one to the input dimension for bias. The input to these weights can be 1.0 or 0.0. In arrays, major representation is to the left, closest to the array name. matrix[layer][node]; layer_dimensions is 1 bigger than the number of layers of the network, because using this nomenclature, a perceptron is a 1-layer network. It has 2 separate dimensions, input and output. ToDo: ------------- eventually a node class might prove very useful: itll have an input(activation), output, activation function( sigmoid) temperature(for boltzmann machines) possibility to link back to other nodes???? PGM class needs to handle comments better, also needs support for more high level image ops such as resize(using interpolation!), etc. how could we have just one copy of screenPrintString?? add square images for training gl in-app command for adding random and square images at run time A graphical issue: in stochastic mode, the yellow node values only reflect teh first(last?) training example.... the batch mode is also borked - it wont do the last training example.... init INPUT node weights to zero for vis save some example networks - load them later also mind the return code for ftime - use a wrapper? negative example count - put it in imagenetwork. - also rename to imageclassifier use 0.1 and 0.9 as high and low values. hopfield network ANNEALING and stop condition annealing of learning rate based on error history calculate error every n iterations(for speed) log scale on the graphs Changelog: ------------ Sat Dec 7 19:57:47 EST 2002 v0.2.8: Parameterized matrix display - some cool new 3d matrix display modes. display presets in 123...0...new mouse control. Added stop and re-randomize conditions to glfacerec.cpp - seem to work! Fri Dec 6 20:51:29 EST 2002 v0.2.7: Squashed some juicy bugs in resizing. attempting to find the cause of blowups - huidden layer size 20 does it. hacked a bias loop in test error...... now its always 0? Tracked it down to the matrix level, copying old values to new space. v0.2.6-1 Added raster dimension indicator. Thu Dec 5 04:07:18 EST 2002 v0.2.6: Added negative count of training examples btw init constriuctors for neuralnet has vestigial matrix(1,1) Looks like i hacked in bias. Added 1 to everty layer's input size. also to iopair. Hacked in a bias on/off switch in neuralnet - simply turned the extra input node on or off(1.0,0.0) very attractive examples in glnettest.. Added DIM command via imnet->setLayerDimensions. Very handy for tooling... Added -layers tag to cmd line - consumes extra args! Wed Dec 4 16:11:46 EST 2002 v0.2.5: Switched out ftw for readdir and scandir - put into class working on adding bias.... should only have to increment the dimensions of all input matrices and lightly hack the examples.... Added cursory count of negative training and test examples... to neuralnet... Wed Dec 4 04:18:22 EST 2002 v0.2.4: More negative examples - use the convert command on raw data from maneesh Added run timing for glnettest. Converted Maneesh's dataset to pgm using a perl script and 'convert'. Looks good!! ./glfacerec -P../data/positive -N../data/negative Straightened out the test set... now shows test set error as red line. Added info to 2d display: error, example count. Flipped rotation axes(middle button). Fixed d_sigmoid - > sigmoid * (1 - sigmoid) Added seed_scale to scale random values of matrix seeding. Tue Dec 3 22:44:12 EST 2002 v0.2.3: [jim@pikkon final_project]$ ./glfacerec -P.. - I see a face!!!!!!!! add cmd line tag for dimension - NOT WORKING YET FIXED MEMORY LEAK!!! found 1 in neuralnet_learning.cpp line 100... another in line 178... still not there... oo a bunch in gradientDescent line 205 got one line 143 - vestigial - its tight! Added run time indicator. Finished save net function... load is gonna take a readLine function. Ported 0.2.2 to windows - glnettest runs, but no ftw so no facerec. - oh btw, the win version behaves funky on learning - differently.... values collapse to begative real fast... to work on windows, load image filenames from file(windows has no ftw) Added depth of point display mode and sum squares(all green for now). Whipped up a perl script for converting images... almost crashed out pikkon with forks. backing up before i screw something up. Tue Dec 3 02:11:07 EST 2002 v0.2.2: Removed the updateWeightsMomentum() function and added a float coeff parameter to updateWeights(float). Added the d_sigmoid function - derivative of sigmoid. Added neuralnet::save/load functions(stubs). Added command prompt to glnettest. Added WEIGHT int int int float command to manually manipulate 1 given weight of the network. Changed updateWeights to take int and work for eta and alpha. Batch momentum now more proper. Tue Dec 3 00:33:50 EST 2002 v0.2.1: Split neuralnet.cpp further into neuralnet_learning.cpp Added test set to neuralnet class. Implemented a state flag to change between stochastic and batch modes. Neuralnet flow is dependent on this variable... weights are either updated at the end of a full loop through the training set or one by one after each training example. New learning can be observed on some of the example nets... stochastic gradient descent now gives a very shaky error history line. Straightened out error calculation(training and test) - now sum errors over whole training set. Adding test set functionality... we need negative test set too? -T tag for positive test images... for negatives?? -t<>?? Added another array of error values for test set error. Added command buffer to glfacerec for more manual control. Added -nb# number of black negative images(like random) Mon Dec 2 06:00:25 EST 2002 v0.2.0: Removed test code from beginning of glfacerec::main. Synced up glnettest and glfacerec a little. Added a jitter function - shake the matrices up a little bit. Hit the J key. Split neuralnet.cpp into neuralnet_opengl.cpp using #include. Added momentum to the Neuralnet class. Seems to help gradient descent in most cases. Cleaned up most code. Mon Dec 2 03:00:03 EST 2002 v0.1.8: Adding rescale size members to imagenetwork. Networks will probably need momentum and maybe bias to learn more properly. Added ramdom instances to the end of the face recognizer net. Added a small offset for the output node in point display of weights. Added argv[3] number of random negative examples to add to the network. Added draw mode 2 sum mode for visualization of network. Added better arg parsing: ./glfacerec -P -N -nr5 Seems to be learning basic chevron patterns(based on observation of weights) Tweaked glnettest a little, plus its a new makefile target. Sun Dec 1 19:10:19 EST 2002 v0.1.7: added dump, next, nameoffile to PGM class. added imagenetwork class moved boltzmann temp. sigmoid graph to neuralnet class Paradigm in glfacerec: 1 network - save on null checking. need to override learning methods in imagenetwork class...... gonna add a new neuralnet display method for 3d matrices.... Sun Dec 1 20:32:47 EST 2002 saving for now. Sun Dec 1 03:02:05 EST 2002 v0.1.6: added display toggle of interface components(emnM) to speed learning tweaking error graph - trying relative scaling vs. absolute and possibly logarithmic scales. adding a class for pgm learning network - imagenetwork.h now that algo seems to be correct, code is ripe for a cleanup.... Sat Nov 30 03:11:09 EST 2002 changed glfacerec to .cpp to attempt to avoid a strange segfault when adding training example to a net in readFromfile. not working..... this only popped up when i put an if(printout) in front of the printfs in backprop... fixed lack of increment bug in training count... Tue Nov 26 00:07:38 EST 2002 ok tweaked things out and probably implemented stochastic backprop correctly. nets are much more stable. added temperature(steepness) to squashing function. net 7 key segfaults immediately now?? Sat Nov 23 01:44:51 EST 2002 not sure whats wrong with the xor learner net, but im getting segfaults for just about any neural net method. and weird ones too..... Fri Nov 22 01:18:00 EST 2002 got back a semblance of a working matrix class. setNodesInLayer works. network display works. Thu Nov 21 20:37:56 EST 2002 These classes should greatly facilitate matrix calculation: matrix.h matrix.cpp matrix class drawing in 2 and 3d is straight. Thu Nov 21 18:25:09 EST 2002 v0.1.5: talked to Bodhi about backprop - get it a lot better now. some reimplementation might be in order, straighten this thing out. getting weird faults at the moment.... also gotta remove the hacky perceptron case. Wed Nov 20 23:59:02 EST 2002 something just occurred to me..... we should perform the calculations for output in stack space... copy the node values in when we need to see them.... Wed Nov 20 21:14:48 EST 2002 tried shifting around some allocation i was crashing on... had the effect of screwing everything up. so made a crash copy and fell back to 0.0.8. adding node_value display to debug network Tue Nov 26 00:12:34 EST 2002 v0.1.4: Learning now a lot more stable. Added Neuralnet::{steepness, learning rate} Perceptron will converge, but not multi-layer nets... added z,x key control of learing rate. Sat Nov 23 17:06:27 EST 2002 v0.1.3: Perceptron case now works again. fixed some bugs. tweaked coloring code for raster and 3d weight matrix values. Added the deltaW matrix in Neuralnet::gradientDescent() for holding weight changes. Sat Nov 23 01:44:51 EST 2002 v0.1.2: cleaned up neural net class. added graphical yellow display of activation values. Having segfault issues with the xor net... Fri Nov 22 01:18:36 EST 2002 v0.1.1: reimplementation with matrix class - display works. setNodesInLayer works. Fri Nov 22 00:09:16 EST 2002 v0.1.0: added class Matrix and straightened out allocation, multiplication and drawing in 2 and 3d. v0.9.0: spent more time making this thing crash more Wed Nov 20 03:29:31 EST 2002 v0.0.8: added the perceptron case in backprop. putting what i have up on the net. Sun Nov 17 00:43:21 EST 2002 v0.0.7: perceptron case now matches that of glneural animal{2,4}.net Sat Nov 16 21:56:17 EST 2002 v0.0.6: matched performance of perceptron example by adding hacked perceptron cases. Tue Nov 12 21:10:24 EST 2002 v0.0.5: added class IOpair for holding training examples in linked list format. IOpair holds 2 pointers to float arrays, 2 integer counts of elements, and a pointer to next. Neuralnet::addTrainingExample(float*,float*) adds an IO pair into a net's list. i wanna change getInputNodes() to Inputs(); added target to iopair added openGL display functions(nerualnet.h now includes glut.h) drawMatrix() and drawRasterMatrix(). Raster will take 4 ints for spread and x,y parameters. Fri Nov 1 18:30:40 EST 2002 v0.0.4: neural net evaluation(forward propagation) seems to work correctly. added rename(), getInputNodes(). switched to tab format. Fri Nov 1 12:06:57 EST 2002 v0.0.35: implemented 1-layer perceptron case for CalculateOutput. Still dirty; watch out. Sun Oct 27 13:31:21 EST 2002 v0.0.3: Added class PGM. Has one constructor which takes a char* filename. Added resize(int,int) function. Sun Oct 27 01:14:14 EDT 2002 v0.0.2: Seem to have set up the matrix dimension allocation properly. Tested on 2 example nets, seeded randomly. They both seem to print out fine using dump(). Sat Oct 26 19:35:13 EDT 2002 v0.0.1: the neural net class added constructors, allocators, dump() and randomWeight() Sat Oct 26 19:04:07 EDT 2002 v0.0.0: pgm loading code hacked in manually(boo to libpgm link errors...)