Generating Structure of a Learning Model

BSc final project. Implemented in Java at the turn of the millenium :-)

This project implemented a framework to "breed" modular neural networks using a genetic algorithm.

Neural networks are modelled on the brain and consist of a number of connected nodes, often organised in layers. Each node can be given inputs and produce an output. Some of the nodes - known as input nodes - are given inputs for a particular task, their outputs filter through the network and produce a solution to the task. Neural networks are trained on a set of training data, often consisting of input-output pairs (i.e. inputs into the network and the desired outputs of the network). Most network architectures learn correlations present in the training data and generalise to better interpret new data - i.e. they learn from experience.

An example of the use of neural networks is for image classification, where a network can be trained on sets of images and their associated class (e.g male/female faces). After training the network should be able to classify new 'unseen' images.

Modular neural networks are networks that consist of a number of modules, each of which is normally a learning model (such as a neural network). These networks are powerful as they can take the inputs and assign the most suitable modules to the task at hand. This allows them to perform different types of task at the same time. It is widely believed that the human brain has a modular structure.

Genetic algorithms model some kind of evolutionary process. You generally start with a population of individuals. Each individual is evaluated in some way, the fittest (i.e. best) are selected and their properties combined. This produces new individuals that make up a new population. To keep the population diverse a number of these individuals are mutated (e.g. in the case of neural networks connections may be removed). The process repeats on the new populations until some stopping criteria is met (e.g when an individual performs well enough).

My Program

The program is called NetSys. It starts by generating a random population of modular neural networks, whose modules are multi-layered perceptrons (a widely used type of neural network). Essentially it works by 'breeding' the modules. First, each modular network in the population is trained. After training, sub-tasks within the overall task are identified and used to select the best modules from the various modular networks. This gives a population of normal, non-modular networks that are 'bred' to create better modules (note that a module is itself a neural network). The best of these are selected to construct a new population of modular networks. This process continues - modular networks are trained, the modules selected, bred, and new modular networks produced - until a modular network has been produced that has sufficient performance.

It was hypothesised that the evolution of modules using a basic network construction method would be more successfull than the evolution of a single large network. The result is a flexible modular neural network implementation combined with a genetic algorithm that can successfully automate network construction. Due to the time limitation, it was not possible to fully explore the evolution of networks, but now that the framework has been defined in an object-oriented language, better algorithms can be quickly and easily defined and used in the future.

I wrote a lot of Java code (I think over 19000, including comments - bloated in retrospect) split into 5 packages that provide classes for the neural network, genetic algorithm, results monitoring, interface and testing. The report was "highly regarded" by the markers, and the project as a whole was graded 1st class.



  • Update 06/12/2011: Converted the doc into PDF.

Last modified: 30/04/2006 (most likely earlier as a site migration in 2006 reset some dates) Tags: (none)

This website is a personal resource. Nothing here is guaranteed correct or complete, so use at your own risk and try not to delete the Internet. -Stephan

Site Info

Privacy policy

Go to top