5. CONCLUSIONS

This report has followed the development a system to generate the structure of a modular network.

5.1 Achievements

The project has achieved a number of goals. The implementations of the sub-systems are correct, and largely successful. The integration into a complete system has produced good results, although it does not exhibit all desired behaviour.

The classes that define the systems can be reused in other Java programs. They are easy to set up and use. Above all, they are very flexible. Every system has a method to alter nearly every one of its properties. See appendix A for a full listing of the methods.

A framework exists to allow different algorithms to be implemented in the future without needing any alteration to the existing code.

5.1.1 Sub-systems

In order to implement the overall modular breeder system, a number of other systems were developed. These systems make up the bulk of the project. They achieved different degrees of success. They are as follows,

1. The implementation of a neural network and a training algorithm was successful. The training algorithm, Backpropagation, has been shown to be capable of training a neural network. A number of methods have been written that can successfully manipulate the structure of the network.

2. The implementation of a modular network and the main training algorithm, Winner-takes-all, was successful. The algorithm has been shown to produce training results that are superior to that of the neural networks tested.

3. The implementation of a genetic algorithm has been shown to work correctly, although not effectively. The crossover and mutation algorithms provided are the likely cause.

5.1.2 Modular network breeder

The main aim of this project was to produce a system capable of generating the structure of a modular network. Preliminary tests indicate that this aim has been achieved. The modular network breeder returned a suitable network on all of these tests, which exhibited better performance than all the other networks tested. There is no user design of the structure of the network, only decisions as to what size limitations to impose on the initial population.

However, while the system has been shown to produce the correct structure, it does not make effective use of the genetic algorithm. The genetic algorithm simply acts as a further training and selection procedure, without making beneficial structural changes.

The networks structures generated proved very different from the predicted structures. This illustrates a point made in section 1.2 – the structure generated served to give a greater understanding of the problem.

5.1.3 Interface

A simple command-line interface was developed, but proved too limited for the testing of the sub-systems. To provide more control over parameters, a graphical interface was written. This implementation greatly improves the ease at which the systems can be tested. The addition of new method buttons can be done with ease, if there is a desire to add greater control.

5.2 Criticisms

While I wanted experience in writing all of the sub-systems, it would have been beneficial for me to make use of an existing neural network program. I felt it was necessary to implement the neural network to provide the desired structural operations. However, I underestimated the complexity of such a task and in retrospect it would have been better to compromise by using an existing program. This would have allowed me to spend more time on the implementation of the genetic algorithm. A large amount of evolutionary techniques remain unexplored. I believe a more effective system would have been produced if this had been possible.

The performance testing of the systems is incomplete. It serves only as an initial indication of performance. The neural network and modular network were shown to work, but optimal parameters remain to be found. A single run of the genetic algorithm can take over 24 hours. The modular breeder was also slow, due to its use of the genetic algorithm. This meant a great deal of time was needed for correctness testing, leaving less time to test performance.

Fast system running times were not stated as a requirement. However, a faster running time is desirable, if only to permit quicker testing.

5.3 Future Work

5.3.1 Improvements

It would have been useful for the network training algorithms to use a validation set to determine when training should stop. This will lead to better generalisation performance. Also, there is no control of when the training should stop, other than the number of epochs. It would be useful to allow the training to stop when a desired error has been reached.

The gating network used by the modular network should be allowed to have different inputs than the ones to the modular network itself. This would produce greater flexibility, allowing the gating of the experts to be dependent on different inputs than those defining the task.

The interface is interrupted by any running processes, and is not refreshed until the process has finished. This is inconvenient when the parameters that were set need to be checked. This could be overcome by making every method called from the interface run in its own thread. This would also allow the method to be interrupted, or terminated when desired.

5.3.2 Testing

The correctness testing of the systems served only to verify that their non-public methods function properly when called with expected arguments. This is sufficient to allow the systems to function reliably, as the possible values sent to methods are known. Future testing of these methods is required to ensure that they can cope with unusual values. The public methods, available for use in classes in any packages, have been tested for a greater range of arguments. This is because they are intended for use by people with no knowledge of the implementation details. The non-public methods will only be of use to people who wish to extend the functionality. This is not suggested until a complete technical study has been carried out.

As mentioned previously, performance testing was limited by the running time of the systems. A range of tests need to be performed on the neural network and the modular network, to find optimal parameters – the methods implemented are common, and should work well. More importantly, the genetic algorithm must be further tested in order to determine what algorithms are ineffective. New ones should be written, and different combinations tried. The individual algorithms used by the genetic algorithm are simple, and new ones can be written with ease. It is more the combination of algorithms that present the difficulty. Once more effective neural network generation is possible from the genetic algorithm, the modular network breeder can be properly tested on a harder task than the one previously presented in this report.

5.3.3 Extensions

Methods to load and save a neural network or modular network would be useful. Currently, the graphical interface requires the structure of the experts in a modular network to be specified. Load and save methods would prevent this being necessary. They should be easy to implement, as serialization of objects to disk is part of the functionality of Java.

A large range of different neural network architectures and their associated training algorithms can be implemented. Some examples are Hopfield nets [8 p234], or Recurrent networks [7 p119].

There are many different fitness, selection, crossover, and mutation algorithms for use with the genetic algorithm that can be developed. Some examples that could be useful are niching [11], and fitness sharing [11].

Some performance gathering tools would be useful. This is to allow the performance of the networks to be easily retrieved. Currently, the files generated by the monitors are read into MATLAB as matrices. Work has to be done editing the matrices in order to display certain results. It would be useful to implement a set of methods that automatically format the monitor data. These can be implemented by the monitor classes themselves.