3. DESIGN

3.1 Requirements

There are three requirements. The systems must be,

1. Flexible

2. Easy to test

3. Easy to extend

3.1.1 Flexibility

This is the primary requirement. There are three reasons for this.

Firstly, the performance of the systems is dependent on a variety of parameters. Mechanisms must be in place to allow these to be altered, both automatically, and by the user, so that optimal values can be found.

Secondly, the success the systems will experience is unknown. If a method does not perform well, it should be possible to change it for another without needing to re-implement any existing code. This is most relevant for the genetic algorithm, and the modular network breeding system. These use original methods whose success cannot be predicted easily.

Finally, both the structural and behavioural properties of the systems need to be able to be dynamically retrieved, altered, and copied. This is primarily to allow the genetic algorithm, and the modular breeding system, to alter the structure of the networks. It is also relevant to the various training algorithms that need to alter the behaviour of the system.

3.1.2 Ease of Testing

As indicated above, the systems are largely experimental. Testing must be a relatively hastle-free process, allowing the parameters to be changed quickly and easily.

The tests should produce a variety of results that can illustrate properties of the systems. These results must be in a format that can be easily interpreted. The reporting of these results must be able to be switched on and off, allowing different results to be displayed when testing for different properties. This has use in both correctness and performance testing.

3.1.3 Ease of Extending

One of my intentions is to produce a system that can be easily extended. It seems a shame to spend a year developing a system that will never again be put to use. If the proper groundwork can be made on a system that has the potential to be used in the future, then an effort needs to be made to allow the system to be as extendable as possible.

Another point is that flexibility and ease of extension go hand-in-hand. For a system to be truly flexible, it must also be extendable.

It’s worth pointing out that speed is not an issue. I am prepared to put up with a potentially slow program, providing the three requirements listed above are met.

3.2 Choice of Language

This project is implemented in Java. This decision was reached relatively quickly, for several reasons.

Firstly, neural networks and modular networks are inherently object-oriented. They can be decomposed into structural elements that link together in a highly organised way. These elements can easily be represented as objects in Java. Genetic algorithms also seem naturally suited to an object-oriented implementation, as they essentially deal with a population of objects.

Secondly, the integration of the different systems into the final modular network breeding system should be easy. The methods that do the bulk of the work will have already been defined on the sub-system classes. It should just be a matter of instantiating the classes I need, and starting the methods running with the appropriate parameters.

The third reason is because of the need to provide a flexible and extendable program. Java has in-built support for the implementation of different methods (that have a common super-class), and allows them to be dynamically switched between. This allows any new methods to be defined and used without changing any existing code.

Finally, Java is a language that I have quite taken to, as it feels to me like a natural progression from C. For these reasons, it’s a language I would like to get more experience of coding in.

3.3 Considerations

In designing the systems to meet the requirements listed above, some considerations come to light.

The first concerns the structure of the code, namely the hierarchical relationships between various classes. This includes the relationships between classes and their super-classes, and between them and the classes they use. Because of the natural decomposition into hierarchies already present in the systems, I don’t believe there will be a great deal of ambiguity as to the class hierarchies required.

The second consideration is the identification of the packages that the classes will be contained in. Again, this will not be a complex process due to the clear identification of the operations performed by the systems implemented.

Progress monitoring of the systems is an issue. A decision has to be made as to what information will be reported in real-time as the systems run, as well as to what information will be needed to analyse the systems performance. This information needs to be stored in such a way that it can be easily interpreted.

The final consideration is that of the interface with the user. A text-based interface will be sufficient to input the parameters, but there is the issue of whether it is easy enough to use, and easy enough to alter parameters.

3.4 Design of the Systems

This section will report the key designs that were used in implementing this project, illustrated with some UML diagrams. The entire project is too substantial to include a complete design. Appendix A provides a full listing of the classes and their methods.

3.4.1 Network class

The abstract class, Network, defines all artificial neural networks. It forces any classes that extend it to implement five methods that it specifies. These methods are the fundamental operations that define an artificial neural network, whether it is a feed-forward, layered network, or a modular network. See figure 3.1 below.

Figure 3.1: The abstract Network class

One example of where this is used is the modular network. The modular network can take any class that extends Network as an expert. This is because the only operations needed for the correct operation of an expert are those that the Network class enforces. This allows the possibility of hierarchical structures, where an expert is in fact another modular network.

3.4.2 Neural Network

The FeedForwardLayeredNetwork class implements the feed-forward, layered, neural network as described in section 2.1. It extends the abstract class Network (figure 3.1). The classes used by FeedForwardLayeredNetwork indicate the structural elements that combine to make a neural network. They are shown below in figure 3.2.

Figure 3.2: The FeedForwardLayeredNetwork class

Class Layer represents the hidden layers that can be present in a neural network. The methods defined on this class take as much work as possible away from FeedForwardLayeredNetwork. Even so, the bulk of the work still has to be done by methods in FeedForwardLayeredNetwork. This is because most of them require manipulation of the rest of the network’s components.

Class Input represents an input node. The number of inputs that a FeedForwadLayeredNetwork has does not vary. Because of this, Input objects are stored directly in FeedForwardLayeredNetwork rather than in a Layer - the number of inputs does not change, so there is no need to use the methods provided by Layer. Input nodes do not have connections to other nodes, but they do store references to the connections from other nodes. The distinction is described shortly.

Class ConnectedNode represents nodes that have connections to other nodes in the network. Most ConnectedNode objects are stored in class Layer. The exceptions to this are the ConnectedNode objects that are used as output nodes. Since the number of outputs in a network will not vary, they can also be stored directly in the FeedForwardLayeredNetwork class, like the Input objects.

Input and ConnectedNode both have a common abstract super-class, Node. Node exists to enforce a set of methods that define a node’s properties. The two most important are to set the activation of the node, and get the activation of the node. FeedForwardLayredNetwork calls these two methods on one ConnectedNode at a time, in a particular order. This propagates the node activations through the network to produce an output.

A ConnectedNode has a threshold, and uses classes Connection, Function, and ConnectivityPattern, as shown in figure 3.3 below.

Figure 3.3: The ConnectedNode class

Class Connection implements the connections in the network. A Connection object stores a reference to a Node, and a weight value. It has a method to return the weighted activation of the Node. Nodes store references to any Connections that are from a ConnectedNodes to the Node. This is to allow easy navigation forwards and backwards through the network. See figure 3.4.

Figure 3.4: The Node class

The Function class is an abstract class that enforces methods to implement a function, such as the Sigmoid function. It takes the ConnectedNode’s set of Connections, and produces an activation.

ConnectivityPattern is also an abstract class. It enforces methods that define a connectivity-pattern, given a number of possible Connections. These methods are used by ConnectedNode to ensure that if any methods are called to alter its connections, the connectivity-pattern is upheld. The connectivity-pattern is returned in the form of a bit-string, indicating which Connections will be kept.

The use of these different classes by ConnectedNode, means that a variety of different functions and connectivity-patterns can be implemented. Some are shown in figure 3.5. This leads to a very flexible neural network implementation.

Figure 3.5: Some implementations of Function and ConnectivityPattern

3.4.3 Modular Network

Class ModularNetwork implements a modular network, as described in section 2.2. It extends the abstract class Network. The UML diagram is shown in figure 3.6.

The ModularNetwork class stores a set of Network objects as the experts. The inputs of the modular network are represented by Input objects, as in FeedForwardLayered-Network. The gating network is implemented using ConnectedNodes with softmax functions, and full-connectivity to the Inputs of the modular network.

Figure 3.6: The ModularNetwork class

3.4.4 Training of the Neural Network Systems

The training of a FeedForwardLayeredNetwork is performed by a FFLNTraining-Algorithm. This is an abstract class that specifies the train method that a training algorithm must implement. The train method takes a FeedForwardLayeredNetwork, and some input-output patterns. It trains the network using the various methods that FeedForwardLayeredNetwork provides. The class BackPropagation implements the Backpropagation algorithm.

Similarly, the training of a ModularNetwork is performed by a subclass of the abstract class MNTrainingAlgorithm. The train method takes a ModularNetwork, and input-output patterns, and trains it using the methods provided by the ModularNetwork class. There are two MNTrainingAlgorithms, ErrorAdjust, and WinnerTakesAll., shown in figure 3.7.

Figure 3.7: The classes that implement the training algorithms

3.4.5 Genetic Algorithm

The genetic algorithm is implemented by class FFLNBreeder. When instantiated, this either takes a population of FeedForwardLayeredNetworks, or parameters that allow a random population to be generated.

There is only one general algorithm for breeding a population of components, which was described in section 2.3. This algorithm is defined by a breed method in FFLNBreeder. To provide a variety of behaviours, FFLNBreeder uses four classes that abstract the fitness evaluator, selection algorithm, crossover operation, and the mutation algorithm. This is shown in figure 3.8 below.

Figure 3.8: The FFLNBreeder class

These classes are extended to provide a range of algorithms. Some of the algorithms implemented are shown below, in figure 3.9.