----------------------------------------------------------------------------------------------------
---- HELLO -----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------

Stephan Dale
stephan.j.dale@2000.alumni.man.ac.uk or stephan@mindspill.net

This file is split into 2 sections...

PROGRAMS: Explains how to compile the code and the resulting programs' functionality.
CODE: Lists the files locations, how the files are related, the credits, and explains how to use my
      code in a new program.

Please also refer to my MSc thesis "Modelling Upper Body Movement in a Virtual Environment using a 
Neural Network". This is particularly relevant to the details of data gathering, pre-processing and 
NN training.

Watch out for the IMPORTANT bits - most of the code was written before some discoveries about the 
data were made. Read my MSc thesis for more info.

NN = Neural Network.

----------------------------------------------------------------------------------------------------
---- PROGRAMS --------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------

---- compilation ----

The environment variable MAV_HOME must point to the directory MAVERIK is installed in and WORK_HOME 
must point to the directory containing my code (ie. where this readme file is).

3 sets of makefiles have been provided. Those named 'Makefile' are identical to those named 
'Makefile-linux' and are written for compilation on Linux boxes. Those named 'Makefile-onyx' are 
written to compile on the Onyx 2. Use whichever are relevant to the machine you are on by renaming 
them 'Makefile' and typing "make all" at the top level (where 'readme' is located). This creates 3 
programs 'main', 'net, and 'datamanip'.

---- main ----

This is the main avatar control program. Type "main" to start. There are a number of command-line 
options, type "main -h" or "main -help" for usage instructions. Once in the program press (or hold) 
'h' to display a list of controls (for the keyboard, mouse and 3D mice). All command-line options 
can be controlled when the program is running.

The program creates an avatar and places it in the middle of a chequered floor. The initial view is 
from the avatar's eyes but this can be changed if desired.  All upper body movements can be 
controlled using the keyboard/mouse as can movement about the environment. Alternatively, body 
movement can be tracked using sensors and movement about the environment controlled using the 3D 
mice.

The arms can be positioned using the existing MAVERIK IK algorithm, using a Neural Network (NN) with
an IK correction algorithm (devised for my MSc thesis), or using the IK correction algorithm alone.

There are 3 main modes; normal, training or testing. The different modes require different sensor 
setups. The lab. setup as of 02/04/02 is assumed and consists of 4 sensor channels; HMD, VELVET, RED
and BLUE.

The modes...

Normal: This is intended for normal use of the system. The user has control of the head, torso and 
BOTH arms. The HMD sensor is attached to the HMD (suprisingly!), the VELVET sensor is attached to 
the chest (its position is translated to represent the back), the RED sensor is embedded in a 3D 
mouse held in left hand and the BLUE sensor is embedded in a 3D mouse held in the right hand. The 
user can move about the environment using the triggers on the BLUE 3D mouse and turn using the 
triggers on the RED 3D mouse. Pressing the top (thumb) button on either 3D mouse will grasp an 
object with the corresponding hand.

Training: This mode is used to gather training data (for later use training a NN). Before doing so a
filename must be specified to which the data is written. The user controls the head, torso, RIGHT 
hand and RIGHT elbow. As with the normal mode, the HMD and VELVET sensors are attached to the HMD 
and chest respectively, and the BLUE 3D mouse is held in the right hand. However, the RED 3D mouse 
is substituted for a sensor that is attached to the RIGHT upper arm to measure elbow movement. The 
user cannot move about the environment but can still grasp objects in their right hand. Instead of 
controlling movement the BLUE 3D mouse top trigger switches the writing of training data on/off and 
the bottom trigger switches tracking on/off. When the writing of training data is switched on the 
following right arm data, needed for training a NN, is written to the previously specified file:
hand_X hand_Y hand_Z hand_Roll hand_Pitch hand_Yaw upper_arm_Roll upper_arm_Pitch upper_arm_Yaw.

Testing: This mode is used to gather data needed to evaluate the NNs performance. It's very similar 
to the training mode. The setup and control is the same, however a NN predicts the right upper arm 
(elbow) yaw and pitch. When desired (ie. when the top trigger is pressed), data is written to the 
previously specified file to allow the performance of the network to be determined. The data written
is as follows:
tracked_upper_arm_Yaw tracked_upper_arm_Pitch predicted_upper_arm_Yaw predicted_upper_arm_Pitch.

IMPORTANT: The upper arm pitch and yaw are relative to the shoulder (calculated as follows; A vector
is cast from the shoulder to the elbow. The pitch is the angle that this vector makes with the -Y 
axis of the torso's co-ordinate system and the yaw is the angle made with the Z axis). THIS DATA 
WILL REQUIRE TRANSLATION BEFORE PRESENTATION TO A NN FOR TRAINING (see 'datamanip' below). The 
reasons for doing so are described in my MSc thesis.

Files...

Some of the functions require files to be loaded. These are:
1. NN file - this is the NN as created by the 'net' program (described below).
2. User file - this contains a user's dimensions, in meters, new-line seperated, as follows:
	       shoulder width (center back -> end of shoulder)
	       upper arm length (top of shoulder -> elbow)
	       lower arm length (elbow -> wrist)

---- datamanip ----

This is a command-line data manipulation program that allows pre-processing of the data files 
gathered using the main program. Start the program by typing "datamanip" and enter 'h' to see a list
of commands. The first step is to load a data file, either containing training data or testing data 
gathered using the program 'main'. You will be prompted with a number of questions about the content
of the file. There is no consistency checking - THE USER IS TRUSTED TO TELL THE TRUTH. 

Once a file has been loaded it can be processed in a number of ways. A file containing either 
training or testing data can be viewed, joined with another file, split into two or randomised. A 
training data file can be normalised, the hand angular data (r, p, y) can be removed if present, 
duplicate lines can be removed (called "cleaning") or the arm angular data (pitch and yaw) can be
calculated relative to the back rather than the shoulder. Finally, a testing data file can be used 
to calculate the real and SSE (Sum Squared Error) error between the targets (tracked data) and the
actual outputs (NN predicted data). 

In particular, the training data requires many of these pre-processing steps before presentation to 
a NN for training. I'd advise the following to be carried out in the specified order:
1. open the training data file (this will contain training data gathered from the 'main' program and
   hence the upper arm (elbow) pitch and yaw is relative to the shoulder).
2. join with as many others as necessary (watch out for the bug in this command - see 'bugs').
3. clean.
4. normalise for a given user (this requires a user dimensions file, as described above).
5. randomise.
6. translate upper arm (elbow) rotation so it's suitable for presentation to a NN for training. In 
   the program this is termed "get arm rotation relative to back (and yaw relative to x-axis) rather
   than relative to shoulder (and yaw relative to z-axis)".
7. check the resulting file for any inconsistencies (ie. values <0, huge values, NaN, blank lines..)
8. you're now ready to present the data to a NN for training.

IMPORTANT: Step 6 translates the pitch and yaw of the upper arm so that they are relative to the 
back rather than the shoulder (this new definition is calculated as follows; A vector is cast from
the torso's origin (the back) to the elbow. The pitch is the angle that this vector makes with the 
-Y axis of the torso's co-ordinate system and the yaw is the angle made with the X axis). THIS DATA 
IS NOW SUITABLE FOR PRESENTATION TO A NN FOR TRAINING. The reasons this is necessary are described 
in my MSc thesis.

Datamanip is by no means finished. There are a number of bugs, please read "bugs" for a list. 
However, the data will be properly pre-processed if the steps recommended above are followed.

---- net ----

This is a command-line program that allows the creation, training and testing of a NN. A NN with 1 
or 2 hidden layers can be created. Type "net" to start the program and enter 'h' to see a list of 
commands. The commands are fairly self-explanatory but it's worth noting what files are created when
you train or test a NN (you will be prompted for what to name them).

When training a network 2 files are created. The first contains the training error (RMS - Root Mean 
Squared) and the second contains the validation error (again, RMS). These have the format:
epoch_number error

When testing a network another 2 files are created. The first contains the targets and actual 
outputs of the NN. It can be fed into the 'datamanip' program to calculate the real error and SSE 
(Sum Squared Error). The format of the file is: 
pattern_number targets actual_outputs
For the upper arm (elbow) movement prediction task this will be:
pattern_number target_yaw target_pitch actual_yaw actual_pitch
The second file that is created contains the testing SSE, with the format:
pattern_number error

---- bugs ----

Please read the 'bugs' file for a (small) list of bugs.

----------------------------------------------------------------------------------------------------
---- CODE ------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------

---- file locations ----

/incl  			- contains headers for the source files in /src/sjd.

/src/ajb 	 	- contains 'nettools.c', Alan Beverage's (ajb) utility functions for using 
			  Jeff Shufelt's NN implementation (used by 'net.c', mentioned later).
/src/js			- contains 'backprop.c', Jeff Shufelt's (js) NN implementation.
/src/new_avatar		- contains 'build.c', the MAVERIK avatar building functions, with corrected 
			  hand offsets (unused since the latest release of MAVERIK is correct).
/src/sjd 		- contains 'main.c', the main avatar control program that creates the 
			  virtual environment and provides an interface to the functions defined in
			  'avatarlib.c'.
/src/sjd/avatar 	- contains 'avatarlib.c', the avatar library used by main, and 
			  'avatarlibio.c', that defines the I/O functions used for data gathering 
			  by 'avatarlib.c'.
/src/sjd/datamanip 	- contains 'datamanip.c', a data manipulation program for pre-processing 
			  data before presentation to a NN.
/src/sjd/NN 		- contains 'net.c', a NN creation, training, and testing program, and 
			  'sjd_backprop.c', a modification of Jeff Shufelt's 'backprop.c' to allow 
			  the creation of networks with 2 hidden layers. 

---- credits ----

/src/ajb			- by Alan Beverage.
/src/js				- by Jeff Shufelt.
/src/new_avatar			- by the MAVERIK team, but the hand offsets were modified by me.
/src/sjd			- by myself, Stephan (James) Dale, except...

/src/sjd/NN/net.c 		- functions trainIt, testIt, and viewPattern are modifications of 
			  	  code by Alan Beverage.
/src/sjd/NN/sjd_backprop.c	- a modification/extension of '/src/js/backprop.c' by Jeff Shufelt.

*** should mention Ashwin

---- how to use my code in a new program ---

The simplest way to use my avatar library (avatarlib) is to include 'avatarlib.h', call the 
functions you want to use, compile 'avatarlib.c', 'avatarlibio.c', 'netlib.c', 'nettools.c' and 
'sjd_backprop.c' into object files and link them when you build your program. The library provides 
a number of functions to instantiate and manipulate an avatar, as well as to track a user and 
gather data for training or testing a NN. Essentially all the functionality of the 'main' program 
described above is defined in this library - all 'main' does is call the relevant functions.

Gather the training data using the functions provided by 'avatarlib.c', pre-process the data using 
the 'datamanip' program, then create and train a NN using the 'net' program.

IMPORTANT: If you do the above, and gather data using the functions provided in 'avatarlib.c', then 
pre-processing step 6 must be carried out to translate the upper arm data before you present it to 
a NN for training (see the description of data pre-processing above, in the explanation of 
the 'datamanip' program).

However, if you wish to gather data and train a NN yourself (very easy using the 'datamanip' and 
'net' programs) here's what you have to do...

1. Gather the training data. For this you need the hand position and orientation, and the upper arm 
orientation. All data must be relative to the torso's co-ordinate system (origin at center of back,
X coming out of chest, Y up, Z along left shoulder). The upper arm rotation is calculated by casting
a vector from the origin of the torso (the back) to the elbow position. Pitch is calculated relative
to the -Y axis and yaw relative to the X axis. The reasons for doing so are described in my MSc 
thesis.

2. Use the 'datamanip' program to pre-process the data using the steps previously outlined (in the 
above description of the program). IMPORTANT: If the data has been gathered as described in the 
previous step, then data pre-processing step 6 should NOT be done. Step 6 is only necessary if the
data has been gathered using the 'main' program or using the data gathering functions provided by
'avatarlib.c'.

3. Create and train a NN. You'll need a NN with 6 inputs (for hand position and orientation), 2 
hidden layers and 2 outputs (for upper arm pitch and yaw). Run a preliminary training session to
determine at what point the validation error starts to rise. Re-initialise the weights (re-create a
NN) and train for slightly fewer epochs than it takes for this to happen.

4. Include 'sjd_backprop.h' (for the BPNN datatype) and 'netlib.h' (for NN tools) in your code.

5. Add the following global variables to your code:

MAV_avatar *avatar - the avatar.
BPNN *net - the NN.

float shoulderWidth - user's shoulder width (center back -> end of shoulder)
float upperArmLength - user's upper arm length (top of shoulder -> elbow)
float lowerArmLength - user's lower arm length (elbow -> wrist)

MAV_vector rightHand - a vector giving the right hand position
float rightHandRoll - right hand roll
float rightHandPitch - right hand pitch
float rightHandYaw - right hand yaw
float *rightArmRoll - right upper arm roll (always =0)
float *rightArmPitch - right upper arm pitch
float *rightArmYaw - right upper arm yaw
float rightArmRollNetwork - NN predicted right upper arm roll
float rightArmPitchNetwork - NN predicted right upper arm pitch
float rightArmYawNetwork - NN predicted right upper arm yaw

There should be another set of globals like the last 10 giving details of the left arm position and 
orientation (and similarly named). In 'avatarlib.c' the pointers to the arm orientations are set to 
point to either the values from the keyboard, the NN or the trackers (so if using only the globals 
mentioned above, set them to point to rightArmRollNetwork, rightArmPitchNetwork and 
rightArmYawNetwork).

6. The following functions must be copied from 'avatarlib.c' and incorporated into your code:

void getPY(MAV_vector vec, float *pitch, float *yaw) - gets the pitch and yaw of vec (where vec is
the vector representing the upper arm).

void flipPY(float pitch, float yaw, float *newPitch, float *newYaw) - flips the pitch and yaw of an 
arm so that they represent the pitch and yaw of the other arm (the NN is only trained on data from 1
arm and hence data from the other arm must undergo a conversion).
MAV_vector flipHand(MAV_vector handPos) - reflects handPos in YZ plane.

int loadNetwork(char *filename) - loads a NN from a file.
void predictElbows(void *unused) - uses the NN to predict the arm pitch and yaw (roll = 0). It sets
the globals rightArmRollNetwork, rightArmPitchNetwork and rightArmYawNetwork (and the corresponding 
left arm globals). This should be added to the rendering loop.

void setArm(int arm) - positions an arm of the avatar (using the global positions and orientations).
This should be added to the rendering loop (in 'avatarlib.c' it's called from a function called 
positionAvatar that is in the rendering loop and positions the whole avatar).

7. Compile 'netlib.c', 'nettools.c' and 'sjd_backprop.c' into object files and link them when you 
build your program. 

Note that for steps 5 and 6 it's probably best to rewrite the code so that it fits nicely into your 
program (say if you don't want to define the globals as I've suggested).

IMPORTANT: In 'avatarlib.c' the upper arm pitch and yaw is relative to the shoulder (previously 
explained in the description of the 'main' program above) and so the functions will only work if 
this is the case. Since the NN has been trained on arm data relative to the back, the predict 
function (defined in 'netlib.c', and called by predictElbows) performs all necessary translations so
that the returned predictions (of yaw and pitch) are relative to the shoulder.

IMPORTANT: In terms of the upper arm there are 2 ways to go:
1. The tracked upper arm data is relative to the shoulder (yaw relative to Z), pre-processing step 6
is carried out before presentation of the data to a NN for training and all functions remain 
unchanged.
2. The tracked upper arm data is relative to the back (yaw relative to X), pre-processing step 6 is 
not carried out, the translation steps are removed from the predict function in 'netlib.c' and the 
other functions (particularly setArms) are altered acordingly.

----------------------------------------------------------------------------------------------------

Well that's it. If you have any problems do not hesitate to contact me. My email addresses are 
given at the beginning of this file.

----------------------------------------------------------------------------------------------------
---- GOODBYE ---------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
