The active appearance model toolbox (AAMToolbox) can be downloaded from the author’s website, AAMToolbox and click on the icon next to the link shown below (Figure 1).

Figure 1 Computational Biology Download Page

Once you have downloaded the zip file Matlab_files_AIH.zip you can either unzip this using WinZip or if you are using Windows XP, you should be able to access the files directly. Either way copy the folder “Matlab_files_AIH” to your specified root “Shape_models” directory (please see the directory hierarchy section for a more comprehensive explanation of this).

Having copied the folder “Matlab_files_AIH”, you can start up your version of Matlab (note: this software has currently only been tested on Matlab version 7).

At the command prompt in Matlab type (notice here we have used the ‘>>’ symbols to show we are at the command line):

>>pathtool

This will bring up a Matlab graphical user interface (GUI) tool that will allow you to set the path to the previously copied Matlab files, see Figure 2. As indicated in the figure you should now click the button entitled “Add with Subfolders”, this will then prompt you for a directory. Please choose the directory “Matlab_files_AIH” that you copied as explained in the previous paragraph.

To test that the installation was successful please type the following at the Matlab command line prompt:

>>InstallTest

If the installation was successful, the user will be presented with a small dialog box like the one shown below:

Figure 2 Setting the path

Add with Subfolders “Matlab_files_AIH” to the path. Save the new path settings and close.

Getting Started

At this stage you should be happy that you have downloaded and copied the Matlab files into a suitable location. The directory structure at this point should be something like the one shown in Figure 3 (notice that the Matlab files are in a directory called “Shape_models” that also contains a folder called “Models”):

Figure 3 Directory structure used by the Shape Model toolbox.

Now you are in a position to start the toolbox, to do this type the following at the Matlab command line prompt:

>>AAMToolbox

Figure 4 First look

When the toolbox is opened from a directory that is not a Project the Current Template and Sample image axes are empty and the Current Working Project indicates ‘No Project Selected’

It can be seen from Figure 4 that if you are not currently working in a project directory then the system will indicate ‘No Project Selected’. Use the ‘Browse’ option to select a project.

Overview of Toolbox

Figure 5 AAMToolbox interface

Opening the toolbox from a directory that is a Project the Current Template and a Sample image is displayed.

The toolbox contains a set of tools that work with a Statistical Model project. There are two ways to access data in a project

Using the graphical user interface (GUI) called AAMToolbox
Programming in Matlab and accessing the data through the SM class library

Here we only discuss the GUI.

The details of the project and directories are displayed in the Project Details section of the toolbox at the top. Selecting a tool causes the toolbox window to disappear and open a window for the tool. To return to the toolbox, close the tool or click on the ‘back to toolbox icon’.

If, for some reason, the windows refuse to close normally they can be forced to close by typing the command ‘CloseAAMtoolbox’.

A project is based on a specific set of images. Each project contains 5 folders that contain the original images, processed images, templates, point models and statistical models. You can change or specify the project you are working on using the browse option in the Project Details section. Projects are created using a series of tools in the toolbox.

1. The first tool is New Project and this asks for the name and location of the desired project. The system automatically adds the prefix: ‘PRJ_’. Be careful in choosing the name to ensure that it reflects the images that the project contains. The New Project tool can also be used to combine or join pre-existing projects. You can also import one project into another.

2. The next stage is to import images into the original and processed images subfolders of the project. This is done manually by copying and pasting the images into the relevant folders. Each jpeg image should have a distinct name (i.e. imagename.jpg). It is useful if the name represents useful information, e.g. experimental details, subject name, date, etc. The processed images should all have the same size in pixels (a convenient size is 800 pixels wide). To help with processing, you can use the image processing tool. Note that if you want to create an allometric model (one that does not normalise size), you need to untick the scale box in the Procrustes section of the SMG (Statistical Model Generator) tool (see below).

3. You then need to import at least one template into the Templates folder of the project. This can either be done manually by copying previous templates and pasting, or using the template editor tool to browse the list of templates. If you do not have a template you will need to create one using the template editor tool. Templates all have the suffix .temp_dat. (In some cases a supplementary file with the suffix .edit_set will be created.)

Figure 6 Point models to Statistical models.

4. To start creating models (Point Models for each image and Statistical models derived from multiple images) you need to assign one template from the Templates folder using the choose template tool. This will create folders with the template name in PointModels and StatisticalModels directories (i.e. Pointmodels/template_name). Thus all models based on this template will carry the template_name as an identifier.

5. To define the positions of points in relation to the images there are two options.

A. Point models are created according to the template (selected previously, see above). They are usual created manually using the Point Model Editor (PM_editor) tool. Points are moved to the appropriate feature points in the image. A Point Model is created for each image in the Cropped directory. The pm (point model) files are stored in the template-name subfolder within PointModels directory. (i.e.pointmodels/template_name/imagename.pm). Note that these are not exclusive – you could first use Autoplacement and then check the points manually using PM_editor

B. For some image shapes (e.g. leaves), you can automatically place points on your images using the Autoplacement tool. When you invoke this tool, it will ask you whether you want to place points through segmentation, warp all points according to a subset of points, or fit using a probabilistic model based on a subset of points. In the segmentation option you will be asked to specify the type of image you want to segment (e.g. petals, leaves etc) and will then create a set of binary images in a processed images/binary subfolder. You then click on a few key points and draft point models are created.

7. When Point Models have been created for each image you need to consider which points to use for your statistical point model. Models associated with each set are stored in separate subfolders of the StatisticalModels folder. By default all points of the template are used to create the statistical point model. This is stored in a subfolder called Set_1 located in StatisticalModels/ template_name. You may also want to select a subset of points. This is done using Point Sets tool that allows you to choose the subset of points to be used. The point sub-set information is stored in a newly created folder called Set_2 (further subsets will be named consecutively) located in StatisticalModels/ template_name. You can also give the selection of points a logical name and this will be displayed in the project details area.

8. To build a statistical model you now use the SMG (Statistical Model Generator) tool. Using this, you first need to choose the images that should be included in the statistical models. By default all images in processed images will be used. This will create a subfolder called ImageList_1 in StatisticalModels/ template_name /Set_1. You may also want to select a subset of images. This is done using a drop-down menu within the SMG tool. This allows you to choose the subset of images to be used and when this is complete, the image set information is stored in a newly created folder called ImageList_2 (further subsets will be named consecutively) located in StatisticalModels/ template_name /Set_1. You can also give the selection of images a logical name and this will be displayed in the project details area. Within SMG you can also generate a combined shape and appearance model. (You can also generate appearance models based on treating each triangle as a patch or using wavelets: to be done).

9. Statistical models can be displayed using the View Stats Model tool. Within this you can

Examine how each principal component affects shape and appearance
Make movies of the shape and appearance changes
Import images to modify the mean appearance
Evaluate how well an image is created by the model compares to the original image
Examine the effect of quantising shape and appearance space

10. Statistical models can be viewed in shape space 2 or 3D shape space using the View Shape Space tool.

Image Processing Tool

Within the Image Processing tool you can perform a number of operations on the images stored in the directory processed images. This tool assumes that you have copied the images from the originals directory and are ready to process them by resizing, cropping and selecting regions to remove background noise. In practice, it is often easier to write your own Matlab image processing tools or even use an image editor such as Photoshop.

When you select the Image Processing tool you will be presented with the following window,

Figure 5 Image processing toolbox

This tool automatically chooses the images in the processed images directory as the ones to be processed.

Selecting images

To navigate through the images you can either click the “Next” and “Prev” buttons or click on the drop down menu that shows the image filename and skip to any image in the list.

Resolution

To change the resolution of an image (i.e. reduce or increase the number of pixels per ) you type in the ratio you require (shown at 0.5 in the above figure which will reduce the number of pixels in the image by half) and then you can either apply this ratio to the shown image by clicking on the “Change Resolution” button, or you can apply it to all the images in the processed images directory by clicking on the “Apply To All” button.

Crop and Select

To crop an image you type in the number of pixels you require for the width and height in the text boxes provided in the ‘Crop Panel’ and then click on the “Crop” button. The mouse cursor will then change to a full crosshair; you are then required to click on the center of the required region. Notice that if you type in a crop size that is greater than the size of the image then it is padded with zeros.

Also within the “Crop Panel” you can select regions of interest in order to reduce the amount of background noise within the images. To do this you click the “Select Region” button in the ‘Crop Panel’, this will change the mouse pointer to a crosshair as shown in the left figure below. You can then click using the left mouse button around the region of interest, when you are satisfied that you have captured the region you simply click with the right mouse button to complete the selection. An example of this is shown in the right image below.

Figure 6 Cropping the image

At any stage of the image processing you can click on the “Undo” button, this will cancel any resolution, cropping and region selection you have performed. If you are happy with the processing you must click on the “Save” button before continuing onto the next image.

Template Editor Tool

In order to build any statistical shape model you must have a number of point models. Each point model is a set of points placed around important features of an image, e.g. around a leaf, petal or face. It is critical that the same features are chosen in every image to be included in the statistical model. To make this easier we use a template. Select an image that represents all the features that characterise the images. Then use ‘Add Points’ to place a point on every feature. Next define some, or all, of the points as Primaries. Primaries should be placed on features that can reliably be identified, e.g. corners. The remaining points, secondaries, will be evenly spaced along a spline that joins the points, between the primaries (press Smooth Landmarks). Finally, join together points that form recognisable shapes. Edges should join secondary points together and each set of secondary points should have a primary at each end.

To construct a template you use the Template Editor tool, when you click this tool you will be presented with the following window,

Figure 7 Template Editor before and after loading (or creating) a template

If you are working with a new project, then this window will appear blank as above, if however you are continuing work on a project this window will show the currently active template.

Importing Templates

If you want to import a pre-existing template into the template folder, click on “Load Template” and then browse to the template you want and click on ok. Then save and quit.

It is also possible to import a point model and image from the set of point models already fitted to images (Template: Import point model from Point Model File). Since it is the same template this will not change the final statistical models, however, it can be easier see how the points should be assigned to features in new images where these and the starting image are similar.

New Templates

To make a new template you must click on the “Template” menu and select “New Template”, this will ask you to select your representative image from the processed images directory. Once you have selected your template image, you can then add the points in the appropriate positions. To do this click on the “Add Point” button, this will change the mouse cursor to a crosshair and allow you place points around the image. Similarly by clicking the “Delete Point” button you can remove points from the image.

Choosing Primary Points

In this tool you can also select the points that you would like to highlight as being primary points, primary points are those that convey important information in the image, i.e. points of high curvature, inflections, points of extrema (width, height). Once you have selected your primary points, you can choose to smooth the other points (known as secondary points) around the contour (spline that goes through the secondary points and the primary points at either end. This is done by clicking the “Smooth Landmarks” button and an example of this is shown in the figure below.

Figure 8 Smoothing secondaries

Red secondary points between green primaries are evenly (smoothly) placed between primaries (right panel).

Point Connectivity (Drawing Lines or edges)

In order that secondary points can be evenly placed along the spline linking them with the primary points at each end, the connectivity must be provided by joining the points with lines (edges). These lines also make the appearance of the statistical models more appealing, and informative, you can select which points are connected to each other; this implicitly defines the notion of groups and can greatly increase the information about a particular model.

To define these edges you click on the “Add/Remove Edge” button, you then click the two points that you would like an edge along. If there is already an edge between these two points then it will be deleted else if no edge exists, then one will be drawn. This process is partially shown in the following figure. Having added the edges press the “Check edges between Secondaries” button to run a continuity check.

Figure 9 Adding edges between points

Saving the Point Model Template

Once you are happy that you have captured all the information in the template you can then save the template by either clicking on the “Quit (and Save)” button or by clicking the “Template” menu and selecting “Save Template”.

Global Point Model Editing

It may be that at some time in the future (after you have made your template, placed your points around your sample images and built some statistical models) you will want to add more points to the template. For example, suppose I have the following point model template, and below this is an example from View Shape Model.

Figure 10 The need to add points to an existing template

But I am now interested to see the variance in the position of the main vein? More points are needed. The Template Editor tool allows you to add points to a template, and then populate this change throughout the other point models. Simply click the “Add Point” button as before and place the new points in the required position.

We have added new points to an existing template and will probably want to use all the point models that place the existing template onto the images. To apply the new template to all the existing point models select ‘Template:Import Point Models’. You will be asked to choose a Point Model Directory from which to copy existing point models. The system will then load these models into memory.

Finally, “Quit (and Save)”. Give a new filename to the new template. Having saved the new template in the Templates folder the system will save copies of the old point models into a new directory. Simultaneously, it will add the new points. The points will not, of course, be in the correct place and so you have to use the Point Model Editor to go through the images, moving the new points into position. To do this, use the ‘Choose Template’ to select the new template (accept the default options). It might take a short while to create the new model directories. The ‘AAM Toolbox’ template model icon will change. You are now in a position to use the ‘Point Model Editor’ to move the new points into place.

Notice that you can build statistical models from just the old points by selecting them with the Point Sets tool.

Figure 11 New points added to an existing template to create a new template.

New Project Tool

When you call the New Project tool, you will be presented with the following dialog box and asked where you would like to store your new project, (it is recommended that you keep all the projects at a level below the root folder “Models” as shown below.)

Figure 12. Selecting the directory into which a new project will be added

Once you have selected your folder where you want the project to reside, you will be asked to name your project as shown below:

Figure 13. Adding the name of the new project (the PRJ_ prefix will be added automatically)

The name is automatically prefixed with ‘PRJ_’. The project should be named according to the images it will contain. If it is a combined project, use a name that indicates the level of combination (e.g. all_paintings or all_petals).

You are then presented with the following question.

Figure 14. A project with new images or a project that uses a combination of existing projects.

If this is a completely new project, click new.

If it is a combination project, you will be presented with a list of existing projects. Highlight those you wish to be combined (either hold down the Ctrl button and single click each project or hold down the Shift button and click on the first and last project to select a block). Then click ‘Ok’. Note that all projects to be combined should have the same template (they could share more than one template). Point models using the same template from all the original projects will be placed in the same directory. However, the existing names will be prefixed by the source project name. if it happens that you try to combine two projects that both have a template with the same name, but this template contains different sets of points, then only the first set of point models will be copied.

Once you have created your project you will notice (using the operating system directory viewer, i.e. windows explorer) that several directories have been created for you. This structure is described in more detail in the Overview.

Making Augmented Projects

You may decide that you want to make an augmented project. This means that you can effectively merge two projects into one. By doing so you will merge two templates into a single template, and you will also merge all corresponding point models for each individual project in to a single set of point models. For example, say you have a project built on leaves, call it LEAFPRJ, and a project built on petals, say PETALPRJ. Then you may find it interesting to merge the two and see how they vary not just within themselves but with respect to each other. To do this you simple click the “Augmented” button when asked for a project type, see Figure 14. Upon doing so you will be asked to select a root directory from which to look for other projects, and then you will be ask to select from a list, which projects to merge.

You will then be asked to select a template for each model you have selected. This is form part of your augmented template at the end of this process.

Once you have complete these steps you will now have to choose which point models from one project go with point models from another project.

From the Project menu select Augmented Projects and then Order Point Models.

Here you will see that you are able to select the ordering for the first project you have decided to merge. You can manually choose the ordering.

Doing this for each project means that you will have built up a correspondence between point models in different projects. For example, here E-0002-00002-FL-00003-a-05-07-27_pm.mat is the first point model for project 000_Set. That point model will be associated with the first point model chosen for the other projects, so ordering here is very important. This also means that you do not have to have adopted a specific naming convention.

You will now want to make your augmented template, to do this simple select Make Augmented Template from Project->Augmented Projects. This will ask you to specify a name under which to store the template.

Finally you are now in a position to choose Project->Augmented Projects->Build Augmented Point Models. This will ask you for a template from which to build the augmented point models, and then the rest is done for you.

You can then continue by clicking Choose Template as usual and build your models.

Importing Group Means

If you have decided to build a project that is a combination of other projects, it might be useful to include the mean shape and mean appearance of each project into your data. To do this select the ‘Import Group Means’ item from the ‘Tools’ menu from the toolbox (circled in the figure below).

Now when you click on the ‘Show Names’ button in the toolbox you will see the files ‘MEAN_TEMPLATE_<template_name>_<project_name>’ added to your list of available shape and image files.

Choose Template Tool

Once you have created a new project or opened up an existing project, you will want to choose a point template model which will serve as your reference image and point model.

To select a particular template from which you will be able to build shape and appearance models you simple click on the “Choose Template” button from the toolbox. When you click on this button, the toolbox will create the relevant directories to store the point models and statistical shape and appearance model information. If these directories already exists you can simply click ok and continue.

Point Model Editor Tool

The Point Model Editor tool allows you to place the points defined by the point model template which has been selected by the Choose template tool. To open the Point Model Editor tool you simply click on the “PM Editor” button in the toolbox and you will be presented with the following window. Here you can see that on the left, the current template is showing the locations of the specified points in their respective positions in the image. This is to be used as a guide when placing the points on the target image on the right.

Figure 15. The template on the left provides a reference as points are manually moved into position for each new image shown on the right.

Moving Points

To move points around the target image, click on the “Move Point” button to select ‘Move mode’. In ‘Move mode’ the mouse buttons perform different operations.

Left Mouse Button

Right Mouse Button

Left + Right Mouse Button

If you click a point using the left mouse button and hold it down, the selected point will move under the mouse point as you drag the mouse around.

If you click on the points using the right mouse button and hold it down, as you move the mouse pointer around, all the points will be translated.

Clicking and holding both the left and right mouse buttons together has the following effect.

Moving the mouse up and down rotates all the points.

Moving the mouse right and left, scales all the points.

Pull in Outliers. Occasionally, points disappear out of the image. This button brings them back.

Point sets for editing. Depending on the template there may be an extra menu item: ‘Point sets for editing’. (This will be created if there is a template ‘helper’ file called ‘*.edit_sets’. This file is created by an auxiliary program especially written for the particular template. It will be called ‘ConversionTools\*_place_fixed_points’ where the * stands for the template name. This menu allows predefined sets of points to be viewed in isolation. It also provides a mode that will allow ‘stacked’ points (i.e. points that overlie each other so closely that they would normally be moved as one) to be separated.

Smoothing Points

Once you have moved all your points around the image, you may find that the secondary points are not evenly placed along an edge between primary points. Secondary points provide interpolation points to give a good representation of shape. The ‘Smooth Landmarks’ button automatically smoothes the secondary points between primary points by fitting a cubic spline and sliding the points along it until they are evenly spaced. An example of this is shown in the figure below.

Figure 22 Using the ‘Smooth landmarks’ button to distribute secondary landmarks (red) evenly between primaries.

Other functions

There are some other functions in this tool that will now be discussed.

If you want to pan or zoom into areas of the target image, you can click on the “Zoom” and “Pan” buttons provided. Notice here that the template image will keep track of the area that you are zoomed in on and hence will update its view accordingly. You should never need to click on the template image.

At the bottom left of the Point Model editor tool there are a selection of manual rotation and translation buttons.

It should be pointed out that the saving of the point models is an automatic procedure in this tool. The point model is saved as soon as you click the next, previous, or skip to another image in the list by using the drop down menu provided.

Automatic Placement Tool

This tool only works for certain templates. It provides several methods to automatic point placement. When you click on this tool you will be provided with the following window.

Figure 23

Here the user selects which method of fitting they would like to use.

Segmentation Fitting

If you choose to fit your point models using the segmentation procedure, then you will be presented with the following window.

Figure 24

This window is asking whether you would like to segment the images you have in processed images directory or if you want to place the points onto a set of already segmented images.

Segmenting

If you have decided to segment your images then you will be presented with the following window

Figure 25

From here you must select the appropriate segmentation algorithm according to the images you are processing. If there is not a suitable algorithm available to you, then you can either contact the administrator or write your own in Matlab and place it is the appropriate directory under ‘Matlab_Files_AIH’. Once the segmentation algorithm has finished you will be asked if you want to continue to place the points automatically or you can return and do this another time.

Placing Points

To automatically place the points you will be asked to choose a template which will be used as the reference template. Then you will be presented with a window similar to the one shown below.

Figure 26

Here we can see that the template is shown on the left and the first image in the list is shown on the right. We can tell that this particular image does not have any points placed already as they are not shown on the image.

To place the points click the “Click Primary Points” button. This will change the mouse pointer to a crosshair and you will click in the positions of the primary points (in this case there are 5 points [the green ones]).

Figure 27

This will give you a result similar to the one shown above. These points have been placed using five clicks of the mouse, a significant saving of time. The point model file for this image has automatically been generated. Once you have done this procedure for the rest of the images in the list you can then use the PM Editor tool to manually place any points that were placed accurately enough.

Subset Fitting

Not Yet Implemented!!

Likelihood Fitting

Not Yet Implemented!!

Set Picker Tool

The set picker tool allows you to select a subset of points to be used when building the statistical models. First you might have to ‘Clear all landmarks from Set’ Figure 28. If you want to select a subset of these points and build a model on those, then you can click the “Select Landmarks” button. This will change the mouse pointer to a lasso and you left click to surround the points you want to select. To finish your selection you simply click the right mouse button. To save the subset click the “Quit and optionally, Save” button. This will ask you to choose a logical name to represent the selected subset of points, i.e. dorsal petal.

Figure 28. Set Picker, for selecting a subset of points. Note the ‘Clear all landmarks from Set’ button on the bottom left. Clicking the ‘Import List’ button causes the system to look through all Projects for matching templates and, from these, existing relevant all subsets. One can then augment and combine these to produce new subsets.

Once you have selected a subset of points, you will then have the option in the statistical model generator (SMG) tool to model them in context with the mean of the other points or as a separate unit of points.

Make a new subset, you must click on the ‘Clear all landmarks from Set’ button. Then you can click the ‘More Landmarks’ button to select the points you wish to include in your statistical model.
Statistical Model Generator (SMG) Tool

At this point you will have gone through the steps of processing your images and placed all the points either manually or automatically and are now ready to create statistical models.

To build statistical shape and statistical appearance models we can now use the statistical model generator (SMG) tool shown below.

Figure 29. The statistical model generator.

Here we can see a set of points defined by the template and any subset that has been selected. These points will be used to build the statistical model. We have, however, to choose which images (point models) to use and the number of pixels to use for the appearance model.

Modelling Subsets in Context

At the bottom right of the statistical model generator (SMG) tool you will notice a check-box with the string ‘in context’ next to it. If you build a statistical model with this checked then your resulting shape model (and indirectly your appearance model) will be built on all the points in the template, with the condition that any points in the template that are not in the point set will be set to the mean shape for that group of selected images. In this way you can see how the points in the point set vary in context with the mean shape for that image set.

Select Images

You are able to select the images that will be used in building the models. To do this click on the “Select Images” button, this will provide you with the following window.

You can either select individual images by holding down the Ctrl button and clicking the images in turn. Or you can select blocks of images by holding the Shift button down and clicking the first and last image in the block to be selected.

Alternatively you can select a set of images by using a filter (namely a regular expression parser). To do this click on the “Filter” button.

The ‘wildcard’ character is a star (*). Thus, using the characters sequence *0002* would select three files in Figure 30 that contain the sequence 0002.

If you click the “Help” menu and select “Web Help” you will be directed to a web page that has more information on writing regular expressions.

Figure 30

Building Statistical Models

To build the statistical shape and statistical appearance model you simply click the “Generate Models” button. This will present you with a progress bar indicating the percentage of completeness of this process.

Once you have successfully build a shape and appearance model, you can then use the View Stats Model (pdmwalk) tool to display the results.

View Stats Model (PDM Walk) Tool

Once we have placed the points around a set of images, chosen the subset of points we want to use to build a statistical point and shape model, and chosen the subset of images that we wish to use. We are in a position to build statistical shape and appearance models using the View Stats Model tool. To open this tool click on the “PDM Walk” button in the toolbox and you will be presented with the following window.

Figure 31. View Stats Model (ModelViewer).

Displayed in the top left is the mean shape and appearance of the point models we have discussed. Inside the boundary of the mean shape is the mean appearance. To deviate from the mean, adjust the sliders on the right hand side. The change will then be added to the mean to give a new shape and/or appearance. The top set of sliders control the shape, and they are ordered so that the first slider corresponds to the largest principle component, the second slider corresponds to the second largest principle component etc.. Similarly with the bottom set of sliders that correspond to the principle components of appearance rather than shape. To automatically run through the shape variations you can simply on the “Full Walk” button, and similarly to automatically run through the appearance you can click the “Full Walk A” button. The full walk step size (smaller is better but slower), the number of components and the image size can be altered using the menus along the top. To increase the number of pixels in the appearance model (more is better), create a new model with more pixels. A movie can be made by starting to record (Movie menu), running a Full walk, and stopping the movie (Movie menu). The movie is accumulated in memory and this might run out. For this reason, it may be necessary to increase the Walk-stepsize.

Shape Space Tool

Once you have built your statistical models it would be useful if we could visualise this space in some way other than a walk along the principle components. This new tool is known as the Shape Space Tool. Here we are able to create, combine and visualise groups of point models. And more importantly if they are point models that have been created using the template that was used to build the model, then we are also able to project or represent these point models in the space created. An example of this would be if we had point models of an F2 population together with the point models of the two parentals. Having this data allows us to create statistical models based on the F2 data set. We can then project the parentals into this space and see the relationship between them.

To start the tool click on the button shown below:

If you have successfully created a statistical shape model, then you will be presented with the following window:

Quick Start

To run this tool you must do the following things

Add/Open groups
Load Groups
Plot Data

Once you have gone through these steps the rest of this tutorial will make more sense.

This is the main control window for the shape space tool, and we will go through the features of this tool in order. Firstly we will address the features in the menus:

File Menu

· Quit – this exists the shape space tool and returns to the main toolbox interface

View Menu

· 1D Projection:

§ Line Graph:

· Range: this option will draw a 1D line that represents the range of the data

· Standard deviation : this option will draw a 1D line that represents a standard deviation for that group when projected onto that principle component

§ Particles: this option will draw the particles in 1D instead of drawing a line.

· Toggle Visuals:

§ Ellipsoids: this feature allows you to specify which ellipsoids to show in the plot. You are required to select the groups for which you want to hide the ellipsoids. To recover all the ellipsoids, simple uncheck and then re-check the ellipsoid check box.

Tools Menu

· Projected t-test: this feature calculates the pairwise t-test of a selected number of groups. Initially you are required to select the groups that define the base group (i.e. wildtype) and then you select the other groups that you want to compare to your base group. Finally you must select the principle components on which you want to perform the t-test. You are then taken through the steps to save the results to an excel file.

· Calc. Variance of Projected: this feature allows you to select a group of groups that will be your base group. You are then asked to define a further set of groups for comparison. Then a shape model based on your base group is constructed. For the selected PC’s each group is projected onto each PC and the distribution is displayed. The distribution of the base group is shown as a dashed line.

· Show PC Variances: this feature displays the variance explained by each principle component. The plot shows percentages and other various forms for clarity.

· Constrained LDA: this feature shows the between and within class scatter for the model data on each PC.

· Units:

§ Raw: this option plots the particles using the raw deviations from the mean.

§ Standard Deviations: this option plots the particles using the standard deviations from the mean.

Groups Menu

· Clear Groups: this option clears the groups that you have opened and loaded. You will see that the Group panel is emptied

· Ordering:

§ Automatic: this option will order the groups in the 1D view according to the mean of the group. The higher the mean the further up the ordering that group will be.

§ User Defined: in this feature you force the 1D view to order the groups according to your choice (see Groups Menu -> Define Order)

· Define Order: here you can define your custom ordering for your groups, this is the order that they will appear in the 1D view.

· Random Colour Groups: selecting this option will assign a random colour to all the open groups. To show the colours you will have to click ‘Plot Data’ to show the difference.

Windows Menu

· Shape Window: this toggles showing the shape window

· Appearance Window: this toggles showing the appearance window

Vectors Menu

· Import: this option allows you to import vectors from an excel file. Each column of the excel file represents a vector, and the row of the excel file corresponds to the principle component for that vector. IMPORTANT: the sheet name must be ‘ShapeVectors’.

· Draw: this option allows you to manually draw vectors. You can only do this when you have two principle components selected (i.e. you have a 2D plot). To draw you simply left click in the axis where your groups are plotted and keep left clicking until you have drawn all your vectors. To finish you right click your mouse button on the plot. You will see a message in the Matlab window to tell you that you have finished.

· Export: you can export your vectors to an excel file, this option creates a sheet named ‘ShapeVectors’ within your specified file.

· Walk: this option allows you to see the effect of walking along your specified vectors.

· Print Vectors: here you can display the currently loaded vectors in the Matlab command prompt.

· Clear Vectors: this option clears any of the current vectors so you can start from scratch.

Right Clicking the Particles

To retrieve information about a particular particle in the current plot you can right click on any point and you will be presented with a drop down menu with several options (shown below):

· Original Image: clicking on this option will open a new window showing the image that corresponds to that point, and its point model.

· Show PC Values: this option will print out the currently selected PC values to the Matlab prompt. Here it would display the PC values for principle components 1 and 2.

· Distance: this procedure has two parts. You first click on the starting particle, then you are asked to click on a second particle. Then a new figure is created and the values of the principle components to get from one to the other is plotted.

· Add Shape To Vectors: this option adds the current deviation vector to the shape vectors (see Vectors Menu). You can build paths like the one shown below:

REMEMBER TO CHECK THE VECTORS CHECKBOX!!

Display Options Panel

· Choose Shape Axes: this lets the user define which shape PCs should be displayed. The maximum number allowed is 3.

· Choose App. Axes: similarly this lets the user define which appearance PCs should be displayed. The maximum number allowed is 3.

· Fit Surface: this button fits a surface around each group, the surface is controlled by the parameters in the Surface Quantization panel. This panel tells the toolbox how coarsely the shape space should be broken up.

· Plot Data: this button does exactly what you would expect. It re-plots any data.

Isosurface Panel

· Isosurface Level: this slider controls the isosurface for each group. If you have clicked the Fit Surface button then each group will be assigned a volume, and surfaces can be extracted from this volume based on their values. This slider controls which surface should be displayed.

Surface Quantization Panel

· dx: edit box to define the number of components in the x direction for building a volume and extracting a surface.

· dy: edit box to define the number of components in the y direction for building a volume and extracting a surface.

· dz: edit box to define the number of components in the z direction for building a volume and extracting a surface.

Plot Options Panel

· Group Means: toggles the group means

· Particles: toggles the particles

· Iso-Surface: toggles the isosurface’s

· Vectors: toggles the shape vectors

· Group Labels: toggles the group text labels

· Ellipsoid: toggles all the groups ellipsoids

· LDA: re-plots the data using LDA vectors

· Area Vector: toggles the vectors that points in direction of increasing area

Dimension Panel

· Dimension: this panel lets the user know how many principle components are currently being plotted.

Group Panel

· Group Pane: this pane shows you all the currently opened groups.

· Add: this button brings up a list of available point models and allows you to select which ones you would like associated with a particular group.

· Import: this button allows you to import groups from another groups file.

· Delete: this button deletes the currently selected groups from the group pane.

· Change Color: this button changes the color of the currently selected group.

· Combine Groups: this button allows you to combine groups into a super group

· Load: this button allows you to load the currently open groups.

· Open: this button allows you to open and display a valid groups file.

· Save: this button allows you to save a groups file.

Group Members Panel

· Group Members: this pane shows the elements of the currently highlighted group. The colour of the text corresponds to the colour of the particles and the ellipsoid for that group.

Creating Figures for Papers

Often you will want to create figures for papers, and it is also clear that these figures can be similar from paper to paper. For this reason we have started to create a library of figure creating scripts.

Principle Component Effects

We have created a script that describes the effect of each PC clearly and concisely. To run the script simple type

>>figure_2_pc_effets

This is open a new dialog box asking you for the location of the model, this will be somewhere similar to the one shown below. Notice that it is NOT asking you for the project directory but the actual folder where you have saved your model.

Then you will be asked several questions before displaying your figure

1. which principle components do you want to display?

2. do you want to use the mean from the model?

3. Enter the vertical space at the command prompt.

4. Enter the horizontal spacing at the command prompt.

5. do you want to flood fill your template in the figure?

6. do you want to use the convex hull from your shape to control the area (this is useful if you have internal points)?

7. how many standard deviations do you want to show?

The result of all this work is a figure similar to the one shown below.

We can see here that I have not entered the correct horizontal spacing for this figure. But this will be different for every project.

Often you start a model with a simple template, perhaps just ten or so points. Then, having annotated a set of images to create a set of point-models and built statistical models it becomes clear that more points are required. Either, create a completely new template (with a new name) and mark up all the images again (create a new set of point-models) or, update the current template by adding new points and give it a new name. In this case, the system can copy all the existing point models into a new directory adding in the new points as it goes. Of course, you still have to use the PM Editor to place the new points in the correct position, however, there is no need to alter the old points. This means that it is easy to add a few new points to an existing set of point-models.

Step 1: Add new points to a template, e.g. Templates\Faces5.temp_dat, and saving it with a new name, e.g. Templates\Faces10.temp_dat.

Use the ‘Template Editor’. Load the old template (Template menu) and add new points. Alternatively, load a copy of the already updated template from another project.

Before saving and quitting …

Step 2: Update the old point-models, e.g. PointModels\Faces5\*_pm.mat, and saving them with a new name, e.g. PointModels\Faces10\*_pm.mat.

In the ‘Template Builder select ‘Import and update Point Models’ from the ‘Template’ menu. Select the old point model directory.

Now, save and quit using the ‘Quit (and Save)’ button

Step 3: Moving the new points into their correct places

Use the ‘PM Editor’

Updating existing templates and point models

If you choose to make a global update in the global editing tool, then you will find that the Point Model Template (PMT) that you store in your project is different from someone else who is using the same template.

To get around this, there is a utility called `UpdatePointModels` which is launched by typing

>> UpdatePointModels

for two template files

· The first template file is the new modified template file

· The second template file is an existing template file that needs to be modified together with its corresponding point models (located in `PointModels\<templatefilename>`).

After you have selected the two template files, all the templates and the corresponding point models will be updated.

In future versions this process will be automated asking the user for a root directory to search from. All templates with the same name and hence corresponding point models below this root directory will be updated.

Conversion between old and new point model templates

If the installation of the Matlab files has been successful, you should be able to access a tool called md2temp_dat by calling

>> md2temp_dat

This tool will prompt you for two files:

1. <template_name>_md.mat (The old style template file)

2. <template_image_name>.jpg (The corresponding image file)

Then the tool with ask you where to store the new converted template:

<new_template>.temp_dat (The new template file)

Comments

It is the hope that this software is sufficiently bug free to be able to build simple models. There will be bugs! If you encounter a bug whilst using the software, please do the following:

e-mail aih@cmp.uea.ac.uk and report what program you were using and please copy the error that appeared at the Matlab prompt, eg.

Also please attach a sample of the data that you were using to create the bug.
Please copy any information in the bug sheet at the bottom of this document giving your name, data, and the name of the program you were using, and the error from the command prompt.

Allometry model – the Mathematics

Descriptive version

Images of node 4 leaves were cropped into single files. They were rotated to make every leaf horizontal and oriented in the same direction. Images were then cropped to the leaf borders, their size reduced to ¼ of the originals. Each leaf was finally centred in a 400x400 pixels image using Adobe Photoshop 7.0 (Adobe Systems Inc.). Final alignment was made by manually rotating each image such that the mid-vein was horizontal and each image was automatically centered to the centroid of the coordinate values.

A leaf point model was created by dotting nineteen points around the leaf silhouette (Fig 2). Primary points (black points on Fig 2) were at the leaf attachment point, the distal limits of the pedicel, the maximum lamina width points and the leaf tip. Secondary points (circles on Fig 2) were equally spaced between the primary points (the pmplace routine helps by automatically sliding the points long a cubic spline fitted to the points). The positions of n points for each leaf ([X_j, Y_j], j=1,…,n) were manually selected using “pmplace” function of the ‘Shape model toolbox’ that automates the rest of following process. The points for each leaf are saved in separate files each containing 2n data values. The mean shape is calculated from M leaves, , and the mean and likewise for Y (ignoring the distinction between primary and secondary points). Differences between shapes associated with differing species are reflected in the way leaf shapes differ from the mean. This is captured by subtracting the mean from each point , notice that the X and Y differences are concatenated into a single data vector forming a column of 2N values. D is a 2n column by M row data matrix where each row represents a leaf and each column a set of measurements.

Unfortunately, the measurements are correlated, i.e. if one compares a wide leaf with a narrow one, adjacent points tend to differ in similar ways. In other words, the measurements do not provide a compact description of shape. To find a compact linear description of shape we can construct the smallest set of linearly independent vectors that span the space of interest. To find independent (orthogonal) measures of shape, the differences for each image, D_i, are represented as a linear combination of orthogonal principal components where p_l is the l^th principal component and b_i,l is a weight. Thus each leaf shape, i, has a vector of weights b_i. To the extent that D can be represented linearly in this way (there may be underlying non-linearities) the weights, b_i,j, associated with leaf i are j independent measures of shape that can substitute for D_i,j.

Principal component analysis (PCA) is used to find P where P=[p₀ p₁… p_2n-1] , such that where the superscript tick, (.)’, denotes the transpose (a capital T is an alternative). The components are ordered to account for decreasing variance and it is found that a good representation of the shape of leaf, i, can be made using the weights of just the first three components [b_i,0 b_i,1 b_i,2]. These account for most of the variance of shape about the mean shape. The estimated shape, , corresponding to just these components, , can be found from , Figure 3. P is called the Point Distribution Model (PDM) and it is obtained from D in Matlab by finding the covariance matrix, C=cov(D), the eigenvectors (E), where [E,V]=eig(C), and by sorting E by decreasing importance according to the eigenvalues, V (the covariance of E). Thus, [vals, I]=sort(diag(V), ‘descend’) and the PDM is the sorted eigenvalues, P=E(:,I). Fo find b from D in Matlab use and conversely Since the original images are not normalised by size (area) the PDM captures variations in size along with shape and is, therefore, a model of leaf allometry.

Formal version

A leaf point model was created by dotting nineteen points (for clarity we will refer to these points as landmarks throughout the rest of the paper) around M leaf silhouettes (Fig 2). Primary points, N_p, (black points on Fig 2) were placed at the leaf attachment point, the distal limits of the pedicel, the maximum lamina width points and the leaf tip. Secondary points, N_s, (circles on Fig 2) were equally spaced between the primary points. The positions of N= N_p+ N_spoints for each leaf (,) were manually selected using the “pmplace” function of the ‘Shape model toolbox’ that automates the rest of following process. Each leaf is associated with a condensed landmark vector given by, where denotes the vector/matrix transpose operator and the set of real numbers. Minimizing the sum of squared deviations via procrustes analysis, involving translation, rotation and scaling terms, we can calculate the mean shape leaf shape, ,, from the M set of landmark points. It is important to note that during this process there is no distinction made between primary and secondary points.

Differences between shapes associated with differing species are reflected in the way leaf shapes differ from the mean. This deviation from the mean can be calculated for each of the M leaves considered by subtracting the mean shape from each of the leaf shapes. If we now let denote the j^th landmark point for leaf i, and the deviation from the mean for leaf i, we can write this deviation in vector notation as, and again notice that we have condensed this into a column vector using the method described above.

Often the landmark measurements are correlated and therefore do not provide a compact description of shape. To find a more compact linear description of shape, we can construct the smallest set of linearly independent vectors that span the space of interest. It is our desire not to only have a set linearly independent basis vectors, but to find a set of orthonormal basis vectors. To find these independent (orthonormal) measures of shape, we perform principle component analysis (PCA) on the landmark data. For now we assume that we have constructed a set of orthogonal basis vectors, called principle components, denoted by. We can represent the differences for each image, as a linear combination of these orthogonal principal components aswhere is the l^th principal component and b_i,l is the l^th weight for shape i. In this way each leaf shape i, has a vector of weights given by a vector. We can represent the principle components in matrix form to give. We can now re-write the deviation for each shape i as. Since the columns of are orthonormal we have where denotes the identity matrix.

To find a suitable matrix we perform PCA on the landmark data by calculating an estimate of the covariance matrix as

The principle axes of the 2N^th dimensional point cloud are now given as the eigenvectors of. If we denote the i^th eigenvector as, then the following identity is true,

The eigenvalues and corresponding eigenvectors are ordered to account for decreasing variance and it is found that a good representation of the shape of leaf, i, can be made using the just the first three principle components,, and . These account for most of the variance of shape about the mean shape and is achieved by setting the corresponding weights for the other principle components to zero. The estimated shape, corresponding to just these components, can be found from.

P is called the Point Distribution Model (PDM) and it is obtained from in Matlab as follows:

Since the original images are not normalised by size (area) the PDM captures variations in size along with shape and is, therefore, a model of leaf allometry.

Contact Information:

Name:	Dr. A. I. Hanna
Address:	School of Computing Sciences, The University of East Anglia, Norwich, Norfolk, NR4 7TJ.
Telephone:	+44 (0) 1603 592 442
E-mail:	aih@cmp.uea.ac.uk