-------------------------------------------- -------------------------------------------- Video Spot Tracker Evaluation Project Manual -------------------------------------------- -------------------------------------------- Created by: Tywandre Harris Modifed by: Russell M. Taylor II, Patrick Hahn ----------------- Table of Contents ----------------- I. Necessary Software II. What is evaluated? III. Layout IV. sequences V. sequence_src VI. How to evaluate performance VII. How to run the scripts --------------------- I. Necessary Software --------------------- For basic performance tests and auto-find tests: -Cygwin with all necessary packages, packages containing 'make utility', tcsh shell, etc., (download from cygwin website). You need to have the ImageMagick "convert" utility available. -Video Spot Tracker (version 05.08 or later) (nsrg installers directory) -airy maker (version 01.06) (nsrg installers directory) -NSRG Runtime and Buildtime (nsrg installers directory) -Spot Tracker testing tools (includes: make_moving_spot_video, add_noise_to_image, check_found_spots, make_multi_spot_test and compare_tracking_files programs) (nsrg installers directory) Additional software needed for tests comparing VST with radial symmetry tracking software: - MATLAB ---------------------- II. What is evaluated? ---------------------- There are general types of tests: One group performance of the program, accuracy and speed (estimates/second) against various conditions including noise, beadsize, etc. The second group tests the performance of the auto-find feature under various conditions including noise, non-uniform backgrounds, and blur settings. The names of these scripts begins with "autofind" The third group tests the accuracy of the Video Spot Tracker against other algorithms, two of which are implemented using a particle tracking program created by Raghuveer Parthasarathy from University of Oregon, which uses an algorithm based on radial symmetry. The names of these scripts begin with "radialsym" Accuracy -------- 1)mean radial error: the average distance between the true position of the bead and the estimated position of the bead by the Video Spot Tracker -the radial errors are found by using the Pythagoren Theorem, radial error = hypotenuse, with the errors in "x" and "y" being the legs -the mean radial error is found by averaging radial errors and dividing by the number of frames in the sequence which is equal to the number of estimates 2)mean error x: the average distance, along the "x" axis, between the true position and estimated position of the bead 3)mean error y: the average distance, along the "y" axis, between the true position and estimated 4)max radial error: the largest distance between the true and estimated position of the bead, for all the estimates in a image sequence 5)x bias: bias of the estimates by the tracker, in terms of the "x" axis 6)y bias: bias of the estimates by the tracker, in terms of the "y" axis Speed ----- The number of estimates that the Video Spot Tracker makes per second ------------- IV. sequences ------------- 'sequences' is where the cooked images, performance measurement files, and HTML files describing performance end up. It is a shadow copy of the sequence_src directory that is built when the makefiles and scripts do their thing. This plus the "index.html" file should be copied into the web-page space. ------------- V. sequence_src ------------- 'sequence_src' contains: - each folder in sequence represents a specific kind of spot tracker test, these folders each contain various simulated image sequences for a specific test (noise test, beadsize, etc.) - VST performance evaluation main web page script 'vstmainpage.tcsh' that runs the makefiles (explained below) located in each test's directory and creates the main web page with links to each test page (containing results in tables) - after running main web page script (above), the index.html file will also appear in the sequences directory - manual document -------------------------------- VI. How to evaluate performance -------------------------------- Various factors affect the performance of the Video Spot Tracker, seperate tests where conducted for several factors - For each test (i.e. bead size vs accuracy and speed) there are a number of image sequence folders (i.e. disc_3_sym_circle.0_0 and gaussian_9_sym_circle.0_0) -each image sequence folder is identified with five values: bead type, bead size, tracker kernel, type of motion and noise level Example: 'disc_3_sym_circle.0_0' image sequence folder 'disc' is the bead type (in this case a disc) '3' is the bead size (in this case 3 pixels) 'sym' is the tracker kernel (in this case symmetric) 'circle' is the type of bead motion (in this case circular) '0_0' represents noise level (in this case none) ----------------------------- The general idea for testing the performance of simulated image sequences is to write Makefiles and running them using Cygwin's bash shell, the Makefiles: create image sequences, automatically track beads in image sequences using the Video Spot Tracker, compare image sequence's true log with VST's tracking log and puts the results in tables; there is a Makefile for each test (that is located within each test folder), the Makefiles for each test all basically follow the same template: *small notes* -in order to understand the makefile code, some general knowledge of bash shell and make (dependencies, targets, commands, etc.) scripting is necessary -the results (statistics) from each test are within the proper html tags, both the results and html tags are sent to a web page file (table.html) using the echo command 1) 'start_html' is the target with a command to add the necessary html code (html tags to start a web page) to the table.html file (web page for that test) 2) 'begin_tablex' is the target with a command to add the necessary html table code (table tags to start a table; specific headings are placed in each table cell to form the table header) to the end of the table.html file; x is a variable that increments by one as the number of tables on the web page increases (i.e. begin_table1, begin_table2) 3) begin creating image sequences: after deciding what kind of image sequences to create for a test, I created targets and named them (using method mentioned above: i.e. disc_3_sym_circle.0_0) 4) the image sequence target(s) are each followed with several commands 4a) command to create directory of the same name as the image sequence and another directory 'tiff_16bit' 4b) command to run the make_moving_spot_video program, which creates a specific image sequence of tiff images, it receives several necessary arguments like (bead type, imagesize, # of frames, motion type, etc.) (run the make_moving_spot_video program with '--help' in bash shell to learn how to successfully use the program to create a certain type of image sequence) 4c) command to move tiff images to the image sequence's tiff_16bit folder 4d) command to move csv true log file (created by the make_moving_spot_video program) to the image sequence folder's tiff_16bit folder 4e) command to copy all tiff images in the image sequence to a temporary storage folder (like C:/tmp or any temporary storage folder) 4f) command to run the Video Spot Tracker; 'time' is placed before the command to run the tracker so that the speed (est/sec) can be displayed; the tracker is given several necessary arguments for it to run a specific set of tiff files and arguments to control the settings of the tracker (like tracker placement, precision and where to put the output tracking log file (use '--help' with tracker to learn how to properly run video spot tracker with a script); the VST tracking log is sent to the temporary folder; speed (estimates/per) is sent to a foo file in the image sequence's tiff_16bit folder so that the speed can be extracted ***important note**** -the make_moving_spot_video program has a argument 'start' that lets the user place the bead in a specific position, with this known position the VST's tracker can be placed in the right location (approx. over the bead); the tracker is placed slightly off the bead to avoid "too good" results 4g) command to copy VST tracking log from temporary folder to image sequence folder's tiff_16bit folder 4h) command to remove all files in temporary folder ***very important*** 4i) command to run compare_tracking_files program; this program is given two arguments: 1) the true log csv (created by make_moving_spot_video program) and 2) the VST tracking log (created by run tracker on bead in image sequence); the program finds the mean radial error, mean x error, mean y error, max radial error, x bias and y bias and places these values in a csv file that I named 'results' and placed in the image sequence folder's tiff_16bit with everything else; the program compares the true to the estimated position of the moving bead and spits out the accuracy statistics 4j) command to extract the first speed (est/sec), recorded by the VST, from the 'foo' file (foo file mentioned earlier); this speed is sent to a text file named simply 'file'; this speed will become the official speed for the VST on the image sequence 4k) echo command to add table code (with specific values pertaining to image sequence in table cells) to the end of the table.html file 4j) command to run the table_line.bash script; this script receives four arguments; it was created to extract each individual value from the 'results' csv file (i.e. mean radial error) and to place each value in the appropriate cell in the table; it then takes the speed (est/sec) in the file.txt file and places in the appropriate cell in the table; finally, the script takes the image sequence's true log csv file and VST tracker log csv file and makes links to them in the table ---- noise test ---------- -for the noise test, the makefile code is slightly more complex -to create noisy image sequences, Suffix rules are used that automatically creates the noisy image sequences out of the original (non noisy image sequence) -as mentioned earlier, in an image sequence, like 'disc_3_sym_circle.0_0', '0_0' represents the noise level (in this case none), for noisy image sequences this value changes based on the specified noise arguments passed to the add_noise_to_image program when it is called -the add_noise_to_image program is capable of adding various amounts of noise to an image (run program using bash shell with '--help' to learn how to use) -for my image sequences, I used the add_noise_to_image program's gaussian function that receives two arguments: the mean and variance of the noise; -once the noise levels were chosen, I created significant suffixes that would automatically make the noisy image sequences out of the original image sequence; 1) all that is necessary for this to occur is for the original image sequence to already exist 2) both the original image sequence's suffix '.0_0' and noisy image sequence's suffix (i.e. '.20000_1e8' where 20000 is the mean and 1e8 is the variance of the noise) have to be significant suffixes 3) the noisy image sequence has to be a prerequisite and this tells make to create the noisy image sequence by taking the original image sequence of the same name but different suffix (ending) and carry out several commands that add noisy to the original sequence thus creating the new noisy sequence - the commands to add noise to image sequences basically creates a folder for the new image sequence, calls the make_noise.tcsh script that receives four arguments: 1) directory of the original image sequence 2) directory of the new, noisy image sequence 3) the background or mean value for the gaussian noise 4) the variance of the gaussian noise - the make_noise script passes each image the original image sequence's directory to the add_noise_to_image program (along with the two noise arguments and the output files for the new noisy image sequence) and in the end a new noisy image sequence is created in a folder of its own - the rest of the commands in the noise test makefile are basically the same (the video spot tracker tracks the beads in the new noisy image sequences and the compare_tracking_files program compares the tracking log to the true log csv file of the original image sequence and spits out a 'results' csv file -------------------- 4k) 'end_tablex' is the target with a command to add the table code, to end the current table, to the end of the table.html file; x value depends on table number (i.e. x=1,2,3) 4k) 'end_html' is the target with a command to add html ending tags to the end of the table.html file --------------------------- VII. How to run the scripts --------------------------- -In order to run the system that creates the web pages and makefiles, you must run cygwin bash shell and change the directory to sequence_src; then simply run the "vstmainpage.tcsh" script by typing "./vstmainpage.tcsh". (You need to edit the file to set VST_VERSION to match the version of spot tracker you have installed and to make sure that the VST_DIRECTORY points at the install directory.) This will run all the makefiles that do all the work. It puts all of its results into the "sequences" directory. You can delete the contents of the sequences directory at will; they can all be rebuilt from the sequence_src directory. -as mentioned earlier, the "vstmainpage.tcsh" script runs all the makefiles and contains links to each test page and to each test's directory containing all the image sequences for that test -In order to choose which specific tests to run, you can edit vstmainpage.tcsh, making the loop on line 68 only contain the names of the tests you want to run. -The tests that compare tracking algorithms use Parthasarathy's GUI program, and because it uses a GUI, each trial will require manual input. The script will open matlab and the GUI automatically for each trial, and the user should follow these steps to complete the tracking process: - Set nsize setting to that specificied by the test's readme file. You can set nsize or any other settings to different values if you want to experiment, but we use default settings with only the nsize adjusted. - If you are tracking a dark spot on a lighter background, check the "Use prvs. ctrs" box - Click the "Track all frames" button and wait for tracking to finish. - Click the "Link Tracks" button - Click the "Save TXT" button, then press "OK" - Click "Exit" to close the GUI, then close the MATLAB command line window that opened automatically (you won't need to use it) The testing scripts will only sleep for 30 to 40 seconds to allow time for this process to be completed, but if find you need more time to go through this process before the script continues, you can edit the the script's line that says "sleep 30s" to whatever amount of time you need. -after learning how the makefiles operate, it should be simple to go in and create new image sequences to test by copying commands to create image sequences and changing several values. For the auto-find testing scripts, it is important that the locations of the spots remain the same, but their properties can still be changed and the tests will yield correct results. **important notes** -for majority of tests, with an exception to the precision and sample spacing test, precision and sample spacing were set to their highest values The original scripts were written by Ty Harris (tdharris@mail.ecsu.edu) and the auto-find and radiay symmetry comparison scripts were written by Patrick Hahn (phahn@live.unc.edu). They have been modified by Russ Taylor (taylorr@cs.unc.edu), who is the point of contact for support.