InteractLabeler 1.0
documentation

by Gabriel Brostow and Julien Fauqueur,
{gjb47,jf330}@cam.ac.uk}
Cambridge University, october 2006


table of content:

1. INTRODUCTION
2. PARAMETERS, PROGRAM AND GUI
3. ANNOTATION
4. SHORTKEYS
5. REFERENCES

1. INTRODUCTION

InteractLabeler is a program to perform region annotation of video frames (or image sequence). As the user browses consecutive frames of a given video, InteractLabeler displays the result of automatic region segmentations of each frame, that he uses to label each region with a list of various annotation keywords. Using the mouse, he can group regions into objects of interest (such as road, pedestrian,...). Each resulting labeled frame is saved as an image in which colors correspond to semantic labels.

2. PARAMETERS, PROGRAM AND GUI

2.1 Setting the environment variables

In text file "InteractLabeler.meta", the following variables shall be set:
  • FILE_TIMELINE : full path and filename of the input video or image sequence to annotate:
                            if input is a video, then specify the AVI filename, e.g. "FILE_TIMELINE F:\Data\ToyTest02.avi"
                            if input is an image sequence, then specify a text file containing the list of image filenames
  • DIR_LABEL_IMGS : full path to the directory to store the output labeled images
                            e.g. "DIR_LABEL_IMGS F:\Data\labeledimages"
  • FILE_LABEL_NAMES : full path and filename of the text file that contains the list of keywords
                            e.g. "FILE_LABEL_NAMES LabelNames01.txt"
  • MAX_WINDOW_DIM : maximum height and width of GUI windows
                            e.g. "MAX_WINDOW_DIM 1000"

2.2 Running the program

The program windows binary is called InteractLabeler.exe and shall be run with the DOS console command as follows:
     InteractLabeler.exe InteractLabeler.meta
where InteractLabeler.meta is the name of the parameter file (see above).

2.3 Graphical User Interface

While annotating each frame, the program always displays the four following windows:

window 1: original image + segmentation controls window 2: map of edges + labeled regions window 3: segmented image + currently focused region window 4: console (information display)

Pressing the CTRL key while moving the mouse will display the list of labels in windows 1 and 3 (see figure)
label list



3. ANNOTATION

3.1 Running the segmentation

When viewing a frame, pressing the "g" key will run the 3 region segmentation algorithms.

Automatic segmentation provides a natural initial step to achieve region grouping and labelling. An automatic segmentation groups pixels which have similar color and texture properties. Since they do not always correspond to "objects" (e.g. a car is a composite of visually different regions), the user can then manually group those regions into semantic ones and then label them.

InteractLabeler uses 3 different segmentation algorithms to give the user more choices in the regions to group (MeanShift [1], Felzenszwalb & Huttenlocher [2], Pyramids [3]). Note that the MeanShift algorithm has 3 modes (from fine to coarse: Quantisation (Q), Oversegmentation (O), Undersegmentation(U)). There is also a manual grouping mode (N). The choice between the various segmentations should be made by visual inspection, depending on the object to label. For example, some algorithms produce more (e.g. mode (Q)) or less regions (e.g. mode (U)). The manual mode can be used to refine the grouping of difficult regions.

By sliding the "N,P,Q,O,U,F" sliderbar, one can switch between the 6 segmentation modes graded from left to right:
grouping mode description
NULL (N) manual painting
Pyramid (P) pyramid segmentation
Quantisation (Q) Mean shift segmentation in "quantisation" mode
Oversegmentation (O) Mean shift segmentation in "oversegmentation" mode
Undersegmentation (U) Mean shift segmentation in "undersegmentation" mode
Felzenszwalb (F) Felzenszwalb & Huttenlocher segmentation

The sliders "PyrSeg1" and "PyrSeg2" are the parameters of Pyramid segmentation. "PyrSeg1" is the error threshold for establishing the links, and "PyrSeg2" is the error threshold for the segments clustering.

The result of the segmentation is always shown in window 3.

3.2 Annotating regions

To annotate a region, first, a label must be chosen from the label list. This list is displayed as long as the CTRL key is pressed. When releasing the key, the selected label will be the last one focused by the mouse. In window 2, when the mouse hovers over a region, the same region is highlighted in red on the segmented image in window 3. Alternatively, a label can be selected by using its corresponding shortcut (shown as a symbol in parentheses in the label list). These shortcuts can be modified in file "LabelNames01.txt". Finally, the third way to pick a label is the following: when right-clicking over a region in window 2, its label becomes the current label.

To apply the current label to the current region, simply press the left mouse button. To expand the label over multiple contiguous regions, drag the mouse over the regions. The resulting labeled regions will be shown in window 2 as regions color-filled with the corresponding label color.

More precisely, the behaviour of the mouse click is the following: the mouse cursor is composed of a green circle, the "paintbrush". When clicking the mouse, actually all regions that intersect with the green circle will be labeled. This allows to speed up the grouping process. The diameter of the paintbrush can be ajusted by using the mousewheel, depending on the required granularity.

When in manual mode (N), the paintbrush can be used to freely draw regions without the help of segmented regions. The paintbrush diameter is still adjustable with the mousewheel.

See 4. SHORTKEYS for the list of available shortcuts while using strokes to group regions.

4. SHORTKEYS

grouping/annotatingg run segmentations on current frame
c commit changes and save current labeled frame
CTRL switch to label list display
CTRL z undo
CTRL y redo
CTRL d clear
mousewheel changes the paintbrush diameter
right-click picks the label of the focused region
[user-defined number 0-9] picks the corresponding label from the list
video navigation[SPACE] go one frame forward
[BACKSPACE] go one frame backwards
+ go 50* frames forward (implicitly also saves, like "c")
- go 50* frames backwards (implicitly also saves, like "c")
*: this number can be changed in the meta file (see variable FILE_TIMELINE_JUMPBY).

5. REFERENCES

[1]D. Comaniciu, P. Meer, "Robust Analysis of Feature Spaces: Color Image Segmentation", IEEE Conf. Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, 750-755, 1997
[2]P. F. Felzenszwalb and D. P. Huttenlocher, "Efficient Graph-Based Image Segmentation", International Journal of Computer Vision, Volume 59, Number 2, September 2004.
[3]OpenCV library.

[31.10.2006]