Local image features or interest points provide compact and abstract representations of patterns in an image. We propose to extend the notion of spatial interest points into the spatio-temporal domain and show how the resulting features often reflect interesting events that can be used for a compact representation of video data as well as for its interpretation. To detect spatio-temporal events, we build on the idea of the Harris and Forstner interest point operators and detect local structures in space-time where the image values have significant local variations in both space and time. We then estimate the spatio-temporal extents of the detected events and compute their scale-invariant spatio-temporal descriptors. Using such descriptors, we classify events and construct video representation in terms of labeled space-time points. For the problem of human motion analysis, we illustrate how the proposed method allows for detection of walking people in scenes with occlusions and dynamic backgrounds.

An inherent property of objects in the world is that they only exist as meaningful entities over certain ranges of scale. If one aims to describe the structure of unknown real-world signals, then a multi-scale representation of data is of crucial importance. This paper gives a tutorial review of a special type of multi-scale representation—linear scale-space representation—which has been developed by the computer vision community to handle image structures at different scales in a consistent manner. The basic idea is to embed the original signal into a one-parameter family of gradually smoothed signals in which the fine-scale details are successively suppressed. Under rather general conditions on the type of computations that are to be performed at the first stages of visual processing, in what can be termed ‘the visual front-end’, it can be shown that the Gaussian kernel and its derivatives are singled out as the only possible smoothing kernels. The conditions that specify the Gaussian kernel are, basically, linearity and shift invariance, combined with different ways of formalizing the notion that structures at coarse scales should correspond to simplifications of corresponding structures at fine scales-they should not be accidental phenomena created by the smoothing method. Notably, several different ways of choosing scale-space axioms give rise to the same conclusion. The output from the scale-space representation can be used for a variety of early visual tasks; operations such as feature detection, feature classification and shape computation can be expressed directly in terms of (possibly non-linear) combinations of Gaussian derivatives at multiple scales. In this sense the scale-space representation canserve as a basis for early vision. During the last few decades, a number of other approaches to multiscale representations have been developed, which are more or less related to scale-space theory, notably the theories of pyramids, wavelets and multi grid methods.Despite their qualitative differences, the increasing propularity of each of these approaches indicates that the crucial notion of scale is increasingly appreciated by the computer.vision community and by researchers in other related fields. An interesting similarity to biological vision is that the scale-space operators closely resemble receptive field profiles registered in neurophysiological studies of the mam- malian retina and visual cortex.

Scale Invariant Feature Transform (SIFT) is an image descriptor for image-based matching developed by David Lowe (1999,2004). This descriptor as well as related image descriptors are used for a large number of purposes in computer vision related to point matching between different views of a 3-D scene and view-based object recognition. The SIFT descriptor is invariant to translations, rotations and scaling transformations in the image domain and robust to moderate perspective transformations and illumination variations. Experimentally, the SIFT descriptor has been proven to be very useful in practice for robust image matching and object recognition under real-world conditions. In its original formulation, the SIFT descriptor comprised a method for detecting interest points from a grey-level image at which statistics of local gradient directions of image intensities were accumulated to give a summarizing description of the local image structures in a local neighbourhood around each interest point, with the intention that this descriptor should be used for matching corresponding interest points between different images. Later, the SIFT descriptor has also been applied at dense grids (dense SIFT) which have been shown to lead to better performance for tasks such as object categorization and texture classification. The SIFT descriptor has also been extended from grey-level to colour images and from 2-D spatial images to 2+1-D spatio-temporal video.QC 20120524