Optical flow or optic flow is the pattern of apparent motion

Motion (physics)

In physics, motion is a change in position of an object with respect to time. Change in action is the result of an unbalanced force. Motion is typically described in terms of velocity, acceleration, displacement and time . An object's velocity cannot change unless it is acted upon by a force, as...

 of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye

Human eye

The human eye is an organ which reacts to light for several purposes. As a conscious sense organ, the eye allows vision. Rod and cone cells in the retina allow conscious light perception and vision including color differentiation and the perception of depth...

 or a camera

Camera

A camera is a device that records and stores images. These images may be still photographs or moving images such as videos or movies. The term camera comes from the camera obscura , an early mechanism for projecting images...

) and the scene. The concept of optical flow was first studied in the 1940s and ultimately published by American psychologist James J. Gibson as part of his theory of affordance

Affordance

An affordance is a quality of an object, or an environment, which allows an individual to perform an action. For example, a knob affords twisting, and perhaps pushing, while a cord affords pulling...

. Optical flow techniques such as motion detection, object segmentation, time-to-collision and focus of expansion calculations, motion compensated encoding, and stereo disparity measurement utilize this motion of the objects' surfaces and edges.

Estimation of the optical flow

Sequences of ordered images allow the estimation of motion as either instantaneous image velocities or discrete image displacements. Fleet and Weiss provide a tutorial introduction to gradient based optical flow .
John L. Barron, David J. Fleet, and Steven Beauchemin provide a performance analysis of a number of optical flow techniques. It emphasizes the accuracy and density of measurements.

The optical flow methods try to calculate the motion between two image frames which are taken at times t and at every voxel

Voxel

A voxel is a volume element, representing a value on a regular grid in three dimensional space. This is analogous to a pixel, which represents 2D image data in a bitmap...

 position. These methods are called differential since they are based on local Taylor series

Taylor series

In mathematics, a Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point....

 approximations of the image signal; that is, they use partial derivatives with respect to the spatial and temporal coordinates.

For a 2D+t dimensional case (3D or n-D cases are similar) a voxel at location with intensity will have moved by , and between the two image frames, and the following image constraint equation can be given:

Assuming the movement to be small, the image constraint at with Taylor series can be developed to get:H.O.T.
From these equations it follows that:
or
which results in
where are the and components of the velocity or optical flow of and , and are the derivatives of the image at in the corresponding directions. , and can be written for the derivatives in the following.

Thus:
or
This is an equation in two unknowns and cannot be solved as such. This is known as the aperture problem of the optical flow algorithms. To find the optical flow another set of equations is needed, given by some additional constraint. All optical flow methods introduce additional conditions for estimating the actual flow.

Methods for determining optical flow

• Phase correlation
  Phase correlation
  In image processing, phase correlation is a method of image registration, and uses a fast frequency-domain approach to estimate the relative translative offset between two similar images.- Example :...
  
   – inverse of normalized cross-power spectrum
• Block-based methods – minimizing sum of squared differences or sum of absolute differences
  Sum of absolute differences
  Sum of absolute differences is a widely used, extremely simple algorithm for measuring the similarity between image blocks. It works by taking the absolute difference between each pixel in the original block and the corresponding pixel in the block being used for comparison...
  
  , or maximizing normalized cross-correlation
  Cross-correlation
  In signal processing, cross-correlation is a measure of similarity of two waveforms as a function of a time-lag applied to one of them. This is also known as a sliding dot product or sliding inner-product. It is commonly used for searching a long-duration signal for a shorter, known feature...
  
  
• Differential methods of estimating optical flow, based on partial derivatives of the image signal and/or the sought flow field and higher-order partial derivatives, such as:
  • Lucas–Kanade method – regarding image patches and an affine model for the flow field
  • Horn–Schunck method – optimizing a functional based on residuals from the brightness constancy constraint, and a particular regularization term expressing the expected smoothness of the flow field
  • Buxton–Buxton method – based on a model of the motion of edges in image sequences
  • Black–Jepson method – coarse optical flow via correlation
  • General variational methods – a range of modifications/extensions of Horn–Schunck, using other data terms and other smoothness terms.
• Discrete optimization methods – the search space is quantized, and then image matching is addressed through label assignment at every pixel, such that the corresponding deformation minimizes the distance between the source and the target image. The optimal solution is often recovered through min-cut max-flow algorithms, linear programming or belief propagation methods.

Uses of optical flow

Motion estimation

Motion estimation

Motion estimation is the process of determining motion vectors that describe the transformation from one 2D image to another; usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion is in three dimensions but the images are a projection of the 3D scene onto a 2D...

 and video compression have developed as a major aspect of optical flow research. While the optical flow field is superficially similar to a dense motion field derived from the techniques of motion estimation, optical flow is the study of not only the determination of the optical flow field itself, but also of its use in estimating the three-dimensional nature and structure of the scene, as well as the 3D motion of objects and the observer relative to the scene.

Optical flow was used by robotics researchers in many areas such as: object detection and tracking, image dominant
plane extraction, movement detection, robot navigation and visual odometry

Visual odometry

In robotics and computer vision, visual odometry is the process of determining the position and orientation of a robot by analyzing the associated camera images...

. Optical flow information has been recognized as being useful for controlling micro air vehicles.

The application of optical flow includes the problem of inferring not only the motion of the observer and objects in the scene, but also the structure

Structure

Structure is a fundamental, tangible or intangible notion referring to the recognition, observation, nature, and permanence of patterns and relationships of entities. This notion may itself be an object, such as a built structure, or an attribute, such as the structure of society...

 of objects and the environment. Since awareness of motion and the generation of mental maps of the structure of our environment are critical components of animal (and human) vision

Visual perception

Visual perception is the ability to interpret information and surroundings from the effects of visible light reaching the eye. The resulting perception is also known as eyesight, sight, or vision...

, the conversion of this innate ability to a computer capability is similarly crucial in the field of machine vision

Machine vision

Machine vision is the process of applying a range of technologies and methods to provide imaging-based automatic inspection, process control and robot guidance in industrial applications. While the scope of MV is broad and a comprehensive definition is difficult to distil, a "generally accepted...

.

Consider a five-frame clip of a ball moving from the bottom left of a field of vision, to the top right. Motion estimation techniques can determine that on a two dimensional plane the ball is moving up and to the right and vectors describing this motion can be extracted from the sequence of frames. For the purposes of video compression (e.g., MPEG), the sequence is now described as well as it needs to be. However, in the field of machine vision, the question of whether the ball is moving to the right or if the observer is moving to the left is unknowable yet critical information. Not even if a static, patterned background were present in the five frames, could we confidently state that the ball was moving to the right, because the pattern might have an infinite distance to the observer.

External links

• Finding Optic Flow
• Art of Optical Flow article on fxguide.com (using optical flow in Visual Effects)
• Optical flow evaluation and ground truth sequences.
• Middlebury Optical flow evaluation and ground truth sequences.
• mrf-registration.net - Optical flow estimation through MRF
• The French Aerospace Lab : GPU implementation of a Lucas-Kanade based optical flow
• CUDA Implementation by CUVI (CUDA Vision & Imaging Library)