Computing Dense Optical Flow with OpenCV's Farnebäck Algorithm: A Practical Guide

Problem

You need a dense, per‑pixel motion field between two consecutive video frames — for example, to drive a motion‑based UI, stabilize footage, or feed a downstream tracker. OpenCV’s calcOpticalFlowFarneback implements the Farnebäck polynomial expansion method, which delivers a full‑resolution flow vector field without requiring feature detection. This guide shows how to set it up, run it, verify the output, and handle the most common failure modes.

Prerequisites

• Python 3.8+ with pip (or conda) available.
• OpenCV built with the cv2 module (pip install opencv-python or conda install -c conda-forge opencv).
• NumPy for array handling (pip install numpy).
• A pair of consecutive frames — either read from a video file (e.g., OpenCV’s sample vtest.avi) or captured from a webcam.

Run the install commands in a terminal with internet access; no elevated privileges are required unless you install into a system‑wide Python environment.

Step‑by‑Step Procedure

1. Load and Pre‑process Frames

import cv2
import numpy as np

cap = cv2.VideoCapture('vtest.avi')          # replace with 0 for webcam
ret, frame1 = cap.read()
ret, frame2 = cap.read()
if not ret:
    raise RuntimeError('Could not read two frames')

# Convert to single‑channel grayscale (required by Farnebäck)
gray1 = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(frame2, cv2.COLOR_BGR2GRAY)

Both gray1 and gray2 must have identical dimensions and be 8‑bit unsigned.

2. Compute the Dense Flow Field

flow = cv2.calcOpticalFlowFarneback(
    prev=gray1,
    next=gray2,
    flow=None,               # allocate new array
    pyr_scale=0.5,           # image pyramid scale (<1)
    levels=3,                # number of pyramid layers
    winsize=15,              # averaging window size
    iterations=3,            # iterations per level
    poly_n=5,                # size of pixel neighbourhood
    poly_sigma=1.2,          # Gaussian sigma for polynomial expansion
    flags=0                  # optional flags (e.g., cv2.OPTFLOW_FARNEBACK_GAUSSIAN)
)
# flow.shape == (height, width, 2)  ->  flow[...,0] = dx, flow[...,1] = dy

Typical defaults (shown above) work for 30 fps, 640×480 footage with modest motion. If your frame rate is higher or motion larger, consider decreasing pyr_scale (e.g., 0.3) and increasing winsize.

3. Visualise the Flow (HSV Colour Wheel)

# Convert Cartesian vectors to magnitude & angle
mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1], angleInDegrees=True)

# Build HSV image: hue = direction, value = magnitude (scaled)
hsv = np.zeros_like(frame1, dtype=np.uint8)
hsv[...,0] = (ang / 2).astype(np.uint8)                 # 0‑179° → 0‑179
# Normalise magnitude to 0‑255 for visibility
hsv[...,2] = cv2.normalize(mag, None, 0, 255, cv2.NORM_MINMAX, dtype=cv2.CV_8U)
hsv[...,1] = 255                                        # full saturation

bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
cv2.imshow('Farnebäck dense flow', bgr)
cv2.waitKey(0)
cv2.destroyAllWindows()

The hue encodes motion direction (red = right, cyan = left, etc.), while brightness reflects speed. Pixels with near‑zero magnitude appear black.

4. Validate the Output

1. Shape check: assert flow.shape[:2] == gray1.shape.
2. NaN/Inf guard: assert not np.any(np.isnan(flow)) and not np.any(np.isinf(flow)).
3. Magnitude sanity: compute mag.mean(); values far above the expected pixel displacement (e.g., > 30 px for 30 fps) indicate parameter mismatch or illumination change.
4. Ground‑truth comparison (optional): on the Middlebury “Yosemite” sequence, tuned parameters typically yield an average endpoint error (AEE) < 1.0 px. Use np.linalg.norm(flow - gt_flow, axis=2).mean() if ground truth is available.

Performance Tuning & GPU Alternative

On a CPU‑only OpenCV build the function can consume 30‑80 ms per 640×480 pair. Options:

• Down‑scale frames to 320×240 before calling calcOpticalFlowFarneback; upscale the flow with cv2.resize(flow, (w, h), interpolation=cv2.INTER_LINEAR) * scale_factor.
• Use the CUDA module if you have an NVIDIA GPU and OpenCV compiled with CUDA: gpu_flow = cv2.cuda_FarnebackOpticalFlow.create(...); gpu_flow.calc(gpu_prev, gpu_next, None). This often reaches real‑time (> 30 fps) at full HD.
• Reduce pyramid levels (levels=2) for speed at the cost of large‑motion capture.

Limitations & Common Failure Modes

• Illumination changes (auto‑exposure, shadows) break the brightness constancy assumption → noisy vectors. Mitigate by applying histogram equalisation (cv2.equalizeHist) or using a robust pre‑filter.
• Large inter‑frame displacement (> winsize) causes aliasing. Increase winsize or add pyramid levels.
• Textureless regions produce unreliable flow; mask them out using the magnitude threshold (e.g., mag < 0.5).
• Non‑rigid motion (deformable objects) is handled, but the polynomial model assumes locally smooth motion; extremely turbulent flow may need a different method (e.g., DeepFlow).

Recovery / Rollback

The computation is stateless — no persistent state is altered. If the flow looks wrong, simply adjust parameters and re‑run. The only mutable resource is the video capture; call cap.release() when done.

Quick Checklist