Optical Flow in DX11

Optical Flow is an algorithm that detects movement in images.Its use ranges from drone positioning to medicine and interactive installations.
This project presents a solution for analysing and processing images four times faster than our eye could perform, with an opensource library written in shaders.

This shows a set of videos analysed by this Optical Flow algorithm.


This project was developed as my bachelor thesis for my bachelor in engineering. It consists of an implementation of a Computer Vision algorithm to detect movement in images.

WHY?

For this thesis, I wanted to show and measure the benefits of GPU programming (code running in the Graphics card) vs. CPU programming (code running in the Central Processing Unit) in the field of image processing, in particular Computer Vision.

If we want to process and detect the movement in images in real time with the CPU, we should be using low resolution images like the one on the left, with the consequent decrease in the quality and level of detail in the results. In the right, the image which could be analysed by the GPU, as you see, high res!

PROJECT

Computer Vision studies the analysis of digital images and videos. The corresponding research field offers an increasing range of possibilities in various areas such as image processing or artificial intelligence.

This thesis proposes a GPU implementation of Optical Flow to discuss the benefits of the use of parallel architectures (GPU) for image processing. The corresponding processing time is fast enough to process live feeds from high speed cameras and post process the resulting data efficiently raising the quality of the prediction. Machine learning, drone positioning or image stabilization are some of its common applications.

USE CASES

Optical Flow has many possible fields of application, such as motion estimation, object tracking or machine vision.
Motion estimation is the process of determining movement from one image to another, used in some video compression techniques like MPEG or DCT. Security cameras or surveillance systems do benefit from the information provided by this algorithm. Drone positioning and machine vision. The quality and speed of the Optical Flow computation is important in terms of reaction speed and overall performance. In the field of medicine and image registration, Optical Flow is used creating visual representations of motion and evolution of organs.

ENGINEERING

This thesis is focused on two main solutions for Optical Flow analysis. Our CPU based methods are written in C++, while the GPU algorithms are written on the GPU language HLSL. The development of the described methods allowed us to evaluate the performance of the algorithms in both processors. Considering different resolutions of images that provide different level of detail, the time of processing changed significantly.

The studied algorythms and the developed codes are based on the original papers and descriptions of Horn and Shunck method and Lucas and Kanade solutions

These are the graphics I got, when comparing the performance of the algorithm in the CPU and the GPU. As you see, the time of processing gets reduced more than 10 times in high resolution images,we can get a 50% decrease of the error in the motion estimation in the GPU due to its computing power.

INSIGHTS

Studying the insights of image processing and computer vision gave me a good understanding of their intricacies, (convolution matrixes, and procedures to get blurs, sharpening, or other image filters).

This site helped me a lot to understand image convolution, as it provides a very visual understanding of the different kernels and how they affect images: (http://setosa.io/ev/image-kernels/).

The library, obtains the Optical Flow upon three general steps:

1. COMPARE ONE FRAME WITH THE NEXT ONE

This motion estimation works by comparing consecutive frames of a camera feed.

2. FIND THE DERIVATIVES OF THE IMAGE - WHAT???

This means, finding the edges of the elements in the image in the right and top. The red channel gets the edges in the X axis, the green channel gets the edges in the Y axis, and the blue channel stores the difference between consecutive frames.

3. CALCULATE THE OPTICAL FLOW

With a series of calculations based on Lucas Kanade and Horn and Shunck, the result is an image, where red channel gives the motion in X axis, and green values the motion in Y. When reading those values and plotting them in lines, we see the motion as on the right.

LINKS

Read the abstract: Optical Flow for GPU
Source download : https://vvvv.org/contribution/opticalflow-dx11-for-real
Developed at Waltz Binaire

DETAILS

YEAR
2017

CATEGORY
Computer Vision, motion analysis

COMPANY
Waltz Binaire

Back to projects