FastTrack and FaceBlur Open Source Motion Tracking, Capture

• realtime HD video stream editing, object tracking
• marker, markerless motion capture (mocap)
• flock, team tracking, multiple teams
• position reporting through application messages
• counting, measuring, 3D positioning, scene

Graphical User Interface Generator

Our other projec, Master Control generates graphical user interfaces (GUI controls) for FastTrack, as well as all other 250+ GStreamer plugins. Mix one or more file sources, video test patterns, cameras, or filters, and at least one destination or sink element. Hit play and tweak sliders in realtime. Enjoy!

FastTrack

The FastTrack element tracks and optionally marks, or cloaks objects in live and pre-recorded video feeds. A threshold value may be adjusted to eliminate false signals. The algorithm gives more weight to color, especialy skin tones, and pays less attention to shading and bad lighting.

To track with Master Control. use the editor to insert one or more track elements in the pipeline, and select the track tab. Click one of the color buttons and select a color from the video with the eyedropper tool. Go to the View menu and view element position messages. Master Control is the ultimate multimedia control center and tester.

Track is and implemented as a GStreamer plugin element, written in C, which allows it to interface directly with hardware buffers, and operate seamlessly with other applications on the multimedia pipeline. Sparse sampling saves even more time by skipping large, empty areas. Track can handle HD video at high frame rates.

FaceBlur

FaceBlur is included in the git version available below. Redact, obscure or remove faces, license plates, and other sensitive information in large batches of video, and even live video feeds!

But wait, there's more! During each frame, if the message property is TRUE, track emits an element message with the count, position, and dimensions of each detected object. GStreamer C projects, Python applications, such as Master Control, or Blender scripts, may intercept these messages and do things with them.

What kinds of things can applications do with Track? Sounding an alarm when a missing hiker is found in gigabytes of drone footage is just one potential application. Tracking, image and video stabilization, motion sensing, quality control, and gesture recognition are just a few more. Imagine what the inventor of the touch screen must have thought. The ideas are limitless!

Track's algorithm can track, count, and measure teams of players. Track can follow blue players with black and orange markings, while ignoring orange players with white markings. Set the object's (player's) background color to blue and the foreground colors to black, and orange. Want to track the other team, too? No problem! Multiple instances of track may be added to the pipeline.

The latest updates contain Master Control scripts for FaceBlur, FastTrack, and zoom-to-object. New effects include outline, recolorize, and edge detection.

Limitations

Detected objects are initially counted and sorted in top-down and left-to-right order. Track will attempt to keep the same tracking numbers assigned to the same objects. Numbers are re-assigned if objects become obscured or leave the frame. Higher-level applications will generally have their own methods and preferences for counting, or sorting out which object is which.

FastTrack does not, and probably shoud not, perform any anti-aliasing, or filtering; the threshold may need fine adjustment for particularly grainy videos. Video denoise filters may be added to the pipeline, if necessary.

Obtaining FastTrack, FaceBlur

FastTrack and FaceBlur come with the Kroll Video Effects plug-ins on GitHub. The code is under heavy revision. The latest version may be obtained using git: git clone https://github.com/themanyone/video_effects

I have limited development time. Want to see a Windows binary? A friendly installer? More options? New plugins? Something else? Vote with your dollars.

Support further development
Windows Version Linux Version More Plugins Something Else

Developer's Notes

FastTrack and effects are built against the newer, faster VideoFilter2 (VF2) in GStreamer's gst-plugins-bad. I couldn't find any docs on VF2, but it had the best performance of all the video filters. My impression, after reviewing the source and testing, is that VF2 does less memory copying and interfaces directly with hardware buffers. Apparently it's now part of Gstreamer 1.0, and we will be working on porting our stuff over to 1.0 in the coming months.

Windows builds of the GStreamer development environment, and plugins are available from ossbuild. Kroll Video Effects are designed to be cross-platform. They haven't been tested on Windows yet (I don't have a Windows box at the moment), but they should work.

Rename the video_effects folder to hkeffects. Install the resulting hkeffects folder to plugins-bad/gst/ and built it according to the ossbuild instructions for building plugins on Windows.

Linux build instructions follow. On Fedora, for example, install these dependencies.

yumdownloader --source gstreamer-plugins-bad-free
su -c 'yum-builddep gstreamer-plugins-bad-free'

While the build dependencies are downloading, install the source RPM and unpack the sources.

rpm -ivh gstreamer-plugins-bad-free*
cd rpmbuild/SPECS/
rpmbuild -bp gstreamer-plugins-bad-free.spec

Get FastTrack from github, if not already downloaded:

git clone https://github.com/themanyone/video_effects

Rename the folder created by git, the video_effects folder, to hkeffects.

mv video_effects hkeffects

Copy the newly made hkeffects folder to rpmbuild/BUILD/gst-plugins-bad*

cp -r hkeffects rpmbuild/BUILD/gst-plugins-bad*/gst/

Patch configure.ac and run autogen.sh to generate Makefiles

cp hkeffects/configure.ac.patch rpmbuild/BUILD/gst-plugins-bad*/
cd rpmbuild/BUILD/gst-plugins-bad*/
patch -p0 < configure.ac.patch
./autogen.sh

This is fine for someone just wanting to try it out. On the other hand, since programs under development, like pitivi, work best with the latest Gstreamer version, advanced programmers may wish to build GStreamer from git and, to keep the old system intact, build and test against the new gstreamer without permanantly installing it.

Compile. Installers may elect to re-run configure using the correct options (copied from Fedora's gstreamer SPEC file). I don't ever bother with this step, but it should work...

./configure \
    --with-package-name="Fedora gstreamer-plugins-bad package" \
    --with-package-origin="http://download.fedora.redhat.com/fedora" \
    --enable-debug --disable-static --enable-gtk-doc --enable-experimental \
    --disable-divx --disable-dts --disable-faac --disable-faad --disable-nas \
    --disable-mimic --disable-libmms --disable-mpeg2enc --disable-mplex \
    --disable-neon --disable-openal --disable-rtmp --disable-xvid

Now run make to build the hkeffects plugins.

cd gst/hkeffects
make

Those who ran configure with the correct options may safely run su -c 'make install'. Otherwise, manually copy the plugins, .libs/libgsthkeffects.so to wherever the gstreamer-plugins go, usually /usr/lib64/gstreamer-0.10/ on a 64-bit Fedora system.

Take a look at the test script before trying it out.

gedit test-fedora

Earlier Incarnations

My first attempt at motion tracking, Pop Can Tracking, builds upon the free ffmpeg-libs. It works off-line, reading and writing .avi files. It does not have direct access to the multimedia system like FastTrack does.

I put a lot of work into it, and then it sat around for a few years on a crashed hard drive. Eventually, I was able to recover an earlier version from backup and fix it up. On last year's compilation attempt it crashed due to a bug in ffmpeg-libs (not my fault). Hopefully they fixed it by now.

Unlike Team Tracking which measures the area of multi-colored objects in different directions, Pop Can counts the number of nearby pixels (hard-coded 50 pixel radius) that resemble a single target color. The center of detection moves zombie-like toward each detected pixel, only less so the more pixels it collects. The markers always center themselves upon the densest area of matching color, even if other colors are present. I described the algorithm as a bastardization of bayesian probability calculus with Bhattacharyya coefficients.

---