Simple Interactive Object Extraction
Encyclopedia
Simple interactive object extraction (SIOX) is an algorithm for extracting foreground objects from color images and videos with very little user interaction. It has been implemented as "foreground selection" tool in the GIMP
(since version 2.3.3), as part of the tracer tool in Inkscape
(since 0.44pre3), and as function in ImageJ
and Fiji (plug-in). Experimental implementations were also reported for Blender
and Krita
. Although the algorithm was originally designed for videos, virtually all implementations use SIOX primarily for still image segmentation. In fact, it is often said to be the current de-facto standard for this task in the open-source world.
Initially, a free hand selection tool is used to specify the region of interest. It must contain all foreground objects to extract and as few background as possible. The pixels outside the region of interest form the sure background while the inner region define a superset of the foreground, i.e. the unknown region. A so-called foreground brush is then used to mark representative foreground regions. The algorithm outputs a selection mask. The selection can be refined by either adding further foreground markings or by adding background markings using the background brush.
Technically, the algorithm performs the following steps:
For video segmentation the sure background and sure foreground regions are learned from motion statistics. SIOX also features tools that allow sub-pixel accurate refinement of edges and high texture areas, the so-called "detail refinement brushes".
As with all segmentation (image processing)
algorithms, there are always pictures where the algorithm does not yield perfect results. The most critical drawback of SIOX is the color dependence. Although many photos are well-separable by color, the algorithm cannot deal with camouflage. If the foreground and background share many identical shades of similar colors, the algorithm might give a result with parts missing or incorrectly classified foreground. SIOX performs about equally well on different benchmarks compared to graph-based segmentation methods, such as Grabcut
. SIOX is, however, more noise robust and can therefore also be used for the segmentation of videos. Graph-based segmentation methods search for a minimum cut and therefore tend to not perform optimally with complex structures.
The algorithm has initially been developed at the department of computer science at Freie Universitaet Berlin. The main developer, Gerald Friedland, is now a researcher at the International Computer Science Institute
in Berkeley, California and continues to support the development through mentoring, e.g. in the Google Summer of Code
.
GIMP
GIMP is a free software raster graphics editor. It is primarily employed as an image retouching and editing tool and is freely available in versions tailored for most popular operating systems including Microsoft Windows, Apple Mac OS X, and Linux.In addition to detailed image retouching and...
(since version 2.3.3), as part of the tracer tool in Inkscape
Inkscape
Inkscape is a free software vector graphics editor, licensed under the GNU General Public License. Its goal is to implement full support for the Scalable Vector Graphics 1.1 standard....
(since 0.44pre3), and as function in ImageJ
ImageJ
ImageJ is a public domain, Java-based image processing program developed at the National Institutes of Health. ImageJ was designed with an open architecture that provides extensibility via Java plugins and recordable macros. Custom acquisition, analysis and processing plugins can be developed using...
and Fiji (plug-in). Experimental implementations were also reported for Blender
Blender (software)
Blender is a free and open-source 3D computer graphics software product used for creating animated films, visual effects, interactive 3D applications or video games. The current release version is 2.60, and was released on October 19, 2011...
and Krita
Krita
Krita is the digital painting and illustration software included based on the KDE Platform and Calligra Suite libraries. Designed as a digital painting and illustration suite, Krita is free software and distributed under GNU General Public License...
. Although the algorithm was originally designed for videos, virtually all implementations use SIOX primarily for still image segmentation. In fact, it is often said to be the current de-facto standard for this task in the open-source world.
Initially, a free hand selection tool is used to specify the region of interest. It must contain all foreground objects to extract and as few background as possible. The pixels outside the region of interest form the sure background while the inner region define a superset of the foreground, i.e. the unknown region. A so-called foreground brush is then used to mark representative foreground regions. The algorithm outputs a selection mask. The selection can be refined by either adding further foreground markings or by adding background markings using the background brush.
Technically, the algorithm performs the following steps:
- Create a set of representative colors for sure foreground and sure background, the so-called color signatures.
- Assign all image points to foreground or background by a weighted nearest neighbor search in the color signatures.
- Apply some standard image processing operations like erode, dilate, and blur to remove artifacts.
- Find the connected foreground components that are either large enough or marked by the user.
For video segmentation the sure background and sure foreground regions are learned from motion statistics. SIOX also features tools that allow sub-pixel accurate refinement of edges and high texture areas, the so-called "detail refinement brushes".
As with all segmentation (image processing)
Segmentation (image processing)
In computer vision, segmentation refers to the process of partitioning a digital image into multiple segments . The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze...
algorithms, there are always pictures where the algorithm does not yield perfect results. The most critical drawback of SIOX is the color dependence. Although many photos are well-separable by color, the algorithm cannot deal with camouflage. If the foreground and background share many identical shades of similar colors, the algorithm might give a result with parts missing or incorrectly classified foreground. SIOX performs about equally well on different benchmarks compared to graph-based segmentation methods, such as Grabcut
Grabcut
GrabCut is an image segmentation method based on graph cuts. There are several open source implementations available including OpenCV .- See also :* Connectivity * Prim's algorithm* Edmonds-Karp algorithm...
. SIOX is, however, more noise robust and can therefore also be used for the segmentation of videos. Graph-based segmentation methods search for a minimum cut and therefore tend to not perform optimally with complex structures.
The algorithm has initially been developed at the department of computer science at Freie Universitaet Berlin. The main developer, Gerald Friedland, is now a researcher at the International Computer Science Institute
International Computer Science Institute
The International Computer Science Institute is an independent, non-profit research organization located in Berkeley, California, USA. Since its founding in 1988, ICSI has maintained an affiliation with the University of California, Berkeley, where several of its members hold faculty appointments...
in Berkeley, California and continues to support the development through mentoring, e.g. in the Google Summer of Code
Google Summer of Code
The Google Summer of Code is an annual program, first held from May to August 2005, in which Google awards stipends to hundreds of students who successfully complete a requested free or open-source software coding project during the summer...
.