I'm working on multisensory object recognition approaches for robotics. In a joint project between the Fraunhofer Society and the Max Planck Society, funded by the Centre for Intergrative Neuroscience (CIN), we combine 2D and 3D visual information to classify unknown objects. 3D shape data is retrieved from a time-of-flight range sensor and combined with 2D appearance information retrieved from color cameras.
Since humans rely on many cues beyond visual information to represent objects we want to investigate how various modalities can be integrated to multimodal object representations. In a collaboration with the Italian Institute of Technology in Genoa we are implemening recognition algorithms on a humanoid robot using vision, haptics and proprioception.
CIN 2D+3D object classification dataset
This dataset contains segmented color and depth images of objects from 18 categories of common household and office objects. Each object was recorded using a high-resulution color camera and a time-of-flight rang sensor. Objects were rotated using a turn table and snapshots taken every 20 degrees.
Color and 3D data is stored a 3-channel PNG image each. To load a view with C++ and OpenCV use the following lines:
cv::Mat colorImage = cv::imread(FILENAME_COLOR, -1);
cv::Mat xyzImage_16U3 = cv::imread(FILENAME_XYZ, -1);
channels -= 1000.0;
channels -= 1000.0;
Humans rely on many cues beyond visual information to represent objects and interact with the environment. Already very early on we see that infants probe and explore the world using all available senses. Computational approaches to object recognition, however, often neglect these additional sources of information.
In collaboration with European robotic research institutes we investigate how various modalities can be integrated into multi-sensory, computational object perception. We are working on approaches that combine visual, proprioceptive and haptic information to form multi-modal object representations that capture the wide range of cues that is available to us.
In joint work with the Fraunhofer IPA in Stuttgart we combine two-dimensional image data with three-dimensional information retrieved from range sensing devices. Both channels are fused in order to recognize and classify unknown three-dimensional objects in real-world scenarios.
In collaboration with the Italian Institute of Technology in Genoa we are implementing active methods for perception-driven, in-hand object recognition on a humanoid robot. Besides visual input these strategies take into account proprioceptive information obtained from sensors in the robot’s body.
Robot platforms for imlementation and evaluation. Left: Care-O-bot® Fraunhofer IPA, Stuttgart, Germany. Right: iCub, Italian Institute of Technology, Genoa, Italy.
We have extensively studied the combination of two-dimensional appearance based object representations with three-dimensional shape information. Results show clearly that joining both modalities leads to a significant improvement in object classification performance .
Furthermore, on experiments with a humanoid robot, we could demonstrate that by using a multi-sensory, perception-driven exploration strategy, objects are recognized much faster, more accurately, and more robustly than by random, vision-only examination .
As important as multi-sensory integration is for human perception, we see that sensor-fusion also is a crucial requisite for the development of perceptual skills for cognitive robots. Further research needs to go in this direction to enable robotic systems to operate in a human centered environment with the flexibility and dexterity that is natural to us.
1. Browatzki B., Fischer J, Graf B., Bülthoff H.H, Wallraven C. (2011) Going into depth: Evaluating 2D and 3D cues for object classification on a new, large-scale object dataset, 1st IEEE Workshop on Consumer Depth Cameras for Computer Vision.
2. Browatzki B., Tikhanoff V., Metta G., Bülthoff H.H, Wallraven C. Active Object Recognition on a Humanoid Robot.
|since 2009||PhD student at Max Planck Institute for Biological Cybernetics|
|2008||Diploma thesis at Max Planck Institute for Biological Cybernetics|
|2007||Intership at Siemens VDO, Regensburg
|2004 - 2007||Software developer at Altasoft GmbH, Stuttgart|
|2002 - 2008||Studied Software Engineering at the University of Stuttgart. Diploma in Computer Science.|
|2002||Abitur, Schickhardt-Gymnasium Herrenberg|
, , , and (October-2014) Active In-Hand Object Recognition on a Humanoid Robot
IEEE Transactions on Robotics 30(5) 1260-1269.
, and (April-2014) A comparison of geometric- and regression-based mobile gaze-tracking
Frontiers in Human Neuroscience 8(200) 1-12.
Conference papers (3):
, , , and (May-2012) Active Object Recognition on a Humanoid Robot, IEEE International Conference on Robotics and Automation (ICRA 2012), IEEE, Piscataway, NJ, USA, 2021-2028.
, , , and (November-2011) Going into depth: Evaluating 2D and 3D cues for object classification on a new, large-scale object dataset In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), , 1st ICCV Workshop on Consumer Depth Cameras in Computer Vision (CD4CV2011), IEEE, Piscataway, NJ, USA, 1189-1195.
, , and (September-2011) Image Retrieval with Semantic Sketches In: Human-Computer Interaction: INTERACT 2011, , 13th IFIP TC13 Conference on Human-Computer Interaction, Springer, Berlin, Germany, 412-425, Series: Lecture Notes in Computer Science ; 6946.
(October-2010): Learning and Recognizing 3D Objects by Combination of Visual and Proprioceptive Information, 11th Conference of Junior Neuroscientists of Tübingen (NeNa 2010), Heiligkreuztal, Germany.
: Multimodal object perception for robotics, Fakultät Konstruktions-, Produktions- und Fahrzeugtechnik der Universität Stuttgart, (October-2014).