Learning human activities and object affordances from RGB-D videos (English)

In: The International Journal of Robotics Research   ;  32 ,  8  ;  951-970  ;  2013

How to get this document?

Free access

Understanding human activities and object affordances are two very important skills, especially for personal robots which operate in human environments. In this work, we consider the problem of extracting a descriptive labeling of the sequence of sub-activities being performed by a human, and more importantly, of their interactions with the objects in the form of associated affordances. Given a RGB-D video, we jointly model the human activities and object affordances as a Markov random field where the nodes represent objects and sub-activities, and the edges represent the relationships between object affordances, their relations with sub-activities, and their evolution over time. We formulate the learning problem using a structural support vector machine (SSVM) approach, where labelings over various alternate temporal segmentations are considered as latent variables. We tested our method on a challenging dataset comprising 120 activity videos collected from 4 subjects, and obtained an accuracy of 79.4% for affordance, 63.4% for sub-activity and 75.0% for high-level activity labeling. We then demonstrate the use of such descriptive labeling in performing assistive tasks by a PR2 robot.

Table of contents – Volume 32, Issue 8

Show all volumes and issues

The tables of contents are generated automatically and are based on the data records of the individual contributions available in the index of the TIB portal. The display of the Tables of Contents may therefore be incomplete.

Optimizing waypoints for monitoring spatiotemporal phenomena
Binney, Jonathan / Krause, Andreas / Sukhatme, Gaurav S. | 2013
Optimality and Robustness in Multi-Robot Path Planning with Temporal Logic Constraints
Ulusoy, Alphan / Smith, Stephen L. / Ding, Xu Chu / Belta, Calin / Rus, Daniela | 2013
Vehicle model identification by integrated prediction error minimization
Seegmiller, Neal / Rogers-Marcovitz, Forrest / Miller, Greg / Kelly, Alonzo | 2013
Towards dynamic trot gait locomotion: Design, control, and experiments with Cheetah-cub, a compliant quadruped robot
Spröwitz, Alexander / Tuleu, Alexandre / Vespignani, Massimo / Ajallooeian, Mostafa / Badri, Emilie / Ijspeert, Auke Jan | 2013
Learning human activities and object affordances from RGB-D videos
Koppula, Hema Swetha / Gupta, Rudhir / Saxena, Ashutosh | 2013
A human-inspired object handover controller
Chan, Wesley P / Parker, Chris AC / Van der Loos, HF Machiel / Croft, Elizabeth A | 2013
Call for Papers: Special Issue on Robot Vision
| 2013
Call for Papers: International Symposium on Robotics Research
| 2013