Enhancing Video-Based Robot Failure Detection Using Task Knowledge

## Контекст Modern robotics relies heavily on the ability to detect and respond to task failures to ensure safe operation and efficient task completion. Despite significant advancements, many existing failure detection methods face challenges in real-world scenarios due to limited generalizability and insufficient contextual understanding. Traditional approaches often rely on low-level sensory data, neglecting task-specific knowledge that could enhance detection accuracy. This limitation underscores the need for integrative methods that leverage both visual and semantic information to improve robustness and reliability in failure detection. ## Метод Our approach introduces a video-based failure detection system that incorporates spatio-temporal knowledge derived from the robot's actions and the task-relevant objects in its field of view. By leveraging these elements, the method enhances the interpretability and accuracy of failure detection. The architecture includes a spatio-temporal feature extraction module, which processes video frames to identify actions and objects. This information is then combined with a failure detection model, enabling the system to reason about task execution and identify deviations indicative of failures. The approach is designed to be adaptable, utilizing existing datasets with additional annotations for task-relevant knowledge. ## Результаты To evaluate the method, we conducted experiments on three datasets: ARMBench, EPIC-KITCHENS, and a custom robotic dataset. These datasets were augmented with annotations for actions and objects relevant to the tasks being performed. The results demonstrate a substantial improvement in performance, with the F1 score increasing from 77.9 to 80.0 on the ARMBench dataset using variable frame rates. Test-time augmentation further enhanced the score to 81.4. These findings highlight the significant impact of spatio-temporal information on failure detection and validate the proposed data augmentation strategy as an effective means to improve model performance. ## Значимость The proposed approach has broad applications in robotic task execution, particularly in domains requiring high reliability, such as healthcare, manufacturing, and domestic service robots. By integrating task-relevant knowledge, the method offers enhanced robustness and adaptability to real-world variations. Its ability to improve failure detection performance without significant computational overhead underscores its practical value. Furthermore, the proposed data augmentation technique provides a novel approach to optimizing model training, paving the way for future research into heuristic-driven enhancements for robotic vision systems. ## Выводы The study underscores the critical role of spatio-temporal knowledge in improving video-based failure detection. The proposed method demonstrates marked improvements in detection accuracy across diverse datasets, highlighting its potential for real-world deployment. Future research will focus on refining heuristics, exploring additional task-relevant features, and extending the approach to more complex robotic tasks. The availability of code and annotations ensures transparency and facilitates further advancements in this field.