ManiWAV
ManiWAV enhances robot manipulation skills through audio-visual learning from human demonstrations in diverse environments.
English
English
What is ManiWAV?
ManiWAV is a research project aimed at enabling robots to learn manipulation skills through audio and visual data collected in outdoor settings. It learns manipulation strategies by directly observing human demonstrations that include synchronized audio and visual feedback. The system demonstrates its capabilities through four tactile-rich tasks requiring the robot to either passively sense contact events or actively perceive material properties and states of objects. Additionally, it can generalize to new environments by learning from diverse human demonstrations.
Who Can Use ManiWAV?
ManiWAV is ideal for researchers and developers in robotics and artificial intelligence. They can use this system to enhance a robot's manipulation skills in complex environments, particularly when visual information is limited or incomplete, by leveraging audio cues to improve perception.
How Can ManiWAV Be Used?
ManiWAV can be used in various scenarios such as:
Cleaning a whiteboard by erasing graphics.
Flipping a bagel in a frying pan.
Transferring dice from one cup to another.
Securing wires with Velcro tape.
What Are the Key Features of ManiWAV?
Key features of ManiWAV include:
Utilizing rich audio signals for interaction and object property information.
Collecting audio and visual feedback using 'ear-in-hand' devices.
Learning manipulation strategies directly from human demonstrations.
Demonstrating capability through four tactile-rich tasks.
Generalizing to unseen environments through diverse human demonstrations.
How to Use ManiWAV?
Here’s how to use ManiWAV:
1. Install and set up the data collection device ensuring audio and visual feedback are synchronized.
2. Collect human demonstration data including contact events and material-specific audio signals.
3. Train the robot manipulation strategy using the collected data.
4. Test and validate the strategy in different outdoor environments.
5. Adjust the strategy based on test results to improve generalization.
6. Apply the trained strategy to real-world manipulation tasks.
