AI Human Activity Recognition for On-board BAS Experiments

Background As humanity aims for space missions such as BAS and lunar missions, real-time ground support becomes impossible due to communication delays. An AI-based HAR system acts as an on-board assistant that supports the execution of scientific experiments, ensuring the success of science beyond Earth's orbit.

In the space environment, AI-based HAR system may act as mission-critical support for astronauts. By tracking astronaut movements and activities in real time, HAR ensures scientific experiments and related protocols are executed flawlessly without requiring constant, high-bandwidth communication with mission control.

Description Challenge is to design and train an AI model that recognizes and validates the sequence of a pre-defined experiment using human activity recognition techniques.

Standalone operation: Space stations operate on restricted data bandwidth to Earth. Rather than streaming raw video to ground control, data is processed locally at the 'edge.' Inputs are given from fixed-payload cameras.

Dataset generation to train model for object detection, pose estimation and hand-object interaction based on the steps of the experiment.

Optional: Another challenge is that Standard 2D or ground-based 3D posture models fail because astronauts do not have a fixed 'up' or 'down' orientation. The AI model should use orientation-agnostic 3D Human Mesh Recovery (HMR) to track the astronaut’s body relative to the payload rack, not the floor.

Expected Solution

• The software should continuously process local video feeds to track the sequence of experiment.
• At the start or after each step, the model should suggest the next step to be performed.
• It should alert when a step is skipped or an out of sequence step is added. It should be a voice based alert.
• Using the live video, it should generate a timestamped and structured lightweight text file of the conducted steps with outcomes/ status.
• Stream the video of the experiment to specific IP and also store the video locally.
• A graphical user interface for monitoring the above activities.
• Deliverable: A trained AI model that runs on offline standalone system