Visual Human Awareness Estimationfor mobile robots
This project investigates how a mobile robot can estimate whether nearby people are aware of its presence and how likely they are to react. Students will develop a camera-based perception system using cues such as gaze, head and body orientation, motion, and gestures, aiming to support decisions such as proceeding, waiting, stopping, or signaling.
Illustrative reference, not project output
Research question
Can ordinary RGB imagery provide reliable evidence of whether a person is aware of a robot, even when observations are incomplete or contradictory?
The planned input is monocular RGB, so every cue the estimate rests on has to be recovered from that one stream.
Person tracking
Who is who, and where they are, from frame to frame.
Gaze
Where the eyes are directed.
Head orientation
Which way the head is turned.
Body orientation
Which way the torso faces.
Motion
How the person is moving, and where.
Gestures
Deliberate movements a person makes.
The pipeline
From one camera stream to one robot decision
The stages the project has to put in place, end to end. The stack targets two robots — a Ridgeback base and a Unitree As2 — both running ROS 2.
1. Input
Monocular RGB
One ordinary camera stream.
2. Detect
Person tracking
Who is in the scene, and where.
3. Read
Cue extraction
Gaze, head, body, motion, gestures.
4. Estimate
Awareness estimate
Does this person appear aware of the robot?
5. Act
Robot decision
Proceed, wait, or signal.
Target platforms
One perception stack, two very different robots
The same estimator is meant to run on a low wheeled base and on a quadruped. The camera sits at very different heights on the two, and they move in different ways.
Ridgeback
Omnidirectional wheeled base
Footprint
960 × 793 × 311 mm
Mass
135 kg
Drive
Four mecanum wheels
Unitree As2
Quadruped
Standing
720 × 378 × 457 mm
Mass
approx. 20 kg
Drive
12 joint motors
Product images and figures: Clearpath Robotics · Unitree Robotics
Reference example
The kind of perception this project targets
Illustrative demo
This example illustrates the type of visual perception capabilities explored in the project, including human detection, tracking, and attention estimation.
Related work
Leader Following
A separate project on the Ridgeback, one of this project’s two target platforms: picking one chosen person out of a crowded scene and following them through clutter, by fusing camera-based detection with point-cloud perception.
It shares this project’s starting point — finding and holding on to a person in the camera view — and takes it in the direction of navigation rather than awareness.
The research question names the difficulty directly: observations can be incomplete or contradictory. Cues can be missing, and the cues that are present can point in different directions.
Handling that is part of the project rather than an edge case left for later. Where the evidence is unclear, the planned response is to signal presence and prompt a reaction, rather than to guess.
Target output
Awareness information for safer robot behavior
The system aims to estimate whether nearby people are aware of the robot and provide this information in a form that can support navigation and interaction decisions.
PROCEED
People nearby show no sign of needing the robot to yield; it can continue its planned path.
WAIT / STOP
A person appears to be attending to the robot and may be about to react; the robot should slow, wait, or stop.
SIGNAL
The evidence is unclear or the person seems unaware of the robot; the robot can signal its presence to prompt a reaction.
