Master’s Thesis on Human-Robot-Interaction Design

Overview

Someone operating a large robot can only attend to part of the workspace at a time. This research asked what changes if the machine is told where that attention currently is. I designed a collision-avoidance system in which the operator’s gaze decides how the robot may behave: whatever falls inside the operator’s central field of view can be handled freely, while anything outside it is protected—the arm slows and stops before it arrives. Nothing has to be labelled beforehand. The distinction is drawn in the moment of the work, from where the operator happens to be looking. See the demonstration video here (on YouTube).

The work was my master’s thesis in Design & Engineering at Politecnico di Milano, carried out during a research internship at Man-Machine Synergy Effectors, Inc. in Japan, and was presented at the 39th Annual Conference of the Robotics Society of Japan (RSJ2021).

It is also where my design philosophy starts to take its present shape. What this system reads is not a command but a cognitive state—whether a thing has been noticed at all—and it turns that state into the conditions under which the machine is permitted to act. The object being designed is not the robot’s motion but the relationship: what the machine may do follows from what the person currently understands. Designing that relationship rather than each individual behaviour is the same move I make now that the party doing the acting is a generative AI rather than a robot arm.

Abstract

Human-operated robots are valuable precisely where autonomous ones are not: in workspaces that were never designed for the task at hand, next to people and equipment that must not be struck. Existing collision-avoidance systems cope with such spaces by having every object declared in advance as either a target of manipulation or something to be protected—a setup cost that grows unmanageable for a machine meant to be carried from site to site.

This research proposed making that distinction at run-time instead, from the operator’s gaze. Objects inside the operator’s central field of view are treated as manipulable, objects outside it as protected, and people as protected under all circumstances. The method introduces an active parameter—a human one—into a control loop that had until then been built entirely from passive sensor data. A prototype was implemented on JINKI type Zero, a 1.9 m anthropomorphic heavy-machinery robot, and behaved as specified across five demonstration trials.

1. Introduction

A human-operated robot combines two things that are hard to obtain together: the power and precision of a machine, and the situational judgement of a person. Fully autonomous robots do not yet supply the second. And where industrial robots work in designed cells behind fences, human-operated ones are expected to work in places arranged for something else—alongside on-site workers, among delicate equipment, in nuclear facilities and on construction sites. Guaranteeing that the robot does not damage what surrounds it is therefore part of what makes it usable at all.

The established defences against accidental contact are sensitive skin, where proximity sensors cover every critical part of the manipulator, and depth sensing, where the surroundings are reconstructed as a point cloud and distances are computed in that virtual space. Both work. Both, however, presuppose that the objects around the robot have already been sorted into ones it may touch and ones it must not: without that sorting, a machine forbidden to collide with anything cannot manipulate anything either. For a general-purpose tool that is transported between sites and set down among whatever is there, the configuration and calibration this demands is more expensive than the safety it buys.

The opening this research works through is that a human-operated robot already has a person inside its control loop. Studies of operational error point to situational awareness—an accurate grasp of what is going on around you—as the decisive factor, and situational awareness is known to correlate strongly with gaze direction. Collision avoidance had conventionally been built from passive information alone, meaning data read off sensors. Adding one active parameter, where the operator is looking, gives the system an indication of whether contact with a given object would be intended or not.

2. System Behaviour

An object the operator intends to touch is called a manipulable object, and may be handled without restriction. An object the operator has no such intention toward is a protected object, and may not be touched. Which of the two an object is at any given instant follows from its position relative to the operator’s central field of view: inside it, the object is under enough of the operator’s awareness to count as intended; outside it, it is not.

Five rules govern the behaviour that follows. The manipulator may interfere freely with objects inside the central field of view. It may not interfere with objects outside it, and gradually halts as it approaches one. Once halted, it is gradually released as the operator’s field of view shifts onto it. No restriction applies if the field of view leaves an object the manipulator is already in contact with, so that work in progress is not interrupted by a glance elsewhere. And the manipulator may not interfere with a human being, wherever that person stands. Where a robot has several manipulators, the rules apply to each separately.

What the operator gets from this is the ability to work at full speed on what they are looking at, while relying on the machine not to strike the equipment they are not. Safety of the people on site is unconditional and does not depend on the operator’s intentions at all.

3. Expected Benefits

Heavy machinery offers the closest existing picture of this kind of work: large forces applied in unstructured places next to people. Collision accidents are among the principal hazards on construction sites worldwide, and inattention is repeatedly named as a cause. Of the 362,600 accidents recorded on European construction sites in 2018, 19.6% involved being struck by an object in motion; an investigation of crane-related incidents attributed 19% of them to operator inattention.

Because attention and gaze direction are strongly correlated, those two proportions can be combined into a rough estimate of the ground a gaze-based system might cover—on the order of thirteen thousand accidents a year in Europe. The figure is an inference from published statistics rather than a measured result, and it is offered as an indication of scale.

4. System Verification

A demonstration system was built and run as a proof of concept.

4.1 Implementation

The system was implemented on JINKI type Zero, a robot developed at Man-Machine Synergy Effectors, Inc. It has the upper body of a human, stands 1.9 m tall, weighs 600 kg and has thirty degrees of freedom. Operator and robot are collocated and coupled as a master-slave system: a VR headset synchronises the operator’s view and head rotation with the robot’s, which gives the sense of being united with the machine rather than driving it.

A LIDAR mounted on the robot scans the surrounding workplace as point clouds, which are represented in a virtual space as voxels, each with its own collider. A digital twin of the robot moves in that same space; when its arm overlaps voxels, the system raises a proximity warning whose effect depends on the direction of the operator’s field of view. That field is modelled as a cone of 60° × 55° (H×V)—close to the region in which people perceive symbols, colour and depth—and its direction is approximated by the forward direction of the headset. The warning is passed to the physical robot and shapes its motion according to the rules above.

4.2 Demonstration Experiment

A preliminary evaluation tested whether the implementation did what had been specified. The operator was asked to move the arm toward a pole while looking away, confirm that the system had issued a warning and halted the arm, then look at the halted arm to release the restriction, and finally take a ball from the top of the pole while keeping it in view. Five trials were run: the first three and the fifth by an experienced operator, the fourth by someone on his second time piloting the robot. Each was recorded. In all five, the system behaved as Section 2 defines. The demonstration can be watched in the video of the system in action.

5. Limitations and Future Work

Human detection was not implemented in the prototype. What needed validating was the relation between the operator’s gaze and the manipulability of objects, and the rule protecting people is unconditional and does not depend on it. Adding that detection would make the system more robust, and would also open applications beyond safety—highlighting the people near the robot in the operator’s view, for instance, or warning before a proximity limit is reached.

Gaze was approximated by the forward direction of the headset. This was chosen for its directness, which matters in a master-slave system, but other instruments—stationary cameras, eye trackers in smart glasses—would estimate it differently and might lower the barrier to introducing such a system in industries not accustomed to wearing one. Applying the method to mobile robots raises a further problem: JINKI type Zero is fixed in place while it is under development, and a machine that moves during operation needs point-cloud processing efficient enough to keep up.

The evaluation itself remains qualitative. A quantitative field study is the right way to judge the system, and waits on the robot reaching a practical stage.

6. Conclusion

The research proposed a collision-avoidance system for human-operated robotic manipulators that uses the operator’s gaze to prevent unintended contact with people and equipment. On the premise that awareness and gaze direction are strongly correlated, objects inside the operator’s central field of view were treated as manipulable and those outside it as protected. Putting a human factor into the control loop made it possible to draw that line at run-time, without any of the preparation the existing methods require.

The system was verified on a general-purpose heavy-machinery robot and behaved as defined. The same principle should carry to any setting where a person is already part of the loop through which a machine is operated.

Acknowledgements

This work was carried out during an internship at Man-Machine Synergy Effectors, Inc., in partial fulfilment of the master’s degree in Design & Engineering at Politecnico di Milano. I am grateful to both organisations for making the opportunity possible.