Research Improving visualization of swarms for better operator control


Simulation of a swarm of 256 robots using the CUDA Swarm Simulator, with communication connections shown in gray and the convex hull shown in blue.



As a Ph.D. student in Intelligent Systems Program at the University of Pittsburgh, I research human control of robotic swarms. My dissertation topic, “Improving Operator Recognition and Prediction of Emergent Swarm Behaviors” is focused on improving how swarms of robots are visualized to operators on computer screens, and how tools can be provided to improve ease of interaction with, and control of, the swarm.

We typically define a swarm as collectives of agent operating cooperating to achieve some desired outcome. The operation of each robot is governed strictly by interaction with it’s neighbors–usually the small subset of robots nearby in location. The desired global behaviors, such as rendezvousing to a point or collective motion in a certain direction, emerge from these local interactions. For example, “flocking” behavior, as is typical of birds or schools of fish, can be achieved using three simple rules:

  1. Repulsion: move away from the average position of your closest neighbors.
  2. Cohesion: move towards from the average position of your farthest neighbors.
  3. Alignment: Steer toward the average direction of your neighbors.

Setting the relative weights and sizes of the zones that denote “close” or “far” neighbors can give different types of flocking behaviors.

Robotic swarms are ideal for situations where large spatial coverage is needed, as they typically involve much larger numbers that other robotic systems and thus can be distributed over a wide area. Furthermore, because each robot interacts only with neighbors and no global information is needed, individuals can fail without significantly affecting the behavior of the swarm as a whole, making swarms particularly robust. Research over the past couple decades have brought swarms from the imagination to the real world–an excellent example of an actual swarm are the Kilobots, developed at Harvard.

Prior Work

My first series of research papers investigated control of swarms with limitations in the communication channel [1] [2]. There, we discovered a phenomenon called neglect benevolence, which describes the fact that, because global behaviors take time to emerge in swarms, the best time to issue a command as an operator may not be as soon as you realize the need for it. Research performed by our collaborators proved neglect benevolence for certain systems [3], and then showed that the performance of human operators improved when the display was augmented to assist the operator in dealing with neglect benevolence [4].

I have also conducted research in control of leader-based swarms. Specifically, we asked how leaders could be selected [5] and how different methods of propagating control inputs through leaders impacted control of the swarm in a movement task [6]. We found that explicit methods of propagating control from leaders, where neighbors directly matched the leaders heading, gave better outcomes than when leaders were weighted equally with other neighbors. The latter case, however, could provide better support when there is significant sensing or control error for the leaders.

Dissertation Summary

My current work is born from both the previous lines of research, and aims to improve how swarms are visualized for simple movement control tasks–something that will need to be understood and controlled easily when swarms move from the laboratory to the real world. Our first experiment asked whether different behaviors were recognized differently by participants, which might suggest they required different methods of visualization. The results showed that certain behaviors are indeed harder to recognize than others against background noise, with rendezvousing being the easiest, and flocking and dispersion being harder.

Two follow up studies looked at how some candidate visualization methods affected operators’ abilities to both predict and control a swarm operating under different behaviors. Both studies confirmed that the visualization did have an impact on the operators’ abilities, and that this impact varied in some ways based on the behavior being performed. In short, the two displays which preserved the view of a swarm as a single entity, rather than independent robots, kept performance high. The two displays which showed only a small subset of the swarm, often with members separated visually by a large distance, did not give as good an outcome.


A simulated swarm of simple robots (right). Illustrations of different display methods that highlight leaders only (bottom right) or a summary ellipse (top right)–grayed robots exist, but are not shown in actual display.

Currently, we are investigating how the trust an operator has in the swarm affects their performance, and more specifically how the swarm can estimate the user’s trust and adjust their performance on-the-fly to account for changes in trust. The visualization plays a key role here, as trust can be gained or lost depending on what information is shown (and how accurate that information is). All of the work for this dissertation published so far (along with many other works) can be found on my Downloads page.