Who Knew Ants Kissing Would Be So Problematic?

 You’ve probably solved a maze like this before [1]:

[1]: A Typical Maze

The goal is to get from the entrance of the maze to the exit. And the constraints are the walls, which restrict your movement and force you to make a decision when the path splits. 

Now imagine being a turtle ant colony on this fake branch [2]:

[2]: The fake branch the ant colonies are placed on. Their movements are tracked and recorded. The red square indicates where the ants begin their journey. The green circle is an example of a “branching point”, where the path splits into two. The blue pentagon is a food source and the magenta hexagon is the location of a competing colony.

The goal is to get from the entrance (red square on [2]) to a suitable nest (dark red cylindrical rods in [2]). There are eight nests to choose from, so the colony has a lot of options. Two important metrics the colony needs to consider are the direct path to a food source (blue pentagon on [2]) and the distance from other competing colonies (magenta hexagon on [2]). And of course, they can only walk on the path provided by the tree branch. 

Given all this, a great metric we can use to gauge the decision-making process of the colony is to keep track of what they do at a “branching point”, points on the branch where it forks into separate branches (green circle in [2] is one example).

Unsurprisingly, where a colony decides to set up its nest is extremely important. Hence, the ants spend a lot of time exploring their different options, giving us a treasure trove of video footage. By cropping the video footage down to focus on each branching point, we can detect and track where an ant enters and where it exits, giving us a glimpse of how ants decide to traverse through the branch. 

However, automatically tracking ants with a computer is easier said than done. Part of the difficulties come about from the multitude of edge cases that occur and the low quality of the cropped videos, as they’re produced by cropping out tiny sections of a video capturing a large arena. These issues can be seen in the following examples. Try tracking each of the ants yourself!

Example 1: The simplest case: tracking one ant. Please watch the video below.

That one ant was pretty simple to track, wasn’t it? The ant appeared on the right side of the frame and traveled leftward until it exited on the left side of the frame. 

Example 2: Slightly less simple: tracking 2 separate ants. Please watch the video below.

Also pretty simple. The first ant appeared on the right side of the frame and traveled leftward. While the first ant was traveling, a second ant appeared on the right side of the frame also traveling leftward. The first ant then exited on the left side and then the second ant also exited on the left side. 

Example 3: Much more complex, tracking multiple ants that are close to one another. Please watch the video below.

Now it’s not as simple. Not only do we have to track three separate ants at the same time, but these three ants are "kissing" at some point in the video. Also, an ant crawls on top of another ant! Even a human, who is exceptionally good at tracking objects, might second-guess each ant's overall trajectory.

In fact, this is the main problem when tracking ants using a computer program: moments when multiple ants are visually “connected” to each other. When these ants are close enough to each other, the computer gets tricked into believing that there is only one ant, rather than two (see image below for an in-depth explanation). 

   [3]: Ant merger and unmerger process. From left to right

Image 1: Two ants (IDs 13 and 14) are coming towards each other.

Image 2: The two ants are now close to each other. The computer has been tricked! It no longer detects ant 14, but rather it believes ant 13 is the only ant present!

Image 3: When they separate, the computer notices a new ant and assigns it ID 15.

Ultimately, we want the computer to be able to deal with situations like this by itself, and we believe a good place to start is to identify when and where these events happen. 

Let’s try to understand how our program gets tricked by first describing how our program detects ants. Our program detects ants by tracing out their edges (see image below for a brief explanation). 

[4]: Process of detecting ants in a given video frame. From left to right

Image 1: We convert the image to grayscale, and color isn’t necessary to detect the position of ants (you can clearly see it)

Image 2: We subtract the background, leaving us with the foreground, in this case the ant

Image 3: We detect the ant’s edge

While this is great for properly detecting isolated ants, when two or more ants are close enough to each other, their edges overlap and then combine into one. This means the tracker loses track of one of the ants, and the other ant (that is still being tracked) suddenly has a massive, positive change in the area.

Aha! An ant suddenly balloons in area and another ant simultaneously disappears at the same place? Surely that didn’t happen by chance, this must be a case where two or more ants are detected as one! Let’s call this situation a merger. 

And the same logic applies the other way around. An ant suddenly suddenly shrinks in area and another simultaneously appears at the same place. That likely didn’t happen by pure chance, this must be a case where one detected ant suddenly splits into two or more detected ants. Let’s call this situation an unmerger.  

While our current ant tracker is great at detecting when and where merges and unmerges occur, it has not reached a point where it can reassign ID numbers to the same ants that briefly disappeared and then reappeared in the video frame. Figure 3 illustrates this problem, as ant 13 refers to the same ant before and after the merge. All we would have to do is recombine ant 14 and 15 together and it would be correct. But, would this always work? What other situations might arise? To figure this out, we would need to look at a lot of examples.

Currently, we are able to detect when and where a merger or unmerger happens. We believe this feature will help us figure out how to reassign ants, as it makes it easy to find many different edge cases so that humans can look at them to figure out whether or not there are any commonalities. Hopefully, future work will allow us to write code that would automatically reassign ants when they detach from a merged ant.

A question you might ask is why spend so much time trying to reassign ant IDs? How does any of this help us learn how ants navigate this branch maze? With proper ant tracking automation, it can quickly and efficiently collect much more data on what choices hundreds of ants make at a given moment. Most importantly, it will save students hundreds of hours, as they will not have to manually measure every ant’s decision at a branching point. And with this information, we can see how a colony’s decision-making process changes over time and hopefully give us a glimpse as to why and how a colony chooses any particular nest as its new home. 

Media Credits

[1]: Veder, Olga. “Download Black Rectangular Labyrinth. Game for Kids. Puzzle for Children. Maze Conundrum. Flat Vector Illustration Isolated on White Background. for Free.” Vecteezy, 9 July 2021, www.vecteezy.com/vector-art/2935254-black-rectangular-labyrinth-game-for-kids-puzzle-for-children-maze-conundrum-flat-vector-illustration-isolated-on-white-background. 

[2]: The HMC Bee-Ant Lab fake branch setup. See how Harvey Mudd College students James Clinton and Simon Woodside constructed the branch here: https://hmcbee.blogspot.com/2021/07/we-make-trees-for-ants.html. The modifications to the image were done by me.

[3]: Assigning IDs to new or existing ant tracks. See the tracking implementation here: https://github.com/beelabhmc/ant_tracker/blob/master/scripts/tracker.py.

The code was written initially by Srini Ananthakrishnan, and I made many modifications so that it better suited our experiment environment (dealing with low-quality videos and an emphasis on data collection)

[4]: Detecting new or existing ants given a video frame. See the detection implementation here: https://github.com/beelabhmc/ant_tracker/blob/master/scripts/detector.py

Similarly, the code was written initially by Srini Ananthakrishnan, and I made many modifications so that it better suited our experiment environment.

Further Reading

1. Anonymous. “MATLAB: Saving Students One Blob at a Time.” HMC Bee Lab, 11 May 2018, hmcbee.blogspot.com/2018/05/matlab-saving-students-one-blob-at-time.html. 

2. Massarat, Arya. “When Ants Get Too Close: A Guide for Splitting Up.” HMC Bee Lab, 1 Mar. 2019, hmcbee.blogspot.com/2019/02/when-ants-get-too-close-guide-for.htmlRead how former HMC Bee Lab member Arya encountered the same problem of ants merging and unmerging but came up with a different potential solution.