The QR Codes of Computer Vision: ArUco Markers

Imagine this: You are a brilliant scientist who was asked to use computer vision to track the movements of an unknown and bewildering animal (turtle ants). Knowing that you will need many hours of footage (they’re particularly slow) and that your computer vision models tend to work better when the camera is static, you decide to mount a camera above their habitat. You are prepared to track these peculiar animals by focusing on specific regions of interest, these regions being when a tree branch forks into two (when a tree branch “splits” to become two, see Figure 1).

[1]: An example of a tree branch forking. Notice how the branch “splits” into two.

And then it hits you. Sure, we humans can easily identify when a tree branch forks, but can a computer? After all, a computer sees an image as a bunch of numbers. What we humans determine as a branch forking means absolutely nothing to a computer. We can try to alleviate this by indicating to the computer each specific coordinate at which a region of interest takes place, but that’s tedious and prone to many errors. A poor student volunteer would have to track every single region of interest for every new video and simultaneously take note of when an ant enters and exits a region of interest. Can you feel it? The aggregate lost hours of that student’s life piling up like a bounty for a delighted Grim Reaper, the student wondering why they signed up for the ant tracking project in the first place…

And on top of that, we must also consider that the camera, despite our best efforts, will not always be in a static position. The camera may be a little closer to the subject, or a little tilted. And whilst it may not seem like a huge difference, when we are trying to track the most minute movements of the tiny turtle ants, it becomes extremely important that the computer can automatically track these regions of interest accurately even with all these sources of error.

[2] Several ArUco Markers

Turns out, there has been a very innovative solution! Former members of the ant tracking lab over the summer of 2021 Catherine Wu, Simon Woodside, and Carter Moyer solved this problem by utilizing ArUco markers [2] and OpenCV (a commonly used computer vision library). ArUco markers are “binary square fiducial markers that can be used for camera pose estimation” (OpenCV documentation), and they look like mini QR codes. In simpler terms, OpenCV can not only detect ArUco markers through a camera input, but it can also further estimate the placement of the camera relative to the ArUco marker. Figure 3 shows us the result of using OpenCV and ArUco markers in tandem. OpenCV can detect the relative position and angle of each ArUco marker and display colorful axes for each marker, indicating their placement on the image. The x-axis is indicated by the red line, the y-axis is indicated by the green line, and the z-axis is indicated by the blue line [3]. 

[3]: ArUco markers at work! Notice the axes drawn (in red, green, and blue) by the computer.

But how do these ArUco markers operate behind the scenes? Recall that ArUco markers are fiducial markers, which are like artificial landmarks, making them easy to recognize, detect, and measure. ArUco markers resemble a square QR code (another type of fiducial marker) for a reason. The four corners of the marker are used to calculate the camera position, and the inner black-and-white region is used for the identification of the specific ArUco marker (remember, there can be multiple ArUco markers in the frame). But this method of tracking isn’t unique to ArUco. Many other systems are modeled as square-based fiducial systems, such as ARToolKit and BinARyID [4], so what makes ArUco superior? What makes ArUco optimal for calculating the camera pose is that it does not use a predefined set of markers. Instead, the algorithm used to create ArUco markers attempts to make each ArUco marker as unique as possible (making for a more accurate detection) by taking into consideration how many ArUco markers the user needs. The algorithm used is far too long and complex to summarize properly here, but if you are interested, please check out the original paper which introduced the concept of ArUco markers.

[4]: Various examples of fiducial markers. ArUco markers aren’t the only ones out there!

That is definitely cool, but how can we utilize ArUco markers to obtain the region of interest of the ant habitat? Since we want to pinpoint the location accurately, we should place our ArUco markers on the (rather fake) tree branch itself, which is where all the regions of interest are located [5]. Since the ArUco markers are fixed to the branch, it means if the branch moves so do the ArUco markers. And their static placement means that the ArUco markers always have the same constant, relative distance between themselves and the regions of interest. Hence it does not matter if there are minute differences in camera placements. As long as the ArUco markers are detectable, we can always calculate the absolute location of the regions of interest by taking into account the precise location of our ArUco marker.

[5]: ArUco markers on the branch (on the edges of the photo)

Figure [6] displays the regions of interest, after calculating them from the initial positions of the ArUco markers. A quick observation tells us that the computer was nearly spot on when drawing these regions. And it will continue to draw reasonably accurate regions of interest as long as the ArUco markers are present. Such a simple QR code-like drawing can help computer scientists with difficult tracking problems. ArUco markers are a relatively straightforward solution to the consistent problem of imperfect camera placements!

[6]: Regions of Interest detected!

Media Credits

[1]: The Tree Fork example. https://en.wikipedia.org/wiki/Tree_fork; By Duncan R Slater - Using the software Illustrator. Previously published: Not used in publication as yet, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=40308681

[2]: An example of generated ArUco markers. https://mecaruco2.readthedocs.io/en/latest/notebooks_rst/Aruco/aruco_basics.html

[3]: The OpenCV documentation website demonstrates the real-time position detection of ArUco markers. https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html

[4]: The original paper which introduced ArUco markers, the algorithm used to generate these markers, and other types of fiducial markers: https://cs-courses.mines.edu/csci507/schedule/24/ArUco.pdf

[5]: The HMC Ant Tracking Lab video footage of ants traversing an artificial branch. See how it was constructed here: https://hmcbee.blogspot.com/2021/07/we-make-trees-for-ants.html

[6]: The same HMC Ant Tracking Lab video footage, but with the areas of interest detected.

Further Readings

Avola, Danilo, et al. “A Practical Framework for the Development of Augmented Reality Applications by Using ArUco Markers.” Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods, 2016, https://doi.org/10.5220/0005755806450654.

“Detection of ARUCO Markers.” OpenCV, https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html.

Garrido-Jurado, S., et al. “Automatic Generation and Detection of Highly Reliable Fiducial Markers under Occlusion.” Pattern Recognition, vol. 47, no. 6, 2014, pp. 2280–2292., https://doi.org/10.1016/j.patcog.2014.01.005.

Tocci, Tommaso, et al. “ARUCO Marker-Based Displacement Measurement Technique: Uncertainty Analysis.” Engineering Research Express, vol. 3, no. 3, 2021, p. 035032., https://doi.org/10.1088/2631-8695/ac1fc7.