Measuring pollinator visitation: Where do they come from? Who are they? Where are they going?

Figure 1: Paul Gauguin D'ou Venons Nous [Public domain], via Wikimedia Commons

Paul Gauguin inscribed the French title of the painting shown above onto its top left corner: D'où Venons Nous / Que Sommes Nous / Où Allons Nous, which translates to: Where Do We Come From? What Are We? Where Are We Going? In our lab, we’re also interested in these questions. It’s just that we focus more on the biological part of the questions, and we have different subjects than humans: bees (pollinators). Our version of the D’ou Venons Nous is:

Figure 2: HMC Bee Lab’s version of D’ou Venons Nous (The clipart of bee comes from: http://www.clipartpanda.com/clipart_images/free-clip-art-555844)

Our lab has been studying pollinator visitation for a while (check out this post from last summer). Yet most of the time we use unfortunate students as the keen observers that’ll record all the visits by different kinds of pollinators. An obvious problem with this approach is its reliability. There might be multiple pollinators on one flower patch. Keeping track of multiple pollinators at once is very hard for a student who also needs to write down observations as thoroughly as possible. Another problem is scalability. Each student can only watch one patch of flowers at a time, and sitting outside in the sun isn’t a job that one can do for a long time. We need a more reliable, scalable and automated system for recording pollinator visitation. Imagine how wonderful it would be if we can just sit in our labs and watch live feeds from field stations on our laptops.

We've teamed up with Professor Chris Clark's lab, LAIR, to develop an automated system. The system that our lab has envisioned includes a stereoscopic camera system set up in front of natural flower patches, and a program that can determine the flight trajectories of pollinators visiting the flower patch, and then determine which pollinators have visited which flowers and when. The system has to be durable in outdoor environments (the blazing California sun). The program has to be as accurate as possible.

The plan of this summer is to develop and use this system to obtain and analyze as much real-world data as possible. After the first week, we already have a roadmap (Figure 3) of this project (more or less). We’ve made some progress in each part of the project.

Figure 3: Preliminary Project Roadmap

Choosing a camera setup for taking stereoscopic videos

We plan to use stereoscopic cameras to take videos of bees visiting flowers, so that we can reconstruct 3D scenes of the flower patches and scenes.

Figure 4: Stereoscopic camera system under consideration: GoPro stereoscopic mount (left); Prime 13 Camera (right)

We will choose between the GoPro stereoscopic camera system and the Prime 13 Optitrack camera system (Figure 4). While the GoPro stereoscopic camera system can provide color footage and is relatively cheap, the distortion (the fisheye effects) of GoPro lenses will increase the error in 3D reconstruction. Prime 13 cameras have little to no distortion due to specially designed lenses and they can record infrared videos, but they only record videos in grayscale, and require special rigs. For both systems, the accuracy of the 3D reconstruction will be affected by orientation, field of view, and many other parameters. So to evaluate the two different systems and choose the best parameters, we carried out field tests (see Figure 5). Based on the video samples we obtained, right now we’re leaning towards Prime 13 cameras.

Figure 5: The setup for field test of the two different camera systems

3D reconstruction: camera calibrations

Figure 6: MATLAB Stereo Camera Calibrator App in action

Every camera lens has distortion, and we need to correct the videos for that distortion to ensure further analysis can be done with fewer errors. How can we know that? We can take pictures of objects with known dimensions and straightness, and compare those with the dimensions and straightness of objects in the pictures taken. Then we can calculate something called the “intrinsic matrix”, which includes the information necessary to correct distorted images.

To combine videos from two cameras together and reconstruct a 3D scene, we need to know the relative positions and orientations of the two cameras with respect to each other. How can we know that? We can again take pictures of objects with known dimensions, and by comparing the change in object orientation across two cameras, we can calculate the relative positions and orientations of the two cameras, which is called the “extrinsic matrix”.

We decided to use MATLAB Stereo Camera Calibrator App, as it can calculate both the intrinsic matrix and the extrinsic matrix for stereo camera pairs.

Tracking of bees

Tracking moving objects in videos has been an active research area for a while. For a comprehensive review on object tracking, see the Wikipedia article on video tracking. Basically most of the object tracking algorithms involve some sort of detection mechanism that detects moving objects, and filtering mechanisms that deals with the scenario where the objects become obstructed.

For the motion detection algorithms, we decided to use a frame difference algorithm. It calculates the difference between frames to figure out what pixels have moved across frames. The problem with this algorithm is that it also detects flowers moving in the wind. We need to improve the algorithm to only detect bees. (Click here to view a short clip of the original video!)

Figure 7: Threshold Frame Difference Image (left: original frame; right: threshold frame difference frame). Can you tell which white dot is the bee?

For the filtering part, we decided to implement a particle filter. A particle filter generates random particles across the image. Particles closer to detected objects will get higher weights. Particles of lower weights will be eliminated, and new particles will be generated near the higher weighted particles. For a more detailed discussion on particle filters, please visit the Wikipedia page on particle filter, and wait for our future posts.

While we have made some progress, we still have a lot to do. We need to improve our detection algorithms for bees, and gather more data from the field station. We’ll be constantly updating the blog throughout the summer. More articles on how we design and implement our computer vision program will be coming. Stay tuned!