For example, we can take this fairly complex image and interpret it as a matrix of integers using Python:
What we see
What the computer sees
How can we possibly pick out objects from this mess?
My work in the bee lab involves tracking the movements of ants over video. As part of that, I need to read individual video frames, identify all the ants present there, and store their location for later. We’re looking through hours of footage, so we need a tool that can do this automatically. But how do we figure out what is an ant and what isn’t an ant when a frame just looks like a bunch of integers?
There are many different techniques to find structures in images (for example blob detection), but ultimately the goal is to create a bitmask or binary mask of the image: a copy of the image with only black and white pixels, where the white pixels represent the objects or areas of the image that we care about. We can apply this bitmask to the original image in a way that lets the white pixels act like holes through which the original image can show. This isolates features of interest and filters out everything else you don’t need so that later computations are much simpler. But to understand bitmasks and how to use them, you have to understand bitwise operations first.
What are bitwise operations?
Logical operations, like the AND and OR operations, compare two boolean values which can be true (1) or false (0). The AND operator will output a 1 only if both of the values passed into it are a 1. Otherwise, it will output a 0.
When we look at the binary representation of integers, which are a series of bits that can equal one or zero, we can think of them as an array of true and false booleans. A bitwise operation on numbers is an operation where we compare all the bits of one integer to all the bits of another integer. In the end, we produce one output integer.
Let’s examine the logical operator bitwise AND. In Python, the bitwise operator AND is implemented with the ‘&’ symbol. For each pair of bits in both binary integers, the ‘&’ symbol only places a ‘1’ in the output integer only if both integers have a binary 1 in the same spot.
Here, I compare the integers ten and three. When we apply the bitwise AND operator, we get a resulting value of two. The two integers only had matching 1s in the second position from the right, so that is the only location in the resulting integer with a 1 bit.
When we try this out in Python, we get the results we expect:
To understand the potential of this operation, it’s helpful to think of it as one integer “masking” another integer. The second integer can be thought of as a filtering mask, where every 1 bit represents a hole for the first integer to shine through.
Here, the integer three only allows the first two least significant bits of the integer ten to shine through. In this way, it “masks” every other bit of the integer ten from appearing in the resulting integer.
We can solve certain problems by selecting or creating an advantageous mask.
This concept of a bit-mask extends beyond simple integer representations and into more relevant areas of interest for the bee lab: image manipulation.
I wrote the following code with the help of this tutorial in order to demonstrate the usefulness of bit masks:
In [40]: img = cv2.imread("baby.jpg")
In [41]: img.shape
Out[41]: (436, 420, 3)
In [42]: circle = np.zeros((436, 420), dtype = "uint8")
Out[42]:
array([[0,0,0, ..., 0,0,0],
An example of two integers in binary
Let’s examine the logical operator bitwise AND. In Python, the bitwise operator AND is implemented with the ‘&’ symbol. For each pair of bits in both binary integers, the ‘&’ symbol only places a ‘1’ in the output integer only if both integers have a binary 1 in the same spot.
Here, I compare the integers ten and three. When we apply the bitwise AND operator, we get a resulting value of two. The two integers only had matching 1s in the second position from the right, so that is the only location in the resulting integer with a 1 bit.
When we try this out in Python, we get the results we expect:
To understand the potential of this operation, it’s helpful to think of it as one integer “masking” another integer. The second integer can be thought of as a filtering mask, where every 1 bit represents a hole for the first integer to shine through.
Here, the integer three only allows the first two least significant bits of the integer ten to shine through. In this way, it “masks” every other bit of the integer ten from appearing in the resulting integer.
We can solve certain problems by selecting or creating an advantageous mask.
This concept of a bit-mask extends beyond simple integer representations and into more relevant areas of interest for the bee lab: image manipulation.
I wrote the following code with the help of this tutorial in order to demonstrate the usefulness of bit masks:
In [40]: img = cv2.imread("baby.jpg")
In [41]: img.shape
Out[41]: (436, 420, 3)
In [42]: circle = np.zeros((436, 420), dtype = "uint8")
Out[42]:
array([[0,0,0, ..., 0,0,0],
[0,0,0, ..., 0,0,0],[0,0,0, ..., 0,0,0],...,[0,0,0, ..., 0,0,0],[0,0,0, ..., 0,0,0],[0,0,0, ..., 0,0,0]], dtype=uint8)
In [44]: bitwiseAnd = cv2.bitwise_and(img, img, mask = circle)
In [45]: cv2.imwrite('result.jpg', bitwiseAnd)
Out[45]: True
In this code snippet, I load in my desired picture:
I create a 436 by 420 pixels plain black image (the same size as my input picture) and draw a 150 by 150 white circle on top of it. This will act as my mask:
Because OpenCV interprets the black-colored pixels as False booleans and the white pixels as True booleans, we’ve created a matrix of booleans that we can compare against the original image with the bitwise AND operation. Every place there is a white pixel in the mask, the pixels of the original image will shine through in the resulting image.
In the ant lab, I am using a strategy similar to this to detect ants in video frames, but using a function called thresholding. This function creates a bit mask by converting the image to grayscale and setting all pixels above a certain darkness value to True, and setting the rest to False. This allows only the darkest pixels to shine through. Since ants are the darkest object in the frame, they should be set to True when we choose a good darkness threshold (not so dark that the ants don’t shine through, but not so light that shadows shine through).
Right now, the current ant detection algorithm converts video frames into greyscale and applies the threshold function to isolate the ants. Then the rest of the tracking algorithm follows the isolated ant around. It works well for some input videos.
But in videos with complicated lighting, this method often mistakes video noise as ants, since the video noise and shadows are as dark as the ants. It tries to track these blobs as ants (producing false paths).
There are lots of incorrectly detected objects circled in green here, with multiple false traces following them.
This is one possible solution that I tried:
My process was to take the original frame,
convert it to HSV colorspace to more easily detect the ants regardless of the lighting,
and create a bit mask for the image targeting the color of the ants. To do this, I created a range of possible colors that the ants might be in this video frame and set every pixel to False if it fell outside of this range.
Then I bitwise AND’d this bit mask over the original frame to target the ant.
Once the ant is isolated in the video frame, we pass it through the same tracking algorithm as before.
Below is a side-by-side comparison of the old detection method and the new detection method working on the same input video clip:
There is a clear difference between the amount of periphery noise being picked up in the first video versus the second, demonstrating the efficacy of this method. But these brief five seconds don’t truly capture the reality of this new method. Although this is much improved from the original version, some noise artifacts are still falsely detected in this version.
Right now, I am working on finding a way to compute the color of the ant from the input videos rather than choosing a frame and selecting it myself. That way this solution could be more applicable to other videos. We could account for changes in lighting and scenery and apply this method to all the video footage that we have. There is room to workshop here, but it is a promising start.
Media Credits:
All images by the author.
Further Reading:
HMC Bee Lab “Do you see what I see? An exploration of computer vision”
http://hmcbee.blogspot.com/2021/07/do-you-see-what-i-see-exploration-of.html?q=threshold
HMC Bee Lab “The Majestic Blobs or: How I Learned to Stop Worrying and Love the Laplacian”
http://hmcbee.blogspot.com/2016/11/the-majestic-blobs-or-how-i-learned-to.html
Learn Learn Scratch Tutorials, Bitwise Operations & Bit Masking
https://www.youtube.com/watch?v=ffPOA7UUDAs&ab_channel=LearnLearnScratchTutorials
Introduction to OpenCV-Python Tutorials
https://docs.opencv.org/3.4/d0/de3/tutorial_py_intro.html
OpenCV, “Changing Colorspaces”.
https://docs.opencv.org/4.x/df/d9d/tutorial_py_colorspaces.html
Pyimagesearch “OpenCV Bitwise AND, OR, XOR, and NOT”
https://pyimagesearch.com/2021/01/19/opencv-bitwise-and-or-xor-and-not/
In [45]: cv2.imwrite('result.jpg', bitwiseAnd)
Out[45]: True
In this code snippet, I load in my desired picture:
I create a 436 by 420 pixels plain black image (the same size as my input picture) and draw a 150 by 150 white circle on top of it. This will act as my mask:
Because OpenCV interprets the black-colored pixels as False booleans and the white pixels as True booleans, we’ve created a matrix of booleans that we can compare against the original image with the bitwise AND operation. Every place there is a white pixel in the mask, the pixels of the original image will shine through in the resulting image.
In the ant lab, I am using a strategy similar to this to detect ants in video frames, but using a function called thresholding. This function creates a bit mask by converting the image to grayscale and setting all pixels above a certain darkness value to True, and setting the rest to False. This allows only the darkest pixels to shine through. Since ants are the darkest object in the frame, they should be set to True when we choose a good darkness threshold (not so dark that the ants don’t shine through, but not so light that shadows shine through).
Right now, the current ant detection algorithm converts video frames into greyscale and applies the threshold function to isolate the ants. Then the rest of the tracking algorithm follows the isolated ant around. It works well for some input videos.
 |
| The timestamp in the top left corner tells us where in the video we are. The two different moving lines are two different tracked ant paths, while the green circle shows the detected ant. The two different traces following the ant are a problem, but not the one I am focusing on right now. As far as detection is concerned, the only object detected is the ant which makes this a successful run. |
But in videos with complicated lighting, this method often mistakes video noise as ants, since the video noise and shadows are as dark as the ants. It tries to track these blobs as ants (producing false paths).
There are lots of incorrectly detected objects circled in green here, with multiple false traces following them.
This is one possible solution that I tried:
My process was to take the original frame,
convert it to HSV colorspace to more easily detect the ants regardless of the lighting,
and create a bit mask for the image targeting the color of the ants. To do this, I created a range of possible colors that the ants might be in this video frame and set every pixel to False if it fell outside of this range.
Then I bitwise AND’d this bit mask over the original frame to target the ant.
Once the ant is isolated in the video frame, we pass it through the same tracking algorithm as before.
Below is a side-by-side comparison of the old detection method and the new detection method working on the same input video clip:
There is a clear difference between the amount of periphery noise being picked up in the first video versus the second, demonstrating the efficacy of this method. But these brief five seconds don’t truly capture the reality of this new method. Although this is much improved from the original version, some noise artifacts are still falsely detected in this version.
Right now, I am working on finding a way to compute the color of the ant from the input videos rather than choosing a frame and selecting it myself. That way this solution could be more applicable to other videos. We could account for changes in lighting and scenery and apply this method to all the video footage that we have. There is room to workshop here, but it is a promising start.
Media Credits:
All images by the author.
Further Reading:
HMC Bee Lab “Do you see what I see? An exploration of computer vision”
http://hmcbee.blogspot.com/2021/07/do-you-see-what-i-see-exploration-of.html?q=threshold
HMC Bee Lab “The Majestic Blobs or: How I Learned to Stop Worrying and Love the Laplacian”
http://hmcbee.blogspot.com/2016/11/the-majestic-blobs-or-how-i-learned-to.html
Learn Learn Scratch Tutorials, Bitwise Operations & Bit Masking
https://www.youtube.com/watch?v=ffPOA7UUDAs&ab_channel=LearnLearnScratchTutorials
Introduction to OpenCV-Python Tutorials
https://docs.opencv.org/3.4/d0/de3/tutorial_py_intro.html
OpenCV, “Changing Colorspaces”.
https://docs.opencv.org/4.x/df/d9d/tutorial_py_colorspaces.html
Pyimagesearch “OpenCV Bitwise AND, OR, XOR, and NOT”
https://pyimagesearch.com/2021/01/19/opencv-bitwise-and-or-xor-and-not/
Very neat, thank you Hayley!
ReplyDelete