Hack for the Sea

The weekend before last, Ocean Alliance participated in the 3rd annual Hack for the Sea hackathon in Gloucester, MA. The hackathon, led by organizer and Ocean Alliance IT guru Mark Henderson,  provides various ocean-related challenges for local hackers to solve in 48 hours.

The challenge provided by Ocean Alliance: Can you identify a whale by its blowhole?

Some background: Photo identification of whales was pioneered by Ocean Alliance founder Roger Payne back in the 1970s. Photo ID forms the backbone of whale research all across the planet. One of the most common forms of photo ID involves identifying a whale based upon the pattern on its fluke, or tail. However, there are some problems with identifying a whale based on this pattern.

1. You need to be in the right place, behind the whale, to get the correct angle.
2. Many species of whale infrequently lift their tails out of the water.
3. Even within species that do frequently lift their tails out of the water, many individuals do not.

So the major problem with this method is that not all whales fluke, and even the whales that do fluke don’t do it 100% of the time. But what do whales do 100% of the time? They breath, and their blowhole always comes out of the water, and there’s nothing more familiar to Ocean Alliance than a whale’s blowhole thanks to SnotBot™. Also, thanks to SnotBot, we have a huge repository of great videos and pictures of blowholes from dozens of whales, across multiple species, from Alaska to Gabon, of all shapes and sizes and everything in between.

So again, our challenge for the hackers of Gloucester was to determine if and how we can ID whales based on their blowholes. A daunting challenge indeed with only 48 hours to come up with a solution. A small team of five whale loving hackers (including a 10 year old graphic designer) self-organized to tackle this project, and what they were able to develop in such a short time with limited data was really impressive. They built a framework to extract still frames from our SnotBot video footage, pass those still images into a testing and a training set to create a convolutional neural network model, and then predict the unique identifier of a whale given an input image. Basically they created a program that uses technology similar to facial recognition and were able to apply it to blowholes! One of the most difficult parts is having to “train” the neural network to understand what is and what isn’t a blowhole. This requires feeding it a few dozen positive pictures of blowholes and probably thousands of images of things that aren’t blowholes that appear in the video footage, like birds, seaweed, floating debris, or reflections on the water. And not only did they have to create this huge data set for training and testing, but the training requires a large amount of computing power and time, so the team worked non-stop all weekend.

All the hard work paid off, though, because the team won best in show for their project dubbed Whalematch! Ocean Alliance is very excited about this proof of concept project and plans to continue its development. The hackathon team identified a list of things to make Whalematch 2.0 even better, including even larger datasets of negative and positive blowhole images, an automatic method for generating the dataset, and investigation into how we can improve results by using blowhole movement time signatures from video streams (basically the blowhole looks different as it opens and closes).