In the last few years, the combination of increasing compute power, big data and improved algorithms have catapulted artificial intelligence from research labs to enabling a multitude of innovative, practical applications. Robotic surgery, for example. A hurricane forecasting system for the Caribbean. Personalizing classroom lessons so kids can get the best education. Predicting what are the best leads for salespeople.
Using AI to visually identify the presence of great white sharks near popular beaches might not immediately come to mind, but a few members of the Salesforce Einstein team — in partnership with UC Santa Barbara’s Benioff Ocean Initiative — are doing that off the coast of Southern California. This handful of AI experts and oceanographers is using AI to analyze drone footage in near real-time to track an apparent recent uptick in great white sightings close to shore. They hope this application of AI can help keep beaches safer, encourage a more informed public, and help people learn how to share the waves with wildlife.
And perhaps most importantly, they can protect the rebounding populations of great white sharks, top predators that are a healthy part of the California ocean ecosystem, according to Benioff Ocean Initiative Director and UCSB professor Douglas McCauley.
Over the past four years, scientists have noted an increase in great white sightings off the California coast — particularly 5- to 10-foot-long juveniles. There are many theories as to why this is: climate change; the effects of 25 years of protected species status in California; banning gill nets in nearshore waters that indiscriminately kill marine life, including sharks; an increase in the population of seals and sea lions, which are a key food source for adult sharks, to name a few potential factors.
The Salesforce Research team theorized they could train the Einstein Vision algorithm to process drone video and recognize the unique attributes of sharks
But recent sightings near popular surfing areas in Santa Barbara were enough to raise the interest and concerns in the local community. McCauley teamed up with Salesforce Director of Research Michael “MJ” Jones, Salesforce Chief Scientist Richard Socher and San Diego State Associate Professor of Computer Science Xiaobai Liu to devise a new way to identify the presence of sharks in coastal waters using artificial intelligence.
The Salesforce Research team theorized they could train the Einstein Vision algorithm — ordinarily used for visual search, brand detection, and product identification — to process drone video and recognize the unique attributes of sharks, thereby identifying them without the need of physically tagging them. Working with McCauley’s team, they devised an innovative workflow for video capture and processing, all without ever touching the water. They dubbed it “Project SharkEye.”
The first step was collecting huge quantities of video data off the coast through drone technology. That video would be the basis for building the AI models used to identify a great white shark. McCauley and his team worked with UCSB students, who piloted drone flights off the beach, filming a 10-mile stretch of the coastline.
“The benefit of drones is that you’ve got an eye in the sky to see more of the ocean,” explains McCauley. “It’s super cheap to fly up and down the beach. That’s the easy part. But what comes after is where AI becomes really valuable. Every hour of video footage requires an hour of a video analyst’s time. If you’re trying to gain insights in real time, that’s not efficient. That’s where computer vision really comes into play.”
With sufficient video data collected, the Salesforce Einstein team worked with the marine biologist teams at UCSB to train the algorithm on a number of factors. It had to learn, for instance, to differentiate between shark species, such as a leopard shark (which is harmless to humans) and a great white. It had to rule out non-shark objects, such as a large floating piece of kelp with a shark-like shape, and determine whether a sighting was a new shark or a second sighting of a shark already counted. With each sighting, the algorithm got smarter.
“The way we explain it is that it’s almost exactly like how kids learn,” says Einstein Project Manager Zineb Laraki, one of the project’s leads. “You tell them, ‘That’s a cat.’ Eventually, kids can generalize and start to understand the concept of a cat — what it looks like, how it moves, predicting what it will do based on watching it. It’s the same thing with the algorithm. You show it examples. You let it make a determination. Right or wrong, you give it feedback.”
Beyond the technology, the Salesforce and BOI teams wanted to get buy-in from a community that bases its livelihood on the coast, such as surf shops and schools, local businesses, and public safety officials. That’s why the extended plan is to offer an adapted version of the Salesforce Field Service Lightning mobile app — typically used for onsite job management, providing real-time collaboration and access to information such as scheduling and knowledge bases — to local stakeholders like lifeguards, surf school teachers, and marine biologists. Those “field workers” will be able to access the collected information, such as how many sharks were spotted at any given time and where. The rationale is that by providing such tools that involve the community in the process, they’ll trust the AI technology as a solution.
For us, it was a fantastic opportunity to apply AI to not only capture data of value to shark researchers, but to also be able to try to be helpful to our community by sharing that data.
Benioff Ocean Initiative Director and UCSB professor Douglas McCauley
“This application is unique because we’ve had the privilege of building it hand in hand with our community partners,” explains McCauley. “For us, it was a fantastic opportunity to apply AI to not only capture data of value to shark researchers, but to also be able to try to be helpful to our community by sharing that data.”
It’s also about understanding a traditionally misunderstood species — where they’re swimming and why, their behavior and feeding patterns. If we’re able to give these creatures their space and predict when they might come into contact with people, McCauley says, we’ll be better prepared to keep beaches safe, too. Not to mention the sharks, who are still a “vulnerable” species — one step up from “endangered” status. The end use case of such technology isn’t just a practical use of AI — although it is that, and an undeniably cool one. It’s also a way to bring technology to a place it hasn’t traditionally been: the beach.
“Lifeguards do a superb job of protecting all of us when we head to the beach,” says McCauley. “But they are constantly on the lookout for next-gen tools and technologies to help them better assess the situation at their beaches. AI could very well be one of those new tools.”
Perhaps in the near future, McCauley suggests, lifeguards could be aided by AI-powered autonomous drones installed at every tower, and those drones could make regular flights and deliver data back to the tower. That opens up a whole universe of knowledge — for instance, what is the status report is on the day for shark sightings? Are the waters pollution free? Are there any dangerous riptides forming?
“AI is doing amazing, audacious things and inspiring customers to take that leap,” says Jones. “But there’s lots of reluctance to actually dream when it comes to AI. Real-world applications like this one can help drive actual impact in the world.”
To find out more about Salesforce’s AI research, check out https://einstein.ai/
