Drones and AI

We are joined by the host of podcast Commercial Drones FM, Ian Smith, who gives us a fascinating understanding of how drones are being used today and in the future. From petri-dish wielding drones that follow whales, to miniature drones working in warehouses, to thermal sensing drones in the mining industry—drones are starting to be used extensively and will continue to grow in the future. We go over the technology, the use cases, the regulations, and the future.

Intro: There’s never been a good way, ever, to get snot from a whale to see how healthy they are or do other types of experiments. It can hover right above the whale as it’s surfacing, and it will just have a little petri dish that when the whale blows it’s blowhole, all the snot just goes on it. Then they bring it back to the boat, and then they analyze it later.

Curtis: One big area that uses AI and will continue to increase use of it is drone technology. One of the big things that machine learning enables drones to do is be aware of its surroundings. Computer vision classifiers help the drones identify objects that it is seeing and take appropriate action, such as avoiding obstacles, performing maintenance recon, and charting autonomous flight paths.

Ginette: Let’s talk to someone steeped in all things drones who can give us insights into drones and how AI currently plays a role and will continue to play a role as drones evolve. This is Ian Smith.

Ian: I got into drones in 2013, but before that I had actually built and flown model aircraft, like RCE aircraft with little tiny gas engines, and the balsa wood, and the glue that you have to wait overnight for it to set, and yeah it was a lot of work, and I wound up flying helicopters for my career, so I’m a commercial helicopter pilot. I was a flight instructor, and I heard in 2013 that RC aircraft that model aircraft had come so far that there was people that were using them. They were calling them drones, and they were taking pictures with them and selling them to people, but it was illegal in the United States because there was no regulation from the FAA at the time.

So of course I decided to get into this as much as I could, since I wasn’t flying at the time, and ever since then in 2013 it’s been my career, and I worked for a company in France called Delair, and today I work for a company in San Francisco where I’m based now called DroneDeploy, and I host a podcast about drones called Commercial Drones FM as a side project.

Curtis: So if you’re looking for more on drones after this episode, go check out Ian’s podcast. He covers all things drone and will keep you up on the latest. Let’s take a broad look at some of the use cases for drones.

Ian: Some of the use cases, some of the industries that are using drones really are . . . agriculture was one that everyone latched on to. The construction industry of course. Inspecting assets, so whether that’s oil and gas or utilities or something else entirely, like wind turbines, or something like that. There’s general land surveyors that use drones for mapping activities, and of course there’s the film and photography. Everybody’s by now has seen a Youtube video of a drone or a drone shot in a movie or TV show. . . . Then there’s the mining industry who use them to calculate volumetrics of stockpiles, and search and rescue for finding people and putting crazy sensors on these drones that can sense thermal signatures.

The way they’re being used, it’s really up to your imagination. Pretty much anything outside that can get a GPS signal these days. They’re going to go towards more indoors things and closed, confined spaces too, so we’re seeing just amazing use cases. People have these incredible imaginations, and the more you ask somebody what would a drone do for you? You just get these awesome responses, and it’s really cool to hear what people come up with.

They’re even using them for wildlife monitoring, so like flying a drone above a whale’s blowhole and capturing the snot that comes out of it literally and then inspecting that. There’s never been a good way, ever, to get snot from a whale to see how healthy they are or do other types of experiments. So a drone is actually a perfect tool to do that. It can hover right above the whale as it’s surfacing, and it will just have a little petri dish that when the whale blows it’s blowhole, all the snot just goes on it. Then they bring it back to the boat, and then they analyze it later.

Ginette: Many businesses are now adopting drones as a typical business purchase.

Ian: As far as adoption goes, companies are now finally starting to have a line item on their budget for drones and drone hardware, software, and pilots, which is something that’s amazing to see because even two years ago, it was still early days, and it was hard to get these companies to start a drone pilot program, and start testing how they can be used in their business, and so what I’ve seen working in the industry and talking to all these folks is that specifically we’ve seen people who have had . . . for example in a construction company, a big construction company, they had one drone and one drone champion, and then they applied the drone data that they collected into their business, and now, just a year later, or eighteen months later, they’re up to 20, 25 different drones, 15 pilots, and it’s just growing from there, so we’re seeing great growth, adoption is still in the early days but it’s coming along, and it’s not really showing any signs of slowing down at all.

Curtis: While regulations for drones have come a long way, there are still some regulations that in Ian’s opinion could stand for a change.

Ian: When I got intro drones that there were zero regulations for commercial drones in the United States. That was in 2013, and then in 2014, the US, the FAA, the regulatory body that oversees drone—so basically you’re flying around in the national airspace system, so in 2014 the FAA finally released something called the Section 333, which was a very arduous process that required usually the consultation of a lawyer, and then a business had to go through and fill out all kinds of forms and specify the pilots and the type of aircraft and all this crazy information. It was was very hard to get one of these section 333 certificates so that you could operate a drone commercially, like literally, for a purpose of furtherance of your business or actually accepting a single penny or even if a buddy bought you a beer for flying their drone over your roof, that’s a commercial operation. As of August 2016, the long awaited part 107 rules were put into effect, and so this is really the first big step I feel was taken by the FAA. It was a great was just a great push by them and that means that right now today in the US I think you have to be 16, don’t quote me on that, but 16 years old, and you can take a written test, which is basically just multiple choice, you can find tons of study guides online. Go to a specific testing center so they’re usually on airports since it’s FAA sanctioned, and you take the test, and if you get a 70 percent or above, then you are a new part 107 certificate holder, which allows you to operate a drone commercially because you’ve shown sufficient aptitude and knowledge that you know how these things work and safety procedures and where you can fly them and where you can’t fly them, and so the regulatory environment is in a very good spot, but there are still three remaining regulatory blockages, if you will, that are stopping even more cool use cases from coming out, and the first one is night operations, so right now you can’t fly a drone at night commercially. You have to get something called a waiver, which you have to apply for with the FAA, and then there’s one called beyond visual line-of-sight which means that right now the drone pilot still has to maintain visual line-of-sight with the drone, so if you’re standing on the ground, you have to be able to see it wherever it is. If it flies behind a building, you just broke that rule. So imagine if you want to have an autonomous drone that lives in a box somewhere on a construction site and everyday it just flies, and you don’t have to if you don’t have a human there, well beyond visual line of site not being regulated, can’t do that. That’s still kind of holding the autonomy of drones back, and then the third one is operations over people, so flying over people, that’s illegal as well right now. Unless you have a waiver, you can’t do that, so there’s a few regulations that still need to be well thought-out and of course is good reason because his safety is the biggest concern and congested airspace, so overall looking very positive. I’ve seen tons and tons of progress, but there’s still some ways to go before really seeing the value of drones.

Ginette: Regulations for hobbyists are a little different than for commercial ventures.

Ian: If you’re a consumer, you have to register your drone, so there was a point where you didn’t have to register your drone. Then you did. Then the FAA took it back, and now there’s, you have to register your drone again, so there’s actually a class action lawsuit against the FAA for this that’s happening right now. But for five bucks, you register your drone, and you give them . . . I forgot what kind of information you have to give them, You get a registration number that you have to put on your drone, and then that’s pretty much it. You don’t have to follow the same rules as commercial. It’s a little bit less strict, but in some ways it is more strict, you can’t access certain airspaces, so it’s still fairly easy to get a drone and fly it for fun as a hobbyist is what we call them. You get much more value from it, I believe, from the commercial side.

That’s where you start getting into the privacy and who owns the airspace over the house. It’s such a big debate on what you’re allowed to do. Should people own the airspace over their house because technically you don’t, unfortunately, but if I’m hovering my drone right over my house or my head, is that legal? Should you have some type of protection there? There are certain laws at the state level, but there’s this whole battle between state and federal, and it goes really, really deep, so I don’t want to opine too much about that dichotomy of state versus federal and privacy and such. It’s a little more lenient for hobbyists.

Curtis: Ian mentioned earlier that he works for a company called Drone Deploy, so he sees a lot of the latest tech and software, and how AI is continuing to integrate with drones.

Ian: It automatomates the flight of the drone so you don’t even have to play with those old joysticks on the transmitter. You just press a button on your smartphone that’s connected to the drone, and then the drone literally flies itself.

So as long as you can see it, it’s fine. There is autonomy happening there, and it’s using just the various sensors on the drone, like the GPS and the compass and the borometer that’s on there that can tell altitude and everything, and so all the accelorometers etc. So there is plenty of autonomy happening already. It’s just in a very sanctioned environment where you can see and take control of the drone if it’s going to do something bad like fall somewhere or if it exhibits behavior, of maybe crashing or something.

Ginette: While AI has a lot of future potential with drone, you’ve probably already seen this splice at work.

Ian: Right now the main area that we’re seeing that in is what I call entertainment drones. So Intel. If you’ve ever seen a drone light show in the news yet, if you’re watching the 2018 Olympics Winter Olympics in pyeongchang, Intel did a bunch of different swarming drone flights. They also did some for the Super Bowl this past Super Bowl. There were swarming drones, and a bunch of drones flying together to accomplish a similar task, and yeah, the technology is there, but the operators themselves actually have to get a waiver, the Intel pilots, if it’s outdoors in the FAA airspace because you’re supposed to only have one drone pilot per drone, so there obviously not going to have 500 pilots out there. I mean, the technology does not require that. The technology is there, and it works really, really well, but they still have to apply for these things called waivers that I was mentioning earlier from the FAA in order to to make that happen in the very kind of clothed environment so swarms of drones will be doing really cool things later on, further in a future, maybe few years, maybe up to seven years out where I imagine they’ll be doing collaborative mapping tasks so one drone can only cover so much ground, but ten drones can cover a lot more ground, and they can work in unison and do some pretty cool things in data gathering and such.

Curtis: It’s really interesting to see how Ian foresees drones using AI in the near future.

Ian: Right now drones are just mainly used as cameras. These remote sensing devices that can be placed anywhere in 3D space, which is really really useful for a lot of things, but that is like vast majority of uses of drones today. If you look at any of them, they have on their payloader, their sensor that they have on them is a camera, but you know certain sensors have gotten quite sophisticated and so I think as we move forward in the future, we’ll see more complicated and more interesting sensors. Today we see like the standard RGB camera, just like the same thing you have in your cell phone. We’ve got some really interesting ones that do multispectral data, and they capture data in different kind of spectrums of light so that’s useful for vegetation and kind of exploration for oil and gas. Hyperspectral sensor, lidar, the same kind of things in self-driving cars.

You’ve got some really interesting ones like methane gas detection, and of course you’ve got thermal for detecting temperature from a distance, but I think, what’s really interesting to me in five years, is not just like these remote sensing cameras and different sensors that will be on the drones, cuz right now drones are, they’re capturing data and then maybe they’ll have some on-board compute, or they’ll send something to the cloud for computing and then automatically detect “oh this is corrosion or damage.” Within 5 years I think it’s feasible to say that drones might be not just actually capturing the data and diagnosing it, but actually going and fixing the problem, so if a drone senses that there’s rust on some type of asset, maybe it’s a wind turbine, maybe the next step to do after its inspection is to go down and maybe swap its payload over to some anti-corrosion paint or something like that, return to the same spot and apply it to the same area, and so that way, it’s kind of like this full 360 degree, holistic drone autonomous operation that can actually truly deliver crazy gains instead of just giving you a report on what a human needs to go do now. So they’re robots, and they need to do robot things, that are dull, dirty, and dangerous.

Ginette: Drones have been used a lot outside, but there’s a huge potential for them being used indoors.

Ian: The sizes of drones, I mean, they can be anything. As far as on the small spectrum, there’s plenty of drones that if you just have like your outstretched palm, that could just land right in the center of your palm and, I might, not even the footprint of it would go outside of the area of your palm. Today, those drones aren’t usually that useful. They’re usually used more for fun, but the amount of data you can capture with a camera is actually really interesting, so if you think about indoor operations in a warehouse, you’re not going to need this huge drone that’s flying around, and when I say huge, I mean relative to the size a drone that can fit in the size of your palm that weighs nearly nothing. You can use this small drone that might be connected to the network that’s inside the warehouse, and it can fly around, and maybe it can scan QR codes or bar codes and just make sure that there’s the stock that remains in a certain area, or solve all sorts of logistical problems, so those will probably used more indoors where the elements like wind aren’t really a factor because they can be really affected by high winds whenever they’re that small, but if you look on the other end of the spectrum, to really large drones, I mean, drones could be any size flying around, so a drone could be the size of a 787, a big, huge Antonov type of airplane, like the biggest in the world, cuz they do have autopilot so technically they’re kind of drone like, but specifically Facebook has a project called Aquella, I think it’s supposed to be pronounced “A-key-la,” but apparently everyone says “a-qwee-la,” anyways, they have something, it’s a high altitude long endurance drone, so it’s it’s it’s kind of an acronym, you just pronounce it as HAIL, high altitude, long endurance, and they’re so high up that they fly months at a time, and you don’t even ever see them, but Facebook’s for example is going to be beaming down internet to areas. And so drones can be really any size, and they can do so many different things than just stick a camera on them. I’m really fascinated with how they can affect the physical world.

Ginette: Applications of AI can range many depths, and Ian sheds some light on the evolution of AI in drone tech.

Ian: The most exciting part is just the fundamental paradigm shift that these little flying robots can provide society and improving the way we can work and interact with the environment specifically like the kind of evolution of AI. It’s been a buzzword for a while in drones, and it’s like this kind of inherent quality that they’re already supposed to have, and they’re using them in different levels, and so I kind of look at like two different phases like onboard computing like edge computing and cloud computing as well, and so drones are going through this evolution of AI where the first step was to stick these cameras on the drone to help them not run into something. So basically collision and obstacle avoidance. And that first step was identify I might hit that object. That is something I should avoid. The second step was connecting that okay, I should avoid this object to the autopilot so the drone makes its own decision to not hit it. And then the third step is okay, I’ve identified I shouldn’t hit and I’m not going to hit it, but the next step is where can I go? And can I go around it? And step four being thinking and planning ahead even further. How far can I predict I can hit something whenever I’m flying in a specific direction, and then you take that, and you apply that to a lot of different types of commercial jobs. Imagine if you just tap on a building on the feed of the FPV stream of the drone, and then the drone can actually just okay, I know this is a building because I’ve been trained and my artificial intelligence tells me that and then I can fly around and get all the perfect shots that I might need from the camera to do a really thorough inspection for that building and then identify those issues, and then in the future what really excites me as I mentioned before is the drones that are going to fix those problems, and the ones that are going to be living in a box on a network of drones that are dispersed around a city and so if I’m a construction company and I have a a construction site, a new site in the city, I don’t own drones anymore. I don’t really need my own drones. I can just tie into the city’s network of drones, log on, and say “Hey K, I want a 3D high resolution point cloud every single Tuesday at 8:30 a.m. in the morning, and I want these people to be notified, and if there’s this type of issue, go ahead and send this kind of robot out to fix it.” That’s kind of where we’re headed with all this convenience and networking and the AI and so the autonomy is really going to play a big part about that, and that’s what really gets me excited about the industry still.