Ian Cassidy: When you did a PCA, a principal component analysis, like, it was like beautiful. There was, like, a red circle in the middle of, you know, the blue on purchase, you know, data points. And there were the red purchase ones and they were all clustered together. It was, it was really interesting. And like the, the machine learning model had a really good time trying to predict that the ones in that red cluster where the things that people were were interested in purchasing.
Ginette: I’m Ginette,
Curtis: and I’m Curtis,
Ginette: and you are listening to Data Crunch,
Curtis: a podcast about how applied data science, machine learning, and artificial intelligence are changing the world.
Ginette: Data Crunch is produced by the Data Crunch Corporation, an analytics, training, and consulting company.
If you want to become the type of tech talent we talk about on our show today, you’ll need to master algorithms, machine learning concepts, computer science basics, and many other important concepts. Brilliant is a great place to start digging into these.
The nice thing about Brilliant is that you can learn in bite-sized pieces at your own pace, and with a bit of consistent effort, you can tackle some really tough subjects. With 60+ courses that combine story-telling, code-writing, and interactive challenges, Brilliant helps develop the skills that are crucial to school, job interviews, and careers.
Sign up for free and start learning by going to Brilliant.org slash Data Crunch, and also the first 200 people that go to that link will get 20% off the annual premium subscription.
Now onto our show. We’ve waited to publish today’s episode because Covid has taken a toll on the travel industry and lots of things have changed since we recorded this episode, but there’s good information in this episode, so we don’t want to wait too long to publish it. Hopefully 2021 changes the travel industry’s fortunes and this information becomes even more applicable. So today we chat with Ian Cassidy, former senior data scientist at Upside Business Travel.
Ian: I’m Ian Cassidy. And my interests are in the machine learning optimization realm, since I have experience with that from my grad school days, and a little bit about Upside is we are a travel company, travel management company. We offer a product that is no fees, 100% free. And in fact, if you spend over a hundred thousand dollars booking travel on our website, we offer a 3% cash back, as well as free customer service, 24/7, no contracts. So that’s you sign up with us, no contracts, you get all of this as soon as you sign up. We are a one-stop shop to book and manage all of your travel. In one place, we offer flights, hotels, rental cars, and we also offer expense integration and reporting for companies looking to, to manage all of their, their travelers and, and their expenses for that.
Curtis: Right on. We talked before about the journey that your company has gone through, uh, to figure out how to best use data, you know, how to target and what really works with, with machine learning and things like this. So I’d love to just talk a little bit about that: where you guys started and how you guys made some decisions, what you learned along the way and what you’re, what you’re up to from a data science perspective.
Ian: Yeah, sure. So, uh, you know, like you mentioned, things have changed quite a bit at Upside. We started off as a B2C company where we were targeting what we were calling do it yourself travelers. You did not have to be logged into our site in order to start doing a search and book flights or hotels. So that kind of made it interesting from a data collection perspective. We had like some unique IDs about who the people were that were doing the searching, but it was, it was largely kind of, you know, we didn’t really know much about you when you, when you were searching. So when we started, one of the main things that we were trying to improve upon was our sorting of inventory. So that’s a pretty hot topic and data science for all of e-commerce, so we started with, well, “how do we surface the inventory to the, to the user, to the customer who’s shopping for these things that they are most likely to, to purchase?”
So it was a propensity model. When we first started the, the kind of the legacy sorting algorithms that they had were largely based on heuristics. They knew that in the top few tiles of things that we were presenting to the customer, we want to present the cheapest option. Maybe also the, the shortest duration option, if it was a flight for hotels would maybe the closest option to where you want it to stay. And then maybe also the, the option that, that gave Upside the most amount of profit on the back end. And what you would see was often very confusing, especially with flights, like the first option you might see might be a, a one-stop flight that left at 5:00 a.m. and had a four hour layover where you knew that there was, you know, this is DC to JFK. There’s a shuttle that runs every hour and it’s nonstop.
Why the heck are they showing me this one-stop? So we came in, and we looked at what this pool of customers were searching for and ultimately buying. And thankfully we were collecting all this data. We were collecting all the shopping data, as well as what people were clicking on and what people were purchasing. So we trained some machine learning models, like right out of the box. We were training. We started looking at regression models and then moved into tree models. And obviously the tree models were performing better. And within the first two months of when I started at Upside, and this was back in September of 2017, we trained and productionalized some models that gave us a 25% increase in conversion rate, like almost overnight. We were A/B testing the old models to the new ones. And pretty much within a week, we had statistical significance that we were improving conversion rate. And, you know, obviously that leads to more and more sales and more profits. So that was great. So we did that on the, on the, uh, for our flight inventory, as well as our hotel inventory. So that’s, that’s kind of where we started,
Curtis: What the heuristics we’re missing there. If I understand, right, was just some of the more nuanced things like you were saying, like, why would you give someone this flight that has this four hour layover when, you know, you can just take the shuttle or whatever. The machine learning models we’re then able to, you know, they don’t really understand that, but by seeing what people typically buy, you can catch those types of nuances. Is that what you think attributed to this 25% conversion? Or is that, are there other things . . .
Ian: Yeah, absolutely. I mean, like, it was really, really neat to have this kind of data set because we treated this as a, as a binary, a binary optimization problem, where we had inventory options that were purchased or not purchased. And when we came up with some basic features about the space, so say flights, let’s take flights for, for instance, that was things like stops, price, layover, time of day, that you were leaving time of day, that you were arriving, all these things you can convert into a very nice feature space. A lot of continuous features, also some, some discrete features. And like when you did a PCA, a principal component analysis, like it was like beautiful. There was like a red circle in the middle of, you know, the blue unpurchase, you know, data points. And there were the red purchase ones and they were all clustered together. It was, it was really interesting. And like the machine learning model had a really good time, or a really easy time figuring out to try to predict that the ones in that red cluster where the things that people were, were interested in purchasing. And then we were obviously able to take the, the, the probabilities of purchase, based on the models that we were using, and sort just straight up through the probabilities.
Curtis: That’s awesome. What was harder in that process? Just, I’m curious, was it more sort of conceptualizing what needed to be done? Was it more actually building the models and training them and optimizing them, or was it more the engineering side, making sure everything is orchestrated and works together and production, stuff like that?
Ian: Yeah, I mean, I think the biggest thing is always, is always on like the productionization of these things and how to, how to structure your, your code and how, you know, where you’re deploying, where you’re storing these models, where you’re deploying them, how you’re deploying them. I know there, there are now things like, you know, AWS SageMaker and tools that will like help you turn machine learning models into a deployable API. But we, we were, you know, writing our own, our own, uh, services to, to handle that. And really, we were just trying to figure out where to store these models. And eventually we kind of got a little bit more complex where we were ensembling models together. So, the storage as well as like the feature engineering and transformation, just making sure that things are consistent. And, and, you know, what really helped with that is like, we had a really strong, really great engineering team that helped, helped me, you know, kind of think about like unit tests and how to make sure that you’re testing things from like an engineering perspective that, you know, when you, when you change code that you get the expected results, because, you know, the, these machine learning models are pretty black boxy. And like, if you transform a feature incorrectly, like you, you have to have some way of capturing that.
Curtis: It sounds like you guys were doing this back before, uh, there were, there was, and it’s still hard. Right. But, but there seems to be better tooling now to at least facilitate this process a little bit.
Ian: Yeah. I think so. We we’ve used a couple of those tools. Like, like I said, we’ve, we tried out SageMaker and, and, you know, a couple other things. Some of the models that we still in production we’re largely using services that we’ve built ourselves. So a lot of it is still custom work for us.
Curtis: Okay. Yeah. That’s interesting. And then, you know, so you guys, you know, figured all that out. That’s like a big win for the business, 25%. That’s awesome. Where’d it go from there?
Ian: Yeah. So from, from there, we ended up doing a bunch of A/B testing on more models, as things went along and we were acquiring more customers. They, you know, sometimes we stumbled upon some features that like helped to give, you know, a couple of percentage point improvement, but nothing as significant as the, you know, the, the early days when we went from heuristics to a machine learning model. And since then, we’ve, we’ve pivoted. We pivoted about a little over a year ago to focus on B2C and building out the, the functionality and capabilities for a travel management company that, that would focus on, on B2C type business. So we’ve kind of, we do still have some, some sorting and pricing experiments running in our production environment and in the product, but we’ve, we’ve kind of transitioned a little bit to focusing a little bit more on, on B2C. And also recently, you know, about nine months ago, we signed a huge partnership with a publicly traded travel company called Flight Center to work with them and build a B2C pro platform that they can start using themselves. So things have changed quite a bit. Um, once we signed that partnership,
Curtis: How would you say the, uh, the data science work has has shifted with, with that model shift?
Ian: Yeah. So one of the things that flight center brings to the table that, that helps Upside actually, you know, I talked a lot about, you know, that our product is completely free to use and we offer this cashback is that we are able to leverage Flight Center’s airline contracts, to help sell air travel and make money. So if you think about it, very large companies that travel, like IBM’s or Microsoft or Amazon, that spend like a billion dollars a year on travel, they are able to negotiate their own special corporate discounts with airlines like Delta and American. Similarly, other large companies are able to, they use large travel management companies, like Amex, Global Business Travel, or BCD travel, or even, you know, flight centers and that space as well. And the TMCs are responsible for managing their clients contracts and putting controls in place, such that their employees book the options that help save them the most amount of money, but small companies do not have this luxury.
Ian: And they, you know, companies and only spend 100 or $200,000 a year on travel, or just either booking directly on, on the supplier website or using like something like Kayak or Google flights, so search for their travel. So now that we’re partnered with Flight Center, we have access to all of these airline contracts. And as you can imagine, these contracts are, they’re written by lawyers and they are, there’s a lot of text. There’s a lot of rules. And so one of the, one of the things that we’ve been challenged with and that the data science team here at Upside, and we also have a labs team, an experimental team of a couple of engineers that help us build some productional things and have helped with experimentation, is how do we, how do we programmatically extract the language that are in these contracts and match them back to, to flight inventory that is being returned by our, our shopping service and in order to present to our customers, the things that both save them the most money, because there are discounts and those contracts for the customers. And then we would make commission on the backend if we sell certain, certain flights. But the problem is there’s like hundreds, if not thousands of rules per contract. So one of the, you know, the challenging, but fun things that we’ve been doing lately is around natural language processing to kind of extract some of these contracts and, and build rules engines to, to encode these contracts and, and help, you know, our customers save money and help upside also make money through these backends commissions.
Curtis: And how’s that going? I mean, that sounds like, like a really big, hairy problem to solve. So I’m curious, like, do you find that the current methods and tooling around NLP is sufficient? Or are you making some assumptions in certain areas? Or I’m just curious how, how you’re tackling that.
Ian: Yeah. You know, we, we’ve kind of had to become all of us experts in the flight industry. Like there’s a lot of terminology and it’s, it’s a very complex thing. Like a, when you go to book a flight, like there’s just, there’s just a lot of complexity there that I think, you know, unless you’re like a flight travel nerd, like you, you don’t, you don’t really think about these things. So we’ve kind of had to level all of ourselves up and, and learn about the travel industry. And, you know, flight center has helped us with that because a lot of what’s baked into to this language, you have to be a subject matter expert to be able to understand it and code it. So there isn’t really a whole lot of NLP tooling out there, like out of the box that would, that would help with something like that.
Largely we’ve been doing this a lot of it custom, although, you know what we’re doing now, where we’re trying to extract certain keywords and phrases that are in these lines of text, largely came out of a hackathon project that we, that we had a couple months ago where one of our engineers said, “okay, I want to take some of this text and extract some keywords and just do your basic one hot and code of, of all of the, the words that were in the corpus of all these rules.” And, you know, he had a hand, we have a hand labeled set of like a couple hundred, and we did get, actually get this nice clustering of certain rules that kind of played well together that, you know, these certain keywords meant that this rule was a certain classification. So we then decided kind of, instead of we, we were hand encoding these rules.
So people were like taking a single line of text. Our engineers were taking single lines of text and then writing a function that would map a flight, a flight that someone searched for and determine, “yes, this, this rule set is satisfied by this flight or is not satisfied.” And we’ve started actually auto encoding some of these rules. And a lot of that came out of just this, this hackathon project. So that’s, that’s been pretty interesting, but again, I would say it’s largely, we’re not really building models. It’s a lot of reg X, a lot of custom rules, a lot of decision tables. And, you know, there’s a lot of engineering that goes into that. And again, I’ll just props to our, to our labs team that are helping build this all out. But from a data science perspective, what’s been interesting is like, we’ve been able to kind of step in and think about how do you test this?
How do you validate that what you’re doing is correct. So we’ve kind of been, you know, approaching it and helping those guys from, from that perspective of building validation sets and thinking about how do we randomly sample from, you know, our set of rules in order to figure out, you know, how are we doing? Like, what’s the false positive rate, what’s a false negative rate. And, you know, you were asking a little bit about progress. We’ve, we’ve encoded something like a couple hundred contracts right now for, we’re starting with the big three domestic airline characters. So that’s Delta, United, American. And between the three of them, there’s something like five, 600 contracts. So where we’re making our way through those. And we’ve, we’ve really built some like great deals tooling to handle these generalized rules and, and figuring out, you know, when you, when we see this line of text, is it something that we haven’t seen before, or is it something that we can auto encode and just map to a function that we’ve written?
Curtis: And can you give me some idea of, of the scale of this? ‘Cause I find a lot of times people don’t appreciate, especially from the business world, just how hard a lot of this is to do still. Can you give me a sense for kind of the size of your data science operations and how many people work on this and, and how difficult it is?
Ian: Yeah, I mean, so we, we started working on encoding these contracts and building a system to handle that probably six months ago. And we have a team of three engineers full time, and then we’ve got myself and another data scientist who have kind of been serving as subject matter experts, both in terms of like, you know, testing and validation. And also in terms of like the, the airline industry and how to handle certain attributes of flights and tickets within flights and cabins and all that kind of stuff. And yeah, it’s, it’s, it’s been a huge effort and we have two custom, home-built services that are, that are extracting these parameters and storing them in databases and another, another service that’s kind of serving them up to on the front end to the user and within our product. So yeah, the scale is, it’s a lot, it’s a lot of work.
Curtis: Yeah, yeah. That sounds like it. There’s so much we could dive into here, but one thing I wanted to make sure that we touch on, ’cause I just thought it was so interesting is, uh, you guys have a tool that can help you understand if your flight is going to be delayed. Right. And the way you guys approach that, some of the data you’re using, Dark Sky, things like that. I just, I just think that’s a really interesting thing. So I’d love to dive into that a little bit.
Ian: Yeah, sure. So that, that’s kinda been my pet project for like the last year. We at one point at, at, at Upside wanted to really focus on what we were calling proactive analytics. So how could we do things to kind of anticipate issues and problems that a traveler might experience on a trip, right? Because when you’re traveling for business, like anything that can go wrong will go wrong. Like people, you know, you’re, you’re not, no, one’s a professional business traveler. Like it’s just something you kind of have to get through. And flight delays seemed like a, a pretty like green space and it still, it still is. It’s like, it’s hard to predict when a, when a flight is going to be delayed. There’s, there’s a couple of companies out there that are, that are doing it or claim to do it. So what we did was, so the FAA has a database that’s online.
The airlines actually have to publish their outcome of, of all of their flights. If the flight, you know, exceeds a certain number, if they have the route, I think it’s like a routing thing where if they, if they run this route N number of times they have to publish the results of that, that route throughout the year to the FAA and the FAA has the state going back, I think it’s like, it’s like 10 or 15 years. So there’s like, there’s a couple of gigabytes worth of very, uh, very, it’s a very clean data set. It’s a really great data set for anyone who’s interested in data science and playing around with it to, to try. And I think even Kaggle has, has used this, um, not in a competition, but I think in one of like there, you know, learn how to be a data scientist, um, modules that they have.
And one of the interesting things that we found is that there’s, uh, about 20% of flights are delayed at any time. And of those 20%, like half of them are due to weather-related delays. So that’s 10%. So that’s one in 10 flights are delayed due to weather. And at the time this was about a year ago, a couple of my, a couple of my coworkers who were engineers were like, “Oh my God, like Dark Sky. Like I have this app. It’s, it’s amazing. Like it’s super easy to use and it’s, it’s very accurate.” And it turns out they also have this great API that is fairly cheap. I think it’s, it’s like a dollar per 10,000 calls. The API will also will give you both historical data and predictions. I think they go up to, I think it’s like a week in advance, everything that you would see on the Dark Sky app on your phone through a single end point, it’s a single get request.
So what I did was is I took the FAA data and merged it with the, with the weather data, for the weather-delayed flights. And just, you know, I started, you know, through the whole data pipeline of feature extraction, then looking at training a bunch of different models and hyper parameter tuning and found that it was a, there was a lot of signal there. And from a binary classification performance, I think we were getting upwards of somewhere around 85 to like 90% accuracy on, um, uh, on a balanced dataset w you know, 50/50 delays and, and non delays. And, you know, that that’s much better than, you know, if you were to flip a weighted coin, knowing apriori that 20% of flight flights are, are delayed. So it was something that we thought was super interesting and that we could help with this proactive analytics initiative.
But what we ultimately ended up doing was we, we have a website that is free and open to the public, uh, labs that upside.com/delays. And you can go in, you can put your flight information in there up to three days prior to your flight, and it will give you a probability of delay between, you know, whether your flight is going to be on time 30 minutes to an hour delayed an hour to two hour delayed or two hours plus delayed. And I’ve been running a pipeline to, to retrain models, to kind of, you know, smooth out some of the seasonality, uh, for over a year now. So it gets retrained every, every week on, on, uh, historical data that is, you know, plus or minus a week of when, uh, the, the training date is. And, and we swap out the models on a, on a weekly basis. And I use it for, for my trips. Uh, my, my wife is a consultant. She uses it for her trips, um, and a bunch of people at Upside, or are still using it. And we found that it’s, it’s, it’s pretty useful. It’s a pretty, pretty cool tool to use.
Ginette: A huge thanks to Ian Cassidy for being on our show. As always head to datacrunchcorp.com/podcast for our transcript and attributions.
Attributions
Music
“Loopster” Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
http://creativecommons.org/licenses/by/3.0/
Some of the biggest machine learning gains organizations see is when they initially implement ML models. That was the case for Upside Business Travel. Ian Cassidy, former senior data scientist for Upside, explains how machine learning has helped the company, specifically how it initially increased conversion rate by 25%.
Ian Cassidy: When you did a PCA, a principal component analysis, like, it was like beautiful. There was, like, a red circle in the middle of, you know, the blue on purchase, you know, data points. And there were the red purchase ones and they were all clustered together. It was, it was really interesting. And like the, the machine learning model had a really good time trying to predict that the ones in that red cluster where the things that people were were interested in purchasing.
Ginette: I’m Ginette,
Curtis: and I’m Curtis,
Ginette: and you are listening to Data Crunch,
Curtis: a podcast about how applied data science, machine learning, and artificial intelligence are changing the world.
Ginette: Data Crunch is produced by the Data Crunch Corporation, an analytics, training, and consulting company.
If you want to become the type of tech talent we talk about on our show today, you’ll need to master algorithms, machine learning concepts, computer science basics, and many other important concepts. Brilliant is a great place to start digging into these.
The nice thing about Brilliant is that you can learn in bite-sized pieces at your own pace, and with a bit of consistent effort, you can tackle some really tough subjects. With 60+ courses that combine story-telling, code-writing, and interactive challenges, Brilliant helps develop the skills that are crucial to school, job interviews, and careers.
Sign up for free and start learning by going to Brilliant.org slash Data Crunch, and also the first 200 people that go to that link will get 20% off the annual premium subscription.
Now onto our show. We’ve waited to publish today’s episode because Covid has taken a toll on the travel industry and lots of things have changed since we recorded this episode, but there’s good information in this episode, so we don’t want to wait too long to publish it. Hopefully 2021 changes the travel industry’s fortunes and this information becomes even more applicable. So today we chat with Ian Cassidy, former senior data scientist at Upside Business Travel.
Ian: I’m Ian Cassidy. And my interests are in the machine learning optimization realm, since I have experience with that from my grad school days, and a little bit about Upside is we are a travel company, travel management company. We offer a product that is no fees, 100% free. And in fact, if you spend over a hundred thousand dollars booking travel on our website, we offer a 3% cash back, as well as free customer service, 24/7, no contracts. So that’s you sign up with us, no contracts, you get all of this as soon as you sign up. We are a one-stop shop to book and manage all of your travel. In one place, we offer flights, hotels, rental cars, and we also offer expense integration and reporting for companies looking to, to manage all of their, their travelers and, and their expenses for that.
Curtis: Right on. We talked before about the journey that your company has gone through, uh, to figure out how to best use data, you know, how to target and what really works with, with machine learning and things like this. So I’d love to just talk a little bit about that: where you guys started and how you guys made some decisions, what you learned along the way and what you’re, what you’re up to from a data science perspective.
Ian: Yeah, sure. So, uh, you know, like you mentioned, things have changed quite a bit at Upside. We started off as a B2C company where we were targeting what we were calling do it yourself travelers. You did not have to be logged into our site in order to start doing a search and book flights or hotels. So that kind of made it interesting from a data collection perspective. We had like some unique IDs about who the people were that were doing the searching, but it was, it was largely kind of, you know, we didn’t really know much about you when you, when you were searching. So when we started, one of the main things that we were trying to improve upon was our sorting of inventory. So that’s a pretty hot topic and data science for all of e-commerce, so we started with, well, “how do we surface the inventory to the, to the user, to the customer who’s shopping for these things that they are most likely to, to purchase?”
So it was a propensity model. When we first started the, the kind of the legacy sorting algorithms that they had were largely based on heuristics. They knew that in the top few tiles of things that we were presenting to the customer, we want to present the cheapest option. Maybe also the, the shortest duration option, if it was a flight for hotels would maybe the closest option to where you want it to stay. And then maybe also the, the option that, that gave Upside the most amount of profit on the back end. And what you would see was often very confusing, especially with flights, like the first option you might see might be a, a one-stop flight that left at 5:00 a.m. and had a four hour layover where you knew that there was, you know, this is DC to JFK. There’s a shuttle that runs every hour and it’s nonstop.
Why the heck are they showing me this one-stop? So we came in, and we looked at what this pool of customers were searching for and ultimately buying. And thankfully we were collecting all this data. We were collecting all the shopping data, as well as what people were clicking on and what people were purchasing. So we trained some machine learning models, like right out of the box. We were training. We started looking at regression models and then moved into tree models. And obviously the tree models were performing better. And within the first two months of when I started at Upside, and this was back in September of 2017, we trained and productionalized some models that gave us a 25% increase in conversion rate, like almost overnight. We were A/B testing the old models to the new ones. And pretty much within a week, we had statistical significance that we were improving conversion rate. And, you know, obviously that leads to more and more sales and more profits. So that was great. So we did that on the, on the, uh, for our flight inventory, as well as our hotel inventory. So that’s, that’s kind of where we started,
Curtis: What the heuristics we’re missing there. If I understand, right, was just some of the more nuanced things like you were saying, like, why would you give someone this flight that has this four hour layover when, you know, you can just take the shuttle or whatever. The machine learning models we’re then able to, you know, they don’t really understand that, but by seeing what people typically buy, you can catch those types of nuances. Is that what you think attributed to this 25% conversion? Or is that, are there other things . . .
Ian: Yeah, absolutely. I mean, like, it was really, really neat to have this kind of data set because we treated this as a, as a binary, a binary optimization problem, where we had inventory options that were purchased or not purchased. And when we came up with some basic features about the space, so say flights, let’s take flights for, for instance, that was things like stops, price, layover, time of day, that you were leaving time of day, that you were arriving, all these things you can convert into a very nice feature space. A lot of continuous features, also some, some discrete features. And like when you did a PCA, a principal component analysis, like it was like beautiful. There was like a red circle in the middle of, you know, the blue unpurchase, you know, data points. And there were the red purchase ones and they were all clustered together. It was, it was really interesting. And like the machine learning model had a really good time, or a really easy time figuring out to try to predict that the ones in that red cluster where the things that people were, were interested in purchasing. And then we were obviously able to take the, the, the probabilities of purchase, based on the models that we were using, and sort just straight up through the probabilities.
Curtis: That’s awesome. What was harder in that process? Just, I’m curious, was it more sort of conceptualizing what needed to be done? Was it more actually building the models and training them and optimizing them, or was it more the engineering side, making sure everything is orchestrated and works together and production, stuff like that?
Ian: Yeah, I mean, I think the biggest thing is always, is always on like the productionization of these things and how to, how to structure your, your code and how, you know, where you’re deploying, where you’re storing these models, where you’re deploying them, how you’re deploying them. I know there, there are now things like, you know, AWS SageMaker and tools that will like help you turn machine learning models into a deployable API. But we, we were, you know, writing our own, our own, uh, services to, to handle that. And really, we were just trying to figure out where to store these models. And eventually we kind of got a little bit more complex where we were ensembling models together. So, the storage as well as like the feature engineering and transformation, just making sure that things are consistent. And, and, you know, what really helped with that is like, we had a really strong, really great engineering team that helped, helped me, you know, kind of think about like unit tests and how to make sure that you’re testing things from like an engineering perspective that, you know, when you, when you change code that you get the expected results, because, you know, the, these machine learning models are pretty black boxy. And like, if you transform a feature incorrectly, like you, you have to have some way of capturing that.
Curtis: It sounds like you guys were doing this back before, uh, there were, there was, and it’s still hard. Right. But, but there seems to be better tooling now to at least facilitate this process a little bit.
Ian: Yeah. I think so. We we’ve used a couple of those tools. Like, like I said, we’ve, we tried out SageMaker and, and, you know, a couple other things. Some of the models that we still in production we’re largely using services that we’ve built ourselves. So a lot of it is still custom work for us.
Curtis: Okay. Yeah. That’s interesting. And then, you know, so you guys, you know, figured all that out. That’s like a big win for the business, 25%. That’s awesome. Where’d it go from there?
Ian: Yeah. So from, from there, we ended up doing a bunch of A/B testing on more models, as things went along and we were acquiring more customers. They, you know, sometimes we stumbled upon some features that like helped to give, you know, a couple of percentage point improvement, but nothing as significant as the, you know, the, the early days when we went from heuristics to a machine learning model. And since then, we’ve, we’ve pivoted. We pivoted about a little over a year ago to focus on B2C and building out the, the functionality and capabilities for a travel management company that, that would focus on, on B2C type business. So we’ve kind of, we do still have some, some sorting and pricing experiments running in our production environment and in the product, but we’ve, we’ve kind of transitioned a little bit to focusing a little bit more on, on B2C. And also recently, you know, about nine months ago, we signed a huge partnership with a publicly traded travel company called Flight Center to work with them and build a B2C pro platform that they can start using themselves. So things have changed quite a bit. Um, once we signed that partnership,
Curtis: How would you say the, uh, the data science work has has shifted with, with that model shift?
Ian: Yeah. So one of the things that flight center brings to the table that, that helps Upside actually, you know, I talked a lot about, you know, that our product is completely free to use and we offer this cashback is that we are able to leverage Flight Center’s airline contracts, to help sell air travel and make money. So if you think about it, very large companies that travel, like IBM’s or Microsoft or Amazon, that spend like a billion dollars a year on travel, they are able to negotiate their own special corporate discounts with airlines like Delta and American. Similarly, other large companies are able to, they use large travel management companies, like Amex, Global Business Travel, or BCD travel, or even, you know, flight centers and that space as well. And the TMCs are responsible for managing their clients contracts and putting controls in place, such that their employees book the options that help save them the most amount of money, but small companies do not have this luxury.
Ian: And they, you know, companies and only spend 100 or $200,000 a year on travel, or just either booking directly on, on the supplier website or using like something like Kayak or Google flights, so search for their travel. So now that we’re partnered with Flight Center, we have access to all of these airline contracts. And as you can imagine, these contracts are, they’re written by lawyers and they are, there’s a lot of text. There’s a lot of rules. And so one of the, one of the things that we’ve been challenged with and that the data science team here at Upside, and we also have a labs team, an experimental team of a couple of engineers that help us build some productional things and have helped with experimentation, is how do we, how do we programmatically extract the language that are in these contracts and match them back to, to flight inventory that is being returned by our, our shopping service and in order to present to our customers, the things that both save them the most money, because there are discounts and those contracts for the customers. And then we would make commission on the backend if we sell certain, certain flights. But the problem is there’s like hundreds, if not thousands of rules per contract. So one of the, you know, the challenging, but fun things that we’ve been doing lately is around natural language processing to kind of extract some of these contracts and, and build rules engines to, to encode these contracts and, and help, you know, our customers save money and help upside also make money through these backends commissions.
Curtis: And how’s that going? I mean, that sounds like, like a really big, hairy problem to solve. So I’m curious, like, do you find that the current methods and tooling around NLP is sufficient? Or are you making some assumptions in certain areas? Or I’m just curious how, how you’re tackling that.
Ian: Yeah. You know, we, we’ve kind of had to become all of us experts in the flight industry. Like there’s a lot of terminology and it’s, it’s a very complex thing. Like a, when you go to book a flight, like there’s just, there’s just a lot of complexity there that I think, you know, unless you’re like a flight travel nerd, like you, you don’t, you don’t really think about these things. So we’ve kind of had to level all of ourselves up and, and learn about the travel industry. And, you know, flight center has helped us with that because a lot of what’s baked into to this language, you have to be a subject matter expert to be able to understand it and code it. So there isn’t really a whole lot of NLP tooling out there, like out of the box that would, that would help with something like that.
Largely we’ve been doing this a lot of it custom, although, you know what we’re doing now, where we’re trying to extract certain keywords and phrases that are in these lines of text, largely came out of a hackathon project that we, that we had a couple months ago where one of our engineers said, “okay, I want to take some of this text and extract some keywords and just do your basic one hot and code of, of all of the, the words that were in the corpus of all these rules.” And, you know, he had a hand, we have a hand labeled set of like a couple hundred, and we did get, actually get this nice clustering of certain rules that kind of played well together that, you know, these certain keywords meant that this rule was a certain classification. So we then decided kind of, instead of we, we were hand encoding these rules.
So people were like taking a single line of text. Our engineers were taking single lines of text and then writing a function that would map a flight, a flight that someone searched for and determine, “yes, this, this rule set is satisfied by this flight or is not satisfied.” And we’ve started actually auto encoding some of these rules. And a lot of that came out of just this, this hackathon project. So that’s, that’s been pretty interesting, but again, I would say it’s largely, we’re not really building models. It’s a lot of reg X, a lot of custom rules, a lot of decision tables. And, you know, there’s a lot of engineering that goes into that. And again, I’ll just props to our, to our labs team that are helping build this all out. But from a data science perspective, what’s been interesting is like, we’ve been able to kind of step in and think about how do you test this?
How do you validate that what you’re doing is correct. So we’ve kind of been, you know, approaching it and helping those guys from, from that perspective of building validation sets and thinking about how do we randomly sample from, you know, our set of rules in order to figure out, you know, how are we doing? Like, what’s the false positive rate, what’s a false negative rate. And, you know, you were asking a little bit about progress. We’ve, we’ve encoded something like a couple hundred contracts right now for, we’re starting with the big three domestic airline characters. So that’s Delta, United, American. And between the three of them, there’s something like five, 600 contracts. So where we’re making our way through those. And we’ve, we’ve really built some like great deals tooling to handle these generalized rules and, and figuring out, you know, when you, when we see this line of text, is it something that we haven’t seen before, or is it something that we can auto encode and just map to a function that we’ve written?
Curtis: And can you give me some idea of, of the scale of this? ‘Cause I find a lot of times people don’t appreciate, especially from the business world, just how hard a lot of this is to do still. Can you give me a sense for kind of the size of your data science operations and how many people work on this and, and how difficult it is?
Ian: Yeah, I mean, so we, we started working on encoding these contracts and building a system to handle that probably six months ago. And we have a team of three engineers full time, and then we’ve got myself and another data scientist who have kind of been serving as subject matter experts, both in terms of like, you know, testing and validation. And also in terms of like the, the airline industry and how to handle certain attributes of flights and tickets within flights and cabins and all that kind of stuff. And yeah, it’s, it’s, it’s been a huge effort and we have two custom, home-built services that are, that are extracting these parameters and storing them in databases and another, another service that’s kind of serving them up to on the front end to the user and within our product. So yeah, the scale is, it’s a lot, it’s a lot of work.
Curtis: Yeah, yeah. That sounds like it. There’s so much we could dive into here, but one thing I wanted to make sure that we touch on, ’cause I just thought it was so interesting is, uh, you guys have a tool that can help you understand if your flight is going to be delayed. Right. And the way you guys approach that, some of the data you’re using, Dark Sky, things like that. I just, I just think that’s a really interesting thing. So I’d love to dive into that a little bit.
Ian: Yeah, sure. So that, that’s kinda been my pet project for like the last year. We at one point at, at, at Upside wanted to really focus on what we were calling proactive analytics. So how could we do things to kind of anticipate issues and problems that a traveler might experience on a trip, right? Because when you’re traveling for business, like anything that can go wrong will go wrong. Like people, you know, you’re, you’re not, no, one’s a professional business traveler. Like it’s just something you kind of have to get through. And flight delays seemed like a, a pretty like green space and it still, it still is. It’s like, it’s hard to predict when a, when a flight is going to be delayed. There’s, there’s a couple of companies out there that are, that are doing it or claim to do it. So what we did was, so the FAA has a database that’s online.
The airlines actually have to publish their outcome of, of all of their flights. If the flight, you know, exceeds a certain number, if they have the route, I think it’s like a routing thing where if they, if they run this route N number of times they have to publish the results of that, that route throughout the year to the FAA and the FAA has the state going back, I think it’s like, it’s like 10 or 15 years. So there’s like, there’s a couple of gigabytes worth of very, uh, very, it’s a very clean data set. It’s a really great data set for anyone who’s interested in data science and playing around with it to, to try. And I think even Kaggle has, has used this, um, not in a competition, but I think in one of like there, you know, learn how to be a data scientist, um, modules that they have.
And one of the interesting things that we found is that there’s, uh, about 20% of flights are delayed at any time. And of those 20%, like half of them are due to weather-related delays. So that’s 10%. So that’s one in 10 flights are delayed due to weather. And at the time this was about a year ago, a couple of my, a couple of my coworkers who were engineers were like, “Oh my God, like Dark Sky. Like I have this app. It’s, it’s amazing. Like it’s super easy to use and it’s, it’s very accurate.” And it turns out they also have this great API that is fairly cheap. I think it’s, it’s like a dollar per 10,000 calls. The API will also will give you both historical data and predictions. I think they go up to, I think it’s like a week in advance, everything that you would see on the Dark Sky app on your phone through a single end point, it’s a single get request.
So what I did was is I took the FAA data and merged it with the, with the weather data, for the weather-delayed flights. And just, you know, I started, you know, through the whole data pipeline of feature extraction, then looking at training a bunch of different models and hyper parameter tuning and found that it was a, there was a lot of signal there. And from a binary classification performance, I think we were getting upwards of somewhere around 85 to like 90% accuracy on, um, uh, on a balanced dataset w you know, 50/50 delays and, and non delays. And, you know, that that’s much better than, you know, if you were to flip a weighted coin, knowing apriori that 20% of flight flights are, are delayed. So it was something that we thought was super interesting and that we could help with this proactive analytics initiative.
But what we ultimately ended up doing was we, we have a website that is free and open to the public, uh, labs that upside.com/delays. And you can go in, you can put your flight information in there up to three days prior to your flight, and it will give you a probability of delay between, you know, whether your flight is going to be on time 30 minutes to an hour delayed an hour to two hour delayed or two hours plus delayed. And I’ve been running a pipeline to, to retrain models, to kind of, you know, smooth out some of the seasonality, uh, for over a year now. So it gets retrained every, every week on, on, uh, historical data that is, you know, plus or minus a week of when, uh, the, the training date is. And, and we swap out the models on a, on a weekly basis. And I use it for, for my trips. Uh, my, my wife is a consultant. She uses it for her trips, um, and a bunch of people at Upside, or are still using it. And we found that it’s, it’s, it’s pretty useful. It’s a pretty, pretty cool tool to use.
Ginette: A huge thanks to Ian Cassidy for being on our show. As always head to datacrunchcorp.com/podcast for our transcript and attributions.
Attributions
Music
“Loopster” Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
Some of the biggest machine learning gains organizations see is when they initially implement ML models. That was the case for Upside Business Travel. Ian Cassidy, former senior data scientist for Upside, explains how machine learning has helped the company, specifically how it initially increased conversion rate by 25%.