Data Science Consulting- When Algorithms Work Too Well

Data Science Consulting Sometimes Predicts Too Well

In data science consulting, we enjoy sharing stories of success and failures. It helps us learn, and gain further insight to how to implement data science processes and policies (the hardest part of the job). One such example a data science project that went a little too well involves a telecommunication company.

About 5 years ago, a telecommunications company wanted to be able to predict whether a customer would cancel their service or not. An example of a great starting project and question to work on for a new data science team.

The executives loved the idea when the project manager pitched it. Who wouldn’t! It makes a lot of sense. If you can catch a customer right before they fall out and convince them to stay on your plan, then you keep revenue. Thus, a bigger bonus for the executives, or higher dividends for shareholders.

So the analysts and data scientists were set off to races on developing a model that could help the company better predict when customers were planning to quit their service.

After about a year and a couple hundred thousand dollars worth of resource hours the team had developed a successful model. It was 85-90% accurate. They then worked with process managers, and customer service teams to develop playbooks and guides to help approach customers who may not be interested in the service. A few more hundred thousands of dollars dropped into the project. Once they felt like the algorithm was set and the team members were trained they went forward to implement the project.

On the first day, the customer service representatives were given a list of people to call and check up on. They were supposed to figure out how they could keep the customer happy. A great motive.

Data Science Project in Action

The first call, according to the customer rep went a little like this:

Customer Rep. : “Hello, my name is Joe. I work for ACME Telecommunications. We are working to better appeal to our customers and we were wondering if you were having any problems with our service.

Customer: Yes, in fact. I have been meaning to call you guys for the last two months to cancel my service. Thank goodness you called, since I have you on the line anyways. Can you help me cancel my service.

A good 30-40% of the calls were like this one. The algorithm worked amazingly well. It was actually predicting people who wanted to cancel the service. The only problem was, some people were determined to cancel the service and it ended up in a huge loss of possible monthly charges that might have not been cancelled for another 3-6 months. This lead to a loss in profit, and a cancellation of the project. In the end, the project was a successful failure.

Conclusion

Sometimes, algorithms do exactly what you tell them to. Yet, the customers responded very differently. One of the common statements the Go Masters that faced off against Google’s Alpha Go said was that they had never faced an opponent that thought the way a human did. That’s just it. Humans don’t always act the way we calculate. We can try, but it’s not always perfect. So when implementing a new data driven strategy, it is important to incorporate the human aspect. Don’t let the numbers and metrics lead away from the human factor.