"In god we trust, all others bring data." - William Edwards Deming
That's what was written on our slides. And then the customer used it against us.
This was a logistics start-up, a marketplace for shipments. I want three palettes delivered the day after tomorrow from Hamburg to Munich. Shippers bid.
We replaced their pricing engine. A big consultancy (pricing experts) sold them an Excel sheet for half a million. They hired a PHP-guy who hard-coded the parameters in their website. In two days we built an alternative and could show that we outperform by reducing prediction errors by 20%. We used a random forest with H2O and built a workflow that would retrain the model every week and deploy it as a Docker Service. Besides, we built a Redshift warehouse and PowerBI dashboards to monitor all sorts of KPIs. Then we built a recommender.
But then we realized that their auction process was broken. It provided the wrong incentives to both the buyer and the seller. We also understood how to it. And the pricing engine we built was not part of the solution. They needed a Vickrey (or second-price) auction. Thats how Ebay or Google Ads operate, that's how they are successful, that's how they got big. There is mathematical proof that Vickrey is optimal. We said, ask the experts. But we failed to convince our customer. Logistics is different they said. Prove it with data (because Deming) they said.
In this case we failed. Well, unfortunately it didn't work out for that start-up. Lesson learned: It's your customer who decides. No reason to get upset.