The European Conference on Machine Learning, Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) is one of the largest regional machine learning conferences in the world, and this year it was hosted in beautiful Vilnius. Meaning that the latest advances in machine learning were accompanied by a spectacular backdrop and perhaps the best food I have seen at a conference anywhere.

Compared to its international peers, ECML PKDD has a primarily academic focus, with only a small number of sponsors and industry talks. In 2022 Criteo was a sponsor of ECML in Grenoble, in 2024 I was the only Criteo attending, but I was very pleased to share an industry talk: “Why the shooting in the dark method dominates recommender systems practice”.

As usual the keynotes were a highlight. Mounia Lalmas-Roelleke’s keynote covered Spotify’s personalization engine. It is always good to learn from comparable companies about the algorithms and strategies that they use to drive massive scale personalization. It was particularly interesting to hear how Spotify were incorporating graph neural networks (GNNs) and generative AI into their service. Deep reflection is always required in order to consider how these ideas might transfer to Criteo’s engine, but it was an inspiring talk all the same.

Moritz Hardt also delivered a very interesting keynote on the science of benchmarks. He made many insightful observations particularly regarding ImageNet, and an alternative that they developed amusingly called “ImageNot”.

Interestingly, both ImageNet and ImgaeNot preserved the same ordering for state-of-the-art algorithms in computer vision. While I have no doubt that these insights hold for computer vision, this talk actually provided a useful point of contrast for my talk on the “shooting in the dark” method. Recommendation and other machine learning that have a causal component are not so well served by standard benchmarks, and this has, in my view, resulted in an unhelpful discrepancy between academic work and work that can usefully be employed in industry.

This is a deep point which at Criteo we are doing our best to share with the community. For example Criteo’s own Flavian Vasile is one of the organizers of the CONSEQUENCES workshop at RecSys ’24 on next month which highlights this issue from a RecSys perspective.

There were several other highlights. The keynote, Gintarė Karolina Džiugaitė, made the intriguing point that it may be impossible to achieve generalization in models without some memorization of training data. This theoretical observation has potentially profound impacts on machine learning in privacy preserving contexts including Google’s Privacy Sandbox for Chrome.

There were too many interesting contributed papers to list, but I would highlight the paper “DISCO: An End-to-End Bandit Framework for Personalised Discount Allocation”, which developed a really clean industry project incorporating Bayesian linear models, Thompson sampling, constrained optimization, and real A/B testing.

Finally, I was very pleased to share my own talk “Why the shooting in the dark method dominates recommender systems practice”. Making progress in recommender systems requires us to better align the academic problems were we focus study and real impactful research. This talk attempts to illuminate this problem and what I think are promising directions to pursue.

If you missed the talk, there will be a longer follow-up session in our new Webinar/Podcast 👇

AI in Industry and Academia Conversations - Criteo AI Lab

ECML PKDD 2024 Comes to Lithuania was originally published in Criteo Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.