Cover art for podcast The Python Data & Science Podcast.__init__

The Python Data & Science Podcast.__init__

100 EpisodesProduced by Tobias MaceyWebsite

The podcast about how the Python language powers work in data and science

50:59

Monitor The Health Of Your Machine Learning Products In Production With Evidently

Summary

You’ve got a machine learning model trained and running in production, but that’s only half of the battle. Are you certain that it is still serving the predictions that you tested? Are the inputs within the range of tolerance that you designed? Monitoring machine learning products is an essential step of the story so that you know when it needs to be retrained against new data, or parameters need to be adjusted. In this episode Emeli Dral shares the work that she and her team at Evidently are doing to build an open source system for tracking and alerting on the health of your ML products in production. She discusses the ways that model drift can occur, the types of metrics that you need to track, and what to do when the health of your system is suffering. This is an important and complex aspect of the machine learning lifecycle, so give it a listen and then try out Evidently for your own projects.

Announcements
  • Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • Your host as usual is Tobias Macey and today I’m interviewing Emeli Dral about monitoring machine learning models in production with Evidently
Interview
  • Introductions
  • How did you get introduced to Python?
  • Can you describe what Evidently is and the story behind it?
  • What are the metrics that are useful for determining the performance and health of a machine learning model?
    • What are the questions that you are trying to answer with those metrics?
  • How does monitoring of machine learning models compare to monitoring of infrastructure or "traditional" software projects?
  • What are the failure modes for a model?
  • Can you describe the design and implementation of Evidently?
    • How has the architecture changed or evolved since you started working on it?
  • What categories of model is Evidently designed to work with?
    • What are some strategies for making models conducive to monitoring?
  • What is involved in monitoring a model on a continuous basis?
  • What are some considerations when establishing useful thresholds for metrics to alert on?
    • Once an alert has been triggered what is the process for resolving it?
    • If the training process takes a long time, how can you mitigate the impact of a model failure until the new/updated version is deployed?
  • What are the most interesting, innovative, or unexpected ways that you have seen Evidently used?
  • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Evidently?
  • When is Evidently the wrong choice?
  • What do you have planned for the future of Evidently?
Keep In Touch Picks Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Educational emoji reaction

Educational

Interesting emoji reaction

Interesting

Funny emoji reaction

Funny

Agree emoji reaction

Agree

Love emoji reaction

Love

Wow emoji reaction

Wow

Listen to The Python Data & Science Podcast.__init__

RadioPublic

A free podcast app for iPhone and Android

  • User-created playlists and collections
  • Download episodes while on WiFi to listen without using mobile data
  • Stream podcast episodes without waiting for a download
  • Queue episodes to create a personal continuous playlist
RadioPublic on iOS and Android
Or by RSS
RSS feed
https://www.pythonpodcast.com/feed/mp3/

Connect with listeners

Podcasters use the RadioPublic listener relationship platform to build lasting connections with fans

Yes, let's begin connecting
Browser window

Find new listeners

  • A dedicated website for your podcast
  • Web embed players designed to convert visitors to listeners in the RadioPublic apps for iPhone and Android
Clicking mouse cursor

Understand your audience

  • Capture listener activity with affinity scores
  • Measure your promotional campaigns and integrate with Google and Facebook analytics
Graph of increasing value

Engage your fanbase

  • Deliver timely Calls To Action, including email acquistion for your mailing list
  • Share exactly the right moment in an episode via text, email, and social media
Icon of cellphone with money

Make money

  • Tip and transfer funds directly to podcastsers
  • Earn money for qualified plays in the RadioPublic apps with Paid Listens