Cover art for podcast Data Engineering Podcast

Data Engineering Podcast

122 EpisodesProduced by Tobias MaceyWebsite

Weekly deep dives on data management with the engineers and entrepreneurs who are shaping the industry

37:12

Snorkel: Extracting Value From Dark Data with Alex Ratner - Episode 15

Summary

The majority of the conversation around machine learning and big data pertains to well-structured and cleaned data sets. Unfortunately, that is just a small percentage of the information that is available, so the rest of the sources of knowledge in a company are housed in so-called “Dark Data” sets. In this episode Alex Ratner explains how the work that he and his fellow researchers are doing on Snorkel can be used to extract value by leveraging labeling functions written by domain experts to generate training sets for machine learning models. He also explains how this approach can be used to democratize machine learning by making it feasible for organizations with smaller data sets than those required by most tooling.

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
  • When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
  • You can help support the show by checking out the Patreon page which is linked from the site.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • Your host is Tobias Macey and today I’m interviewing Alex Ratner about Snorkel and Dark Data
Interview
  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by sharing your definition of dark data and how Snorkel helps to extract value from it?
  • What are some of the most challenging aspects of building labelling functions and what tools or techniques are available to verify their validity and effectiveness in producing accurate outcomes?
  • Can you provide some examples of how Snorkel can be used to build useful models in production contexts for companies or problem domains where data collection is difficult to do at large scale?
  • For someone who wants to use Snorkel, what are the steps involved in processing the source data and what tooling or systems are necessary to analyse the outputs for generating usable insights?
  • How is Snorkel architected and how has the design evolved over its lifetime?
  • What are some situations where Snorkel would be poorly suited for use?
  • What are some of the most interesting applications of Snorkel that you are aware of?
  • What are some of the other projects that you and your group are working on that interact with Snorkel?
  • What are some of the features or improvements that you have planned for future releases of Snorkel?
Contact Info Parting Question
  • From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Educational emoji reaction

Educational

Interesting emoji reaction

Interesting

Funny emoji reaction

Funny

Agree emoji reaction

Agree

Love emoji reaction

Love

Wow emoji reaction

Wow

Listen to Data Engineering Podcast

RadioPublic

A free podcast app for iPhone and Android

  • User-created playlists and collections
  • Download episodes while on WiFi to listen without using mobile data
  • Stream podcast episodes without waiting for a download
  • Queue episodes to create a personal continuous playlist
RadioPublic on iOS and Android
Or by RSS
RSS feed
https://www.dataengineeringpodcast.com/feed/mp3/

Connect with listeners

Podcasters use the RadioPublic listener relationship platform to build lasting connections with fans

Yes, let's begin connecting
Browser window

Find new listeners

  • A dedicated website for your podcast
  • Web embed players designed to convert visitors to listeners in the RadioPublic apps for iPhone and Android
Clicking mouse cursor

Understand your audience

  • Capture listener activity with affinity scores
  • Measure your promotional campaigns and integrate with Google and Facebook analytics
Graph of increasing value

Engage your fanbase

  • Deliver timely Calls To Action, including email acquistion for your mailing list
  • Share exactly the right moment in an episode via text, email, and social media
Icon of cellphone with money

Make money

  • Tip and transfer funds directly to podcastsers
  • Earn money for qualified plays in the RadioPublic apps with Paid Listens