Weekly deep dives on data management with the engineers and entrepreneurs who are shaping the industry
The Alluxio Distributed Storage System - Episode 70
Distributed storage systems are the foundational layer of any big data stack. There are a variety of implementations which support different specialized use cases and come with associated tradeoffs. Alluxio is a distributed virtual filesystem which integrates with multiple persistent storage systems to provide a scalable, in-memory storage layer for scaling computational workloads independent of the size of your data. In this episode Bin Fan explains how he got involved with the project, how it is implemented, and the use cases that it is particularly well suited for. If your storage and compute layers are too tightly coupled and you want to scale them independently then Alluxio is the tool for the job.
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
Your host is Tobias Macey and today I’m interviewing Bin Fan about Alluxio, a distributed virtual filesystem for unified access to disparate data sources
How did you get involved in the area of data management?
Can you start by explaining what Alluxio is and the history of the project?
What are some of the use cases that Alluxio enables?
How is Alluxio implemented and how has its architecture evolved over time?
What are some of the techniques that you use to mitigate the impact of latency, particularly when interfacing with storage systems across cloud providers and private data centers?
When dealing with large volumes of data over time it is often necessary to age out older records to cheaper storage. What capabilities does Alluxio provide for that lifecycle management?
What are some of the most complex or challenging aspects of providing a unified abstraction across disparate storage platforms?
What are the tradeoffs that are made to provide a single API across systems with varying capabilities?
Testing and verification of distributed systems is a complex undertaking. Can you describe the approach that you use to ensure proper functionality of Alluxio as part of the development and release process?
In order to allow for this large scale testing with any regularity it must be straightforward to deploy and configure Alluxio. What are some of the mechanisms that you have built into the platform to simplify the operational aspects?
Can you describe a typical system topology that incorporates Alluxio?
For someone planning a deployment of Alluxio, what should they be considering in terms of system requirements and deployment topologies?
What are some edge cases or operational complexities that they should be aware of?
What are some cases where Alluxio is the wrong choice?
What are some projects or products that provide a similar capability to Alluxio?
What do you have planned for the future of the Alluxio project and company?