WHY IT MATTERS: Digital Transformation

This frequently-updated page contains real-life examples and diagrams from Upsolver customers who are using AWS data lakes. While the Upsolver platform itself is a key component in all of these architectures, we hope even non-customers can learn from these examples when it comes to the use cases, considerations, and design patterns that go into a typical cloud data lake.

In this article, we’ll present seven of the key best practices you should adhere to when designing, implementing and operationalizing your data lake. These are things that will mostly be automatically implemented if you’re using Upsolver, but if you’re building your data lake ‘manually’ you’ll need to ensure they’re done correctly if you want to get the most value out of your data lake architecture.
While we will mostly be referring to Amazon S3 as the storage layer, these best practices would also apply for a data lake built on a different cloud or using Hadoop on-premise.

Within a matter of hours of each day beginning, we ingest tens of billions of records from online sources to HDFS, aggregated across 50+ offline data flows, and visualize 40+ metrics across our business lines to support a dashboard providing company leaders with timely and critical business insights. Dubbed the Daily Executive Dashboard (DED), this dashboard contains critical growth, engagement, and bookings metrics that indicate the health of LinkedIn’s business. In this blog post, we’ll spotlight the challenges to DED’s delivery and performance posed by the growing complexity of computing and aggregating these metrics, and how this dashboard transformed our approach to data pipelines across the company.