Who should own the data lake?

Assign a platform team that owns the infrastructure, and assign each dataset to a named domain owner who is responsible for its quality and meaning. Clear ownership is the best protection against a data swamp.

Building a Data Lake Module 7: Governance, Security, Data Quality, and Cost

Introduction

A lake that holds customer and company data must be secure, trustworthy, and affordable. This last module covers access control, encryption, data quality checks, lineage and documentation, and the cost habits that prevent surprise bills, with a checklist to review your own lake.

Prerequisites

Modules 1 to 6 completed

Step-by-Step Guide

Step 1: Control access with IAM and Lake Formation

IAM policies decide who can call AWS services and read S3 objects. AWS Lake Formation adds data-level permissions on catalog databases, tables, and columns, so you can grant an analyst access to a table or to certain columns without giving them bucket access. It can also provide row-level filtering and tag-based access control. Follow least privilege: give each role only what it needs.

Principle: people and jobs get the minimum access they need.

Examples
- Analysts: SELECT on curated tables only (Lake Formation grants)
- Glue jobs: read raw bucket, write clean bucket (IAM role per pipeline)
- Nobody: direct write access to the raw zone except ingestion roles

Step 2: Encrypt and protect sensitive data

Encrypt data at rest (S3 server-side encryption, ideally with KMS keys for sensitive zones) and in transit (HTTPS). Identify personal or regulated data and handle it deliberately: mask or tokenize it in the clean zone, restrict access to the raw copy, and follow retention and deletion rules that apply to your organization. Amazon Macie can scan S3 for sensitive data.

Step 3: Add data quality checks

Check data as it moves between layers: required columns are present, keys are unique, values are in valid ranges, and row counts match expectations. AWS Glue Data Quality lets you express rules in a simple language, and tools such as Great Expectations or Deequ do the same in code. Fail or quarantine data that breaks critical rules, and alert the dataset owner.

# AWS Glue Data Quality (DQDL) example ruleset
Rules = [
    RowCount > 0,
    IsComplete "order_id",
    IsUnique "order_id",
    ColumnValues "amount" >= 0
]

Step 4: Document data and track lineage

People trust data they understand. Give every dataset an owner, a description, column meanings, update frequency, and sensitivity level in the catalog. Lineage shows where a dataset came from and what depends on it, which speeds up impact analysis and debugging. Services such as Amazon DataZone and open standards like OpenLineage provide this.

Step 5: Control cost

Storage is cheap but not free, and queries and jobs add up. Use lifecycle rules to move or delete old data, store data in compressed Parquet, partition so queries scan less, compact small files, set budgets and alerts in AWS Budgets, tag resources by project, and review the biggest cost items monthly. Delete experiment data and sandbox resources.

Monthly lake health checklist
[ ] Top 5 cost items reviewed (S3 requests, Athena scans, Glue DPU-hours)
[ ] Lifecycle rules cover every bucket
[ ] No public buckets (S3 Block Public Access on)
[ ] Every dataset has an owner and a description
[ ] Quality rules run on critical tables, and failures reach a person
[ ] Unused tables, crawlers, and jobs removed

FAQs

Who should own the data lake?: Assign a platform team that owns the infrastructure, and assign each dataset to a named domain owner who is responsible for its quality and meaning. Clear ownership is the best protection against a data swamp.

Summary

Secure the lake with least-privilege IAM and Lake Formation, encrypt data, check quality between layers, document ownership and lineage, and review costs monthly to keep the lake trustworthy and affordable.