Module 8 – Govern, Secure & Observe

Data Governance

Policies, ownership, and accountability

What governance actually means in practice

▼

What it is

Data governance is the set of policies, processes, and accountabilities that ensure data is trustworthy, discoverable, secure, and compliant. It answers: Who owns this data? Who can access it? What does this field mean? How fresh is it?

Key components

Data catalog: a searchable inventory of all datasets, their schemas, owners, and descriptions (tools: DataHub, Atlan, OpenMetadata, Unity Catalog)
Data lineage: visual map of how data flows from source through transformations to consumption (built into dbt, Dagster, or standalone tools like Marquez)
Ownership: every dataset has a named owner responsible for quality and access decisions
Business glossary: agreed-upon definitions for business terms (what exactly is "revenue"?)
Access policies: role-based access control (RBAC), column-level security, row-level filtering

Security & Privacy

Protecting data at every layer

Security layers for data engineering

▼

Defense in depth

Encryption at rest: AES-256 for stored data (S3, ADLS default encryption, warehouse encryption)
Encryption in transit: TLS/SSL for data moving between systems
Access control: IAM roles, service accounts, least-privilege principle, RBAC
Column masking: hide sensitive columns (SSN, credit card) from unauthorized roles
Row-level security: filter rows based on user context (region, department)
Audit logging: track who accessed what data and when

PII, GDPR, and CCPA compliance

▼

What you need to know

PII (Personally Identifiable Information): names, emails, phone numbers, IP addresses, social security numbers. Must be classified and protected.
GDPR (EU): right to be forgotten (deletion), data portability, consent management, 72-hour breach notification
CCPA (California): right to know what data is collected, right to delete, right to opt out of sale
Data residency: some regulations require data to stay within geographic boundaries
Techniques: tokenization, pseudonymization, hashing, dynamic masking

Data Observability

Know when things break before stakeholders do

The five pillars of data observability

▼

What to monitor

Freshness: Is the data up to date? When was the last successful load?
Volume: Did the expected number of rows arrive? Is it within normal range?
Schema: Did columns change? Were columns added or removed?
Distribution: Are value distributions within normal ranges? (null rates, value frequencies)
Lineage: Which upstream change caused this downstream anomaly?

Tools

Monte Carlo: automated anomaly detection across freshness, volume, distribution
Elementary: open-source dbt-native data observability
Great Expectations / Soda: programmatic quality checks integrated into pipelines
Built-in warehouse monitoring: Snowflake, BigQuery, and Databricks all offer query and resource monitoring

Knowledge Check: Module 8

1. What is data lineage?

Lineage traces the path of data from its origin through every transformation to its final destination. It is essential for debugging ("which upstream change broke this dashboard?") and compliance ("where does this PII data end up?").

2. What is the "right to be forgotten" under GDPR?

GDPR Article 17 gives individuals the right to request erasure of their personal data. For data engineers, this requires knowing where PII exists across all systems (lineage + catalog) and the ability to delete it (MERGE/DELETE operations on lake tables).

3. Which observability pillar checks whether the expected number of rows arrived?

Volume monitoring checks whether row counts are within expected ranges. A sudden drop to zero rows or an unexpected spike often indicates upstream issues.

4. What security principle says users should only have the minimum access needed for their role?

The principle of least privilege grants users only the minimum permissions required to perform their job. This limits the blast radius of compromised credentials and reduces accidental data exposure.

5. What is a data catalog?

A data catalog is a centralized, searchable inventory that helps users discover, understand, and trust available datasets. Tools include DataHub, Atlan, OpenMetadata, and Unity Catalog.