Day 4 - Data Governance: Catalog, Lineage, Sharing, and Auditing

📌 Key Takeaways

• Governance is a cycle of "find (catalog) → trust (lineage) → collaborate (sharing) → track (auditing)." If you have only one axis, the others leak.
• The Glue Data Catalog is the metadata store shared by Athena, Redshift Spectrum, EMR, and Glue. Crawlers infer schemas and register them, and for streams the Schema Registry manages versions and compatibility.
• Lineage records the flow from source → ETL → target → BI, enabling impact analysis and root cause analysis.
• Sharing is distinguished by method. Data Exchange = hands over data (or access rights), Clean Rooms = only agreed-upon analysis results without exposing the raw data, Redshift datashare = live sharing without copying, Lake Formation = cross-account sharing of catalog resources.
• CloudTrail management events are recorded by default, but data events such as S3 object access must be turned on explicitly. Redshift audit logging is also off by default.

Data Catalog: The Starting Point for Finding Data

Data governance deals with "how do we find, trust, share, and track data?" The first step is the catalog. A catalog manages the metadata of data assets (schema, location, owner, tags) in one place.

[S3 data] --crawler--> [Glue Data Catalog: databases/tables/partitions]
                              ↑ share the same definitions
        [Athena] [Redshift Spectrum] [EMR] [Glue ETL]

• The Glue Data Catalog holds the metadata of databases, tables, and partitions. The data itself stays where it is, in S3 and elsewhere.
• A crawler scans a data store, infers the schema and partitions, and registers them as tables.
• Without a catalog, each engine ends up defining its own schema for the same S3 data, and when only one side gets fixed the results diverge. A shared catalog means "define once, use in every engine."

• If the question says "schema consistency across engines," it's the Glue Data Catalog; if it says "search data assets and request/approve access," it points to the business catalog.

Who Manages the Schema: Crawlers and Schema Registry

• Schema Registry supports Avro, JSON (JSON Schema), and Protobuf formats, and has no usage charge. It provides libraries (SerDe) so producers and consumers serialize and deserialize with the same schema.
• The purpose is to treat the schema as a "contract between producer and consumer." Consumers are no longer broken by compatible changes.

• Under BACKWARD, adding a required field is rejected. That's because a new consumer can't read old messages that lack that field. Adding it as an optional field passes.
• "Deploy consumers first and producers later" is the deployment order that fits BACKWARD; "producers first" fits FORWARD.

Data Lineage: The Basis of Trust

Lineage is a record of where data came from, what transformations it went through, and where it went.

• Amazon DataZone data lineage is compatible with OpenLineage. For Glue tables and Redshift added to the catalog, lineage can be set up to be collected automatically, and Spark jobs on Glue 5.0 or later can be configured to send lineage events at run time. For other systems, you push lineage events in directly through the API.
• Lineage nodes are datasets (tables, views) and jobs (runs), and column-level lineage is also available. Every event is versioned, so you can compare "what the path was last month."
• On the machine learning side there is a separate SageMaker ML Lineage Tracking. It records the lineage of ML workflow steps: dataset → training → model → endpoint. If the question asks "what data was the model trained on," this is the one.
• It connects to yesterday's GDPR deletion. When you have to erase one customer's data, lineage tells you which tables and extract files hold copies.

💡 Related theory: Without lineage, a schema change becomes "change it and wait to see what breaks." The Week 8 incident (a downstream job failed because an upstream column name changed) could have been prevented by checking the blast radius with lineage before the change.

Four Ways to Share Data

AWS Data Exchange

• A service for subscribing to datasets created by data providers, or publishing your own data as a product. Besides data products on AWS Marketplace, there is also a data grant that gives a specific account access without going through the marketplace.
• Delivery differs by dataset type. With the Files type, the subscriber exports to their own S3 and holds a copy. With the Amazon S3 data access type, they read the provider's bucket directly; with the Amazon Redshift type, they query read-only without ETL; with the API type, they call it to receive data. The Lake Formation data permission type is a preview according to the documentation.
• "Data Exchange always copies the data to you" is the old explanation. It applies only to the Files type.

AWS Clean Rooms

• Multiple parties analyze jointly without exposing raw data to each other. The data is neither moved out of AWS nor pooled in one place; it is read from where each party keeps it.
• You create a collaboration and invite members, then each member associates its own configured table and applies an analysis rule to that table. The analysis rule decides which SQL is allowed and the output constraints (for example, aggregations only).

• The collaboration creator also decides who pays the query costs. The member who queries and the member who pays can be different.

Party A data ┐
             ├──> [Clean Rooms collaboration: only agreed aggregate queries] ──> only results returned
Party B data ┘     (neither can see the other's raw data)

Redshift Data Sharing (datashare)

• Shares live data without copying across clusters, serverless workgroups, accounts, and Regions. Changes committed on the producer are immediately visible in a new transaction on the consumer (transactional consistency).
• Shareable units are schemas, tables, views (regular, late-binding, and materialized views), and SQL UDFs. You can grant write permission as well as read.
• The typical use is an ETL-dedicated cluster distributing data to several BI clusters. Workloads are isolated and costs are tracked separately per workload.

-- Producer
CREATE DATASHARE sales_share;
ALTER DATASHARE sales_share ADD SCHEMA sales;
ALTER DATASHARE sales_share ADD TABLE sales.orders;
GRANT USAGE ON DATASHARE sales_share TO ACCOUNT '444455556666';
 
-- Consumer (another account): attach the share as a database, then query
CREATE DATABASE sales_from_prod FROM DATASHARE sales_share
  OF ACCOUNT '111122223333' NAMESPACE 'a1b2c3d4-5678-90ab-cdef-111122223333';
SELECT COUNT(*) FROM sales_from_prod.sales.orders;

• This differs from snapshot sharing. A snapshot restores a copy as of a particular point in time, so it isn't the latest data and it also incurs storage cost.

💡 Related theory: Data Exchange is the model that "hands over data (or access rights)"; Clean Rooms is the model that "doesn't hand over data and gets only the aggregated result." If a question says "neither party may see the other's raw data," bucket sharing, datashare, and Data Exchange are all eliminated, because the other party ends up reading the raw rows.

Auditing: Who, When, What

{
  "eventName": "GetObject",
  "userIdentity": { "arn": "arn:aws:sts::111122223333:assumed-role/RegionAnalyst/alice" },
  "requestParameters": { "bucketName": "lake-curated", "key": "sales/dt=2026-06-25/part-0.parquet" },
  "eventTime": "2026-06-25T09:14:02Z"
}

• You need data events like the one above to prove "which role read which object and when." Management events alone only answer as far as "who changed the bucket policy."
• Data events are high in volume. Don't turn them on for every bucket; narrow them to sensitive buckets and prefixes.
• Redshift audit logging produces three logs. The connection log (authentication attempts, connections, disconnections), the user log (changes to user definitions), and the user activity log (query text before execution). Turning on audit logging alone gives only the first two; to also keep the query text, you must separately turn on the parameter enable_user_activity_logging. This is the cause of "I turned on audit logging but queries aren't recorded."
• Redshift API calls (creating or modifying a cluster) are recorded in CloudTrail, while SQL activity inside the database is recorded in the audit logs. They are different layers.
• CloudTrail Lake stores events so you can query them with SQL, and Security Hub gathers findings from multiple security services in one place. Collect audit logs in a bucket in an account separate from the operations account and keep them immutable.

CloudTrail management events are recorded by default, but S3 object-level access (data events) is off by default because of volume and cost. If you turn it on after an incident, there is no record from the time of the incident. Sensitive buckets must be enabled in advance.

Integrating Governance

[Catalog] find → [Lineage] trust → [Sharing] collaborate → [Audit] track
   Glue         DataZone       Clean Rooms       CloudTrail
   Catalog      Lineage        Data Exchange     audit logs
                               datashare

Symptom → Cause → Action

Clue → Answer Lookup Table

Tomorrow we'll review this week's four axes (access control, encryption, sensitive data protection, governance) by tying them together in one end-to-end scenario.

📖 Glossary

• Data catalog: A metadata store that records where data is and what structure it has.
• Crawler: A Glue feature that scans a data store, infers the schema, and registers it in the catalog as tables.
• Schema Registry: A Glue feature that stores the schemas of streaming data by version and blocks incompatible changes.
• Compatibility mode: A rule that decides in which direction a new schema version must be compatible with previous versions for registration to be allowed.
• Lineage: A record of where data came from, what transformations it went through, and where it went.
• Impact analysis: Using lineage to check the downstream that a change would reach before changing anything.
• datashare: A bundle of objects in Redshift shared with other clusters and accounts without copying.
• Analysis rule: A constraint in Clean Rooms that defines which queries and outputs are allowed on a table.
• Differential privacy: A technique that adds mathematically calibrated noise to results so individuals can't be identified.
• Data event: A record of access to the data itself, such as reading or writing an object. It is only recorded if you turn it on explicitly in CloudTrail.