Day 3 - Protecting Sensitive Data: Macie and Masking

📌 Key Takeaways

• Macie finds sensitive data in S3 and tells you what is where. It does not mask or delete anything itself. Follow-up actions belong to a separate pipeline.
• There are two kinds of detection criteria: managed data identifiers that AWS provides, and custom data identifiers you build with regular expressions and keywords.
• Masking hides a value when it is displayed (usually irreversible), while tokenization replaces a value with a token and stores the mapping separately so you can reverse it when needed (reversible).
• Redshift dynamic data masking (DDM) leaves the stored values untouched and shows them differently per role at query time. A Lake Formation data filter does not return the value at all.
• Automate the response with detection → EventBridge → Lambda. Handle GDPR deletion with row-level DELETE in Iceberg or Hudi, but you also have to clean up snapshots before the data is physically gone.

Macie Does Two Things

Unintended PII (personally identifiable information) creeps into data lakes: a national ID number typed into a free-text field, a card number printed in a log. Amazon Macie is the service that finds it. The findings Macie produces fall into two categories.

• Policy findings are not generated for settings that were already open before you turned Macie on. "Find the buckets that are already public" is a job for viewing the bucket inventory yourself or for another tool (such as Config).
• Findings are retained for 90 days. To keep them longer, export them through EventBridge or Security Hub, or configure Macie to save discovery results to S3.
• Macie analyzes S3. It does not look directly at PII inside Redshift tables or RDS.

Two Discovery Methods: Automated Discovery and Discovery Jobs

• Automated discovery picks representative objects from each bucket. The objects it picks are actually retrieved and their contents inspected. It means it does not sweep everything, not that it does not look at contents.
• You use the two in sequence. Narrow down the risky buckets with automated discovery, then run a discovery job on those buckets for exhaustive verification.

Identifiers: What Counts as Sensitive

• Managed data identifiers: Detection criteria AWS has built in, such as credit card numbers, national ID and passport numbers for various countries, and credentials like AWS secret keys.
• Custom data identifiers: Organization-specific patterns (employee IDs, customer account numbers) defined with a regular expression and supporting conditions.
• Allow list: Values you want to exclude from detection. Putting in things like a publicly listed main phone number reduces false positives.

{
  "name": "internal-employee-id",
  "regex": "EMP-[0-9]{6}",
  "keywords": ["employee", "employee ID"],
  "ignoreWords": ["EMP-000000"],
  "maximumMatchDistance": 50
}

• Keywords are a device for reducing false positives. Six-digit numbers are everywhere, but a six-digit number within 50 characters after the word "employee ID" is very likely an employee ID. For columnar data like CSV, a keyword in the column name also counts.
• A custom data identifier cannot be modified after it is created. This is so that the criteria behind past findings do not change. To fix one, create a new one. That is why you test with sample data before creating it.

How Far Can Macie See into Encrypted Objects

• "I ran Macie and got no findings" does not mean "there is no sensitive data." If you did not include Macie in the customer managed key's key policy, it simply could not open the contents. If the key belongs to a different account, you must create a grant in addition to allowing it in the key policy.

💡 Related theory: Macie is a detection tool. Its results go as far as "what is where"; it does not mask, quarantine, or delete. On the exam, answer choices like "mask with Macie" or "delete with Macie" are wrong. Actions after detection are handled by other services chained on through EventBridge.

Connecting Detection to Response Automatically

[Macie finding] → [EventBridge rule] → [Lambda]
   → restrict access with bucket policy / tag object / SNS notification / record in Security Hub

{
  "source": ["aws.macie"],
  "detail-type": ["Macie Finding"],
  "detail": { "type": [{ "prefix": "SensitiveData:S3Object/" }] }
}

• Automated response should not delete objects. The evidence disappears, and you cannot undo it if it was a false positive. Narrow the access and add a tag so that a person makes the call.
• If you aggregate findings in Security Hub, you can prioritize them together with findings from other security services on one screen.

Comparing Techniques for Reducing Exposure

• If a question states "keep joins" together with "original can be restored," it is tokenization. If it says "no need to restore," it is masking or hashing.
• If you hash a column with few distinct values (gender, postal code), it can be reverse-engineered by hashing every possible value and matching. Hashing is not a universal anonymization method.

💡 Related theory: Tokenization can hide the original while preserving joins and analytics, so payment regulations such as PCI-DSS prefer it. The cost is that the vault holding the mapping between originals and tokens becomes a new asset to protect. The heart of the design is minimizing the systems that can access the vault.

Where to Mask: Implementation Location

• The Glue Detect PII transform scans in two ways: inspecting every cell, or sampling rows to find only the "columns that contain PII." The latter costs less. For the values it finds, you choose an action such as replacing with a fixed string, partial replacement, or applying a SHA-256 hash.
• Its role differs from Macie's. Macie investigates objects already accumulated in S3, while the Glue transform processes data flowing through the pipeline on the spot.

-- Redshift dynamic data masking
CREATE MASKING POLICY mask_card_full
WITH (card_number VARCHAR(20))
USING ('****-****-****-****'::TEXT);
 
CREATE MASKING POLICY mask_card_last4
WITH (card_number VARCHAR(20))
USING ('****-****-****-' || RIGHT(card_number, 4));
 
-- All users get full masking, the analyst role gets the last 4 digits (the higher-priority policy applies)
ATTACH MASKING POLICY mask_card_full
ON payments(card_number) TO PUBLIC;
 
ATTACH MASKING POLICY mask_card_last4
ON payments(card_number) TO ROLE analyst_role PRIORITY 10;

• DDM does not transform and store the data; it changes only the presentation at query time. Even when privacy requirements change, you fix the policy without touching the original data or the SQL.
• Only a superuser or a principal with the sys:secadmin role can create and attach policies.
• When multiple policies apply to a single column, the policy with the higher priority (PRIORITY) is applied. If you omit the priority, it is 0. To make a specific role's policy apply over a policy attached to PUBLIC, give it a higher priority.
• A masking expression can use not only SQL but also user-defined functions. If you mask with a hash, you can join without ever seeing the original.
• If masking means "show a masked value," a data filter means "do not return the value." Layering the two gives you defense in depth.

GDPR Deletion Is Not Finished with a Single DELETE

• To remove one person's row from a regular Parquet table, you have to rewrite the entire file that contains that row. Open table formats such as Iceberg and Hudi support row-level DELETE, turning this job into a table operation.
• However, an Iceberg DELETE adds a marker (a delete file) saying "this row is deleted." The original data files and the snapshots that point to times before the deletion remain, and can be read again with time travel queries.

-- 1) Logical delete
DELETE FROM lake_clean.payments WHERE customer_id = 'GDPR-REQ-2026-042';
 
-- 2) Merge the delete markers into the data files and rewrite
OPTIMIZE lake_clean.payments REWRITE DATA USING BIN_PACK;
 
-- 3) Expire old snapshots + remove files that belong to nothing
VACUUM lake_clean.payments;

• "Turning on versioning" works against a deletion requirement. It keeps more copies.

Regulations and Tool Mapping

{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "StringNotEquals": { "aws:RequestedRegion": ["ap-northeast-2"] }
  }
}

• An SCP like the one above blocks operations outside the allowed Regions across the whole account. In practice you must carve out exceptions for global services that have no Region, such as IAM. Along with the Region restriction, you also need to check cross-Region replication and snapshot copy settings so that data does not leave.

Symptom → Cause → Action

Question Clue → Answer Cheat Table

Tomorrow we cover governance: finding data (catalog), trusting it (lineage), sharing it, and tracking it (auditing).

📖 Glossary

• PII: Information that can identify an individual. Names, national ID numbers, card numbers, email addresses, and so on.
• Managed data identifier: The sensitive data detection criteria that Macie provides by default.
• Custom data identifier: Detection criteria you define yourself with a regular expression and keywords. It cannot be modified after creation.
• Finding: A report of one issue that Macie found. There are two categories: bucket configuration issues and sensitive data.
• Masking: Hiding a value or showing only part of it. It usually cannot be reversed to the original.
• Tokenization: Replacing a sensitive value with a meaningless token and storing the mapping to the original separately.
• Vault: A protected store that holds the mapping between tokens and originals.
• Dynamic data masking: A Redshift feature that leaves stored values alone and masks them according to the role at the moment of querying.
• Snapshot (Iceberg): The state of a table at a point in time. Even deleted data can be read again if an old snapshot remains.
• Data residency: A requirement that data be kept only within a specified country or Region.