Day 3 - Data Visualization: Distributions, Correlations, QuickSight, and Drawing Insights

📌 Key Takeaways

• The question decides the chart. For distributions use histograms, box plots and violin plots; for relationships use scatter plots, correlation heatmaps and pair plots.
• Anscombe's quartet: the same statistical summary can hide completely different shapes → never trust summary values alone; always plot.
• Pearson captures only linear relationships. Check monotonic nonlinear relationships with Spearman and general dependence with mutual information.
• QuickSight is a serverless BI service. Remember SPICE (in-memory engine), ML Insights (anomaly detection, forecasting, automatic narratives) and Q (natural-language queries).
• The goal of visualization is not the picture but the follow-up action: skew → log transform, strong correlation → handle multicollinearity.

Chart Selection: The Question Decides the Chart

Once cleaning and dimensionality work are done, the next job is understanding the data by eye. Good visualization reveals the distribution shape, outliers, relationships and clusters that summary statistics hide. MLS-C01 Domain 2 asks both "which chart for which question" and "which AWS service for visualization and BI."

Before choosing a chart, first ask "what question am I trying to answer right now?" The wrong chart hides or distorts patterns.

Misunderstandings Created by the Wrong Chart

⚠️ Pitfall: The same data can give the opposite impression depending on how the axes are set. If a bar chart's y-axis starts at a non-zero value, a trivial difference looks dramatic, and overlaying different units on a dual axis makes a nonexistent correlation appear. Whether reading or making a chart, check the axis range first.

Anscombe's quartet shows datasets whose summary statistics (mean, variance, correlation) are identical but whose shapes are entirely different. The four datasets have nearly the same means, variances, correlations and regression lines, but as scatter plots one is linear, one is curved, and in one a single outlier drags the whole fit. The lesson is "don't trust summary statistics alone; always visualize," and this is why visualization is indispensable in EDA.

Distribution Visualization

import seaborn as sns
import matplotlib.pyplot as plt
 
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.histplot(df["income"], kde=True, ax=axes[0])   # shape + density
sns.boxplot(x=df["income"], ax=axes[1])             # quartiles, outliers
sns.violinplot(x=df["region"], y=df["income"], ax=axes[2])  # distribution by category

• Histogram/KDE: see the distribution shape (skew, number of peaks)
• Box plot: quickly catches the median, quartiles and IQR-based outliers
• Violin: box plot + density. Strong for comparing distributions across categories

For a skewed distribution, apply a log transform and plot again to check whether it moves closer to normal.

 Normal (symmetric)  Right-skewed (right tail)  Bimodal            Outliers present
   ▁▃▅█▅▃▁           ▂█▆▄▃▂▁▁▁▁            ▃█▄▁▁▄█▃          ▂▅█▅▂       ·  ·
 mean ≈ median       mean > median          two peaks          extreme values in the tail
 → use as is         → log(1+x) transform   → hidden segments  → check IQR · domain

Distribution Diagnosis → Preprocessing Prescription

⚠️ Pitfall: Changing a histogram's number of bins can make the same data look entirely different. With too few bins, peaks are smeared; with too many, noise looks like peaks. If the shape is ambiguous, vary the bin count or plot a KDE alongside.

Relationship and Correlation Visualization

# Correlation heatmap: linear relationships at a glance
corr = df.select_dtypes("number").corr()
sns.heatmap(corr, annot=True, cmap="coolwarm", center=0)
 
# Pair plot: pairwise relationships + diagonal distributions
sns.pairplot(df, hue="target", vars=["age", "income", "score"])

• A correlation heatmap simultaneously reveals multicollinearity candidates (pairs with large |r|) and relationships with the target
• A pair plot reveals nonlinear relationships, clusters and outliers between variable pairs

Comparison of Correlation and Dependence Metrics

Pearson correlation captures only linear relationships. Even a perfect curved relationship (e.g., y = x²) can give a Pearson r near 0. Spearman rank correlation better catches relationships that are monotonic but nonlinear, and mutual information better catches general dependence. So do not conclude "low correlation = no relationship"; confirm the shape with a scatter plot.

A large |r| in a correlation heatmap does not immediately imply causation. Two variables moving together can be due to a common third cause or a time trend. Visualization is a tool for generating hypotheses, not for proving causation.

Three Distribution Charts: When to Use Which

Amazon QuickSight

QuickSight is AWS's serverless BI service for building and sharing dashboards and visualizations.

QuickSight suits operational dashboards and business reporting. For pre-training EDA exploration, a notebook (matplotlib/seaborn) or SageMaker Data Wrangler is closer to the workflow.

QuickSight's ML Insights automatically detects anomalies in time series with built-in ML, provides forecasts based on Random Cut Forest, and explains changes in natural language. Analysts get ML insights inside the BI dashboard without writing a model themselves. On the exam, a question saying "business users who do not write code want an anomaly detection and forecasting dashboard" points to QuickSight ML Insights.

Comparison of AWS Visualization and Analysis Tools

Situation → Tool Choice

Drawing Analytical Insights

The goal of visualization is not the picture but an actionable insight.

EDA Visualization Order

Deciding in advance what to plot first reduces what you miss.

[Secure the training set — visualization is done on the training set only]
        │
  ① One variable at a time: histogram / box plot
        └→ Find skew · outliers · constant columns
        │
  ② Target distribution: class bars or histogram
        └→ Check for imbalance (connects to Day 4)
        │
  ③ Variable ↔ target: per-category violin / scatter plot
        └→ Find good feature candidates
        │
  ④ Variable ↔ variable: correlation heatmap / pair plot
        └→ Find multicollinearity candidates (connects to Day 2)
        │
  ⑤ If there is a time axis: line chart
        └→ Check trends · seasonality · change points
        │
  ⑥ Turn the findings into preprocessing decisions (transform · remove · add features)

Visualization is also an act of looking into the data. If you decide preprocessing by looking at plots that include the test set, it becomes human-mediated leakage. The principle is to do distribution checks, outlier judgments and correlation checks on the training set only, and never open the test set until the final evaluation.

Question Clue → Answer Mapping

The next post looks at how to actually deal with the class imbalance that visualization reveals.

📖 Glossary

• EDA (Exploratory Data Analysis): The process of scanning the data's shape, relationships and anomalies with the eye and with statistics before building a model.
• KDE (Kernel Density Estimation): A method that turns a histogram's steps into a smooth curve to show the distribution shape.
• Box plot / IQR: A chart that draws the median and quartiles as a box / the range between the first and third quartiles. Used as the criterion for judging outliers.
• Violin plot: A chart that overlays a density curve on a box plot. Good for comparing distributions across categories.
• Hexbin: A scatter-plot variant that expresses density with hexagonal cells instead when there are so many points that they smear together.
• Anscombe's quartet: Four datasets with nearly identical mean, variance and correlation but completely different scatter-plot shapes. A classic example of why visualization is needed.
• Spearman rank correlation: A correlation computed on ranks rather than the values themselves. Catches relationships that are monotonic but curved.
• SPICE: QuickSight's in-memory compute engine. Data is loaded in advance to make aggregation and visualization fast.
• ML Insights: QuickSight's automatic anomaly detection, forecasting and natural-language explanation feature inside dashboards.
• QuickSight Q: A feature that builds visualizations when you ask in natural language, such as "sales by region last month."