Eigenvalues And Eigenvectors What They Actually Re

You’ve probably called PCA(n_components=2).fit_transform(X) dozens of times. It works. You get your reduced dimensions, you feed them into a classifier, and life is good. But if someone at a meetup asked you, “Hey, what’s an eigenvalue?” — would you freeze? Would you mumble something about “the amount of variance explained” without really knowing why that’s true?

You’re not alone. A quick look at data science forums shows this is one of the most common points of confusion. We use the tool, but the math underneath feels like a black box.

Here’s the promise of this article: by the time you finish reading, you’ll be able to draw the eigenvectors of a dataset onto its scatter plot. You’ll be able to read an eigenvalue and say, in plain English, how much variance that direction captures. No hand-waving. Real understanding.

Let’s build that intuition together, starting with a simple story.

You’ve used PCA a hundred times. Could you draw what an eigenvector is?

Imagine you’re a teacher with 100 students. Each student took two exams: Math and Reading. You notice something right away — students who do well on Math tend to do well on Reading. The scores are correlated.

Here’s a scatter plot of those scores. Each dot is one student.

import numpy as np
import matplotlib.pyplot as plt

# Set seed for reproducibility
np.random.seed(42)

# Generate correlated exam scores
mean = [75, 70]
cov = [[100, 60], [60, 100]]  # covariance matrix
scores = np.random.multivariate_normal(mean, cov, 100)

# Plot
plt.figure(figsize=(8, 6))
plt.scatter(scores[:, 0], scores[:, 1], alpha=0.6)
plt.xlabel('Math Score')
plt.ylabel('Reading Score')
plt.title('Exam Scores for 100 Students')
plt.grid(True, alpha=0.3)
plt.show()

You can see the cloud of points stretches diagonally from bottom-left to top-right. That diagonal is the direction of most variation. If you had to draw one arrow through this cloud that best captures the overall spread, you’d probably draw a 45-degree line from the bottom-left to the top-right.

That arrow is an eigenvector. And the amount of spread along that arrow is the eigenvalue.

But to really understand why that’s true, we need to take a small detour into what a matrix actually does to space.

Before eigenvectors: what a matrix actually does to space

Think of a matrix as a machine. You feed it a vector (an arrow), and it spits out a new vector. Some vectors get spun around — rotated to a completely new direction. Others just get stretched or squished along their original line.

An eigenvector is a vector that only gets stretched or squished — never rotated off its original line. The eigenvalue is that stretch factor.

Let’s see this with a concrete example. Here’s a simple 2x2 matrix called a shear matrix:

import numpy as np
import matplotlib.pyplot as plt

# Define a shear matrix
A = np.array([[1, 0.5],
              [0, 1]])

# Create a grid of points
x = np.linspace(-2, 2, 10)
y = np.linspace(-2, 2, 10)
X, Y = np.meshgrid(x, y)
points = np.vstack([X.ravel(), Y.ravel()]).T

# Apply the transformation
transformed = points @ A.T

# Plot original vs transformed
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

ax1.scatter(points[:, 0], points[:, 1], s=10, alpha=0.5)
ax1.axhline(0, color='gray', linewidth=0.5)
ax1.axvline(0, color='gray', linewidth=0.5)
ax1.set_title('Original Grid')
ax1.set_xlim(-3, 3)
ax1.set_ylim(-3, 3)
ax1.set_aspect('equal')

ax2.scatter(transformed[:, 0], transformed[:, 1], s=10, alpha=0.5)
ax2.axhline(0, color='gray', linewidth=0.5)
ax2.axvline(0, color='gray', linewidth=0.5)
ax2.set_title('Transformed Grid')
ax2.set_xlim(-3, 3)
ax2.set_ylim(-3, 3)
ax2.set_aspect('equal')

plt.tight_layout()
plt.show()

Now, here’s the key question: are there any vectors that stay on their original line after this transformation? Let’s check one candidate: the vector [1, 0] (pointing right along the x-axis).

import numpy as np

A = np.array([[1, 0.5],
              [0, 1]])

v = np.array([1, 0])
Av = A @ v

print(f"Original vector: {v}")
print(f"Transformed vector: {Av}")
print(f"Is it on the same line? {np.isclose(Av[1], 0)}")

It stays on the x-axis! The y-component is still zero. It just got stretched a bit. So [1, 0] is an eigenvector of this matrix. The eigenvalue is the stretch factor: 1 (since the x-component didn’t change).

What about [0, 1]? That one gets rotated — it becomes [0.5, 1], which is a different direction entirely. Not an eigenvector.

This is the hardest part of linear algebra for most people — but once you see it visually, it clicks. Eigenvectors are the special directions that the matrix doesn’t rotate, only scales.

The problem PCA was invented to solve (and it wasn’t ‘find eigenvectors’)

Here’s a surprising fact: PCA wasn’t invented to find eigenvectors. It was invented to solve a geometry problem.

In 1901, Karl Pearson asked: given a cloud of points, what’s the line that passes through them such that the sum of squared perpendicular distances from each point to the line is minimized?

Let’s look at our exam scores again. Which line best summarizes the cloud?

import numpy as np
import matplotlib.pyplot as plt

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Plot with two candidate lines
plt.figure(figsize=(8, 6))
plt.scatter(scores[:, 0], scores[:, 1], alpha=0.6)

# Horizontal line (ignores Reading variance)
plt.axhline(y=mean[1], color='red', linestyle='--', linewidth=2, label='Horizontal line')

# Diagonal line (45 degrees)
x_line = np.linspace(40, 110, 100)
y_line = mean[1] + (x_line - mean[0])  # slope 1 through the mean
plt.plot(x_line, y_line, 'green', linestyle='--', linewidth=2, label='Diagonal line')

plt.xlabel('Math Score')
plt.ylabel('Reading Score')
plt.title('Which line best summarizes the cloud?')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

Look at the red horizontal line. If you measure the perpendicular distance from each point to that line, you’ll get large distances for points far from the center. Now look at the green diagonal line. Points are clustered much closer to it. The diagonal line clearly minimizes those perpendicular distances better.

The first principal component is that diagonal line.

PCA didn’t start as an eigenvector algorithm; it started as a geometry problem. And eigenvectors turned out to be the elegant mathematical answer.

Here’s the connection: the eigenvectors of the covariance matrix ARE the principal components

Now we need to bridge from geometry to algebra. The key player is the covariance matrix.

For our exam scores, the covariance matrix is a 2x2 matrix. The diagonal entries tell us how much each exam varies on its own. The off-diagonal entry tells us how Math and Reading vary together.

import numpy as np

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Compute the covariance matrix
cov_matrix = np.cov(scores, rowvar=False)
print("Covariance matrix:")
print(cov_matrix)
print()
print(f"Variance of Math scores: {cov_matrix[0,0]:.2f}")
print(f"Variance of Reading scores: {cov_matrix[1,1]:.2f}")
print(f"Covariance between Math and Reading: {cov_matrix[0,1]:.2f}")

The covariance is positive (about 60), confirming what we saw visually: when Math goes up, Reading tends to go up too.

Now here’s the load-bearing claim: the direction that minimizes perpendicular distance (the first principal component) is exactly the eigenvector associated with the largest eigenvalue of this covariance matrix.

Here’s that claim worked through for our covariance matrix — the eigenvalues drop straight out of a 2×2 determinant:

Let’s compute those eigenvectors and eigenvalues.

import numpy as np

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Compute covariance matrix
cov_matrix = np.cov(scores, rowvar=False)

# Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

print("Eigenvalues:")
print(eigenvalues)
print()
print("Eigenvectors (as columns):")
print(eigenvectors)
print()
print(f"First eigenvector (largest eigenvalue): {eigenvectors[:, 0]}")
print(f"Second eigenvector: {eigenvectors[:, 1]}")

The first eigenvector is approximately [0.707, 0.707] — that’s a 45-degree diagonal! Exactly the line we guessed earlier. The second eigenvector is perpendicular to it.

Let’s overlay these eigenvectors on our scatter plot to see the match.

import numpy as np
import matplotlib.pyplot as plt

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Compute covariance and eigenvectors
cov_matrix = np.cov(scores, rowvar=False)
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

# Plot
plt.figure(figsize=(8, 6))
plt.scatter(scores[:, 0], scores[:, 1], alpha=0.6)

# Draw eigenvectors as arrows (scaled by sqrt(eigenvalue) for visibility)
for i in range(2):
    vec = eigenvectors[:, i]
    scale = np.sqrt(eigenvalues[i]) * 2  # scale for visibility
    plt.arrow(mean[0], mean[1], vec[0]*scale, vec[1]*scale,
              head_width=3, head_length=3, fc=f'C{i}', ec=f'C{i}',
              label=f'Eigenvector {i+1} (eigenvalue={eigenvalues[i]:.1f})')

plt.xlabel('Math Score')
plt.ylabel('Reading Score')
plt.title('Eigenvectors of the Covariance Matrix Overlaid on Data')
plt.legend()
plt.grid(True, alpha=0.3)
plt.axis('equal')
plt.show()

The first eigenvector points along the diagonal — exactly the line that minimizes perpendicular distances. The second eigenvector is perpendicular to it, capturing the remaining variation.

The first eigenvalue (about 155) tells us the variance of the data when projected onto that direction is 155 squared points. That’s over 80% of the total variance.

Eigenvalues = variance captured. In plain English, here’s what that tells you.

Let’s make this concrete. We’re going to project each 2D data point onto the first eigenvector direction. The result will be a single number — the coordinate on the new axis. Then we’ll compute the variance of those projected numbers.

import numpy as np

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Center the data (subtract the mean)
centered_scores = scores - mean

# Compute covariance and eigenvectors
cov_matrix = np.cov(scores, rowvar=False)
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

# Project onto first eigenvector
v1 = eigenvectors[:, 0]
projected_onto_v1 = centered_scores @ v1  # dot product for each point

# Compute variance of projected points
variance_along_v1 = np.var(projected_onto_v1)

print(f"Eigenvalue 1: {eigenvalues[0]:.2f}")
print(f"Variance of projected points: {variance_along_v1:.2f}")
print(f"They match? {np.isclose(eigenvalues[0], variance_along_v1)}")

They match exactly. This is what the math guarantees, but seeing it happen in code makes it real.

Now let’s do the same for the second eigenvector.

import numpy as np

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Center the data
centered_scores = scores - mean

# Compute covariance and eigenvectors
cov_matrix = np.cov(scores, rowvar=False)
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

# Project onto second eigenvector
v2 = eigenvectors[:, 1]
projected_onto_v2 = centered_scores @ v2
variance_along_v2 = np.var(projected_onto_v2)

print(f"Eigenvalue 2: {eigenvalues[1]:.2f}")
print(f"Variance of projected points: {variance_along_v2:.2f}")
print(f"They match? {np.isclose(eigenvalues[1], variance_along_v2)}")

The second eigenvalue is much smaller. This means the data barely varies in the direction perpendicular to the diagonal. Makes sense — once you account for the overall Math-Reading correlation, there’s not much left.

Here’s another check: the sum of eigenvalues should equal the total variance of the original data.

import numpy as np

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Compute covariance and eigenvalues
cov_matrix = np.cov(scores, rowvar=False)
eigenvalues, _ = np.linalg.eig(cov_matrix)

# Total variance of original data
total_variance = np.trace(cov_matrix)  # sum of diagonal

print(f"Sum of eigenvalues: {np.sum(eigenvalues):.2f}")
print(f"Total variance (trace of covariance): {total_variance:.2f}")
print(f"They match? {np.isclose(np.sum(eigenvalues), total_variance)}")

And the explained variance ratio — the number sklearn prints — is simply each eigenvalue divided by the total.

import numpy as np

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Compute eigenvalues
cov_matrix = np.cov(scores, rowvar=False)
eigenvalues, _ = np.linalg.eig(cov_matrix)

# Explained variance ratio
explained_ratio = eigenvalues / np.sum(eigenvalues)

print(f"Explained variance ratio for component 1: {explained_ratio[0]:.3f} ({explained_ratio[0]*100:.1f}%)")
print(f"Explained variance ratio for component 2: {explained_ratio[1]:.3f} ({explained_ratio[1]*100:.1f}%)")
print(f"Sum: {np.sum(explained_ratio):.3f}")

The first component captures over 80% of the variance. The second captures the rest.

So when you call PCA.fit_transform(), this is what’s happening under the hood

Let’s prove that sklearn’s PCA is doing exactly what we just did manually.

import numpy as np
from sklearn.decomposition import PCA

# Recreate the data
np.random.seed(42)
mean = [75, 70]
cov = [[100, 60], [60, 100]]
scores = np.random.multivariate_normal(mean, cov, 100)

# Manual calculation
cov_matrix = np.cov(scores, rowvar=False)
eigenvalues, _ = np.linalg.eig(cov_matrix)
manual_ratio = eigenvalues / np.sum(eigenvalues)

# sklearn PCA
pca = PCA(n_components=2)
pca.fit(scores)

print("Manual explained variance ratio:")
print(manual_ratio)
print()
print("sklearn PCA explained variance ratio:")
print(pca.explained_variance_ratio_)
print()
print("They match?", np.allclose(manual_ratio, pca.explained_variance_ratio_))

They match exactly. Under the hood, sklearn uses a slightly different method (Singular Value Decomposition, or SVD) because it’s more numerically stable, but the result is the same. The columns of the V matrix from SVD are the eigenvectors of the covariance matrix.

This isn’t just theory trivia. Understanding this helps you debug when PCA gives unexpected results:

• Why scaling matters: If Math scores ranged from 0-100 and Reading from 0-10, the covariance matrix would be dominated by Math. The eigenvectors would point almost along the Math axis, ignoring Reading. Always standardize your data before PCA.
• Why correlated features produce small components: When features are highly correlated, the data lives mostly along one direction. The remaining eigenvalues are tiny — you can drop those components with almost no information loss.
• Why you can’t interpret components as independent features: Each component is a weighted combination of all original features. The eigenvector entries tell you those weights.

Let’s recap the whole journey:

1. Pearson’s problem: Find the line that minimizes perpendicular distances to a cloud of points.
2. The covariance matrix: Captures how variables vary together.
3. Eigenvectors of the covariance matrix: These are the principal components — the directions that solve Pearson’s problem.
4. Eigenvalues: The variance captured by each component. sklearn calls this explained_variance_.
5. Explained variance ratio: Eigenvalue divided by total variance. sklearn calls this explained_variance_ratio_.

What you’ve learned — and what’s next

Let’s lock in what you now know:

• Eigenvectors are directions in the original feature space that maximize variance of projected data. They’re the special vectors that only get stretched, not rotated, by the covariance matrix.
• Eigenvalues are the amount of variance captured in those directions. Bigger eigenvalue = more important component.
• sklearn’s explained_variance_ratio_ is just eigenvalue / sum(eigenvalues).

This intuition is the foundation for understanding:

• How many components to keep: Look at the scree plot (eigenvalues in descending order). A common rule of thumb is to keep components with eigenvalues > 1 (the Kaiser criterion), but more practically, keep enough to explain 80-90% of variance.
• How to interpret component loadings: The eigenvector entries tell you which original features contribute most to each component. A large positive entry means that feature is strongly associated with that component.
• Why PCA can fail on unscaled data: If one feature has much larger variance than others, it will dominate the first component. Always standardize.

In the next article, we’ll dive deep into reading the eigenvectors — what the entries mean for feature weights (the dreaded ‘loadings’ matrix) and how to use them to explain your components to a stakeholder. You’ll learn to answer questions like “What does PC1 actually represent in terms of the original features?”

Check Your Understanding

Remember: What is an eigenvector in the context of a matrix transformation?

Understand: Explain in your own words why the first eigenvector of the covariance matrix is the direction of maximum variance.

Apply: Given a 2D dataset with covariance matrix [[5, 3], [3, 5]], compute the eigenvalues and eigenvectors manually (or with code). Which direction captures more variance?

Analyze: If you run PCA on a dataset where all features are perfectly uncorrelated (covariance matrix is diagonal), what would the eigenvectors look like? What would the eigenvalues represent?

Evaluate: A colleague says, “I ran PCA and the first component explains 99% of variance, so I can safely drop all other components.” When might this be a bad idea? (Hint: think about the original features and what the component actually represents.)

Create: Generate a synthetic 3D dataset where the first two dimensions are highly correlated and the third is independent noise. Use PCA to find the principal components. Write code to project the data onto the first two components and visualize the result.