Linear Algebra — Vectors, Matrices & Transformations

Vectors & Vector Spaces

A vector is an ordered list of numbers representing magnitude and direction. Vectors in ℝⁿ form a vector space with operations: addition ⃗u + ⃗v and scalar multiplication c⃗u satisfying 8 axioms (closure, associativity, commutativity, identity, inverse, distributivity). Key concepts: linear independence (no vector is a linear combination of others), span (all possible linear combinations), basis (maximal linearly independent spanning set), dimension (number of basis vectors). The standard basis for ℝ³: {ê₁, ê₂, ê₃}. Inner (dot) product: ⃗u·⃗v = ∑uᵢvᵢ = |⃗u||⃗v|cosθ.

Matrices & Operations

A matrix is a rectangular array of numbers. An m×n matrix A has m rows and n columns. Operations: addition (element-wise, same dimensions), scalar multiplication, matrix multiplication (Aₘ×ₙ × Bₙ×ₚ = Cₘ×ₚ, not commutative in general). Transpose : (Aᵀ)ᵢⱼ = Aⱼᵢ. Inverse : A⁻¹A = I (exists iff det(A) ≠ 0). Determinant : for 2×2, det = ad – bc; for n×n, cofactor expansion or row reduction. Determinant = 0 ⇔ singular (non-invertible) matrix. Row echelon form and Gaussian elimination solve linear systems Ax = b. Applications: computer graphics (transformation matrices), network flow, Markov chains.

Linear Transformations

A linear transformation T: V → W satisfies T(⃗u + ⃗v) = T(⃗u) + T(⃗v) and T(c⃗u) = cT(⃗u). Every linear transformation can be represented by a matrix. Common 2D transformations: rotation by θ (cosθ, –sinθ; sinθ, cosθ), scaling (sₓ, 0; 0, sᵧ), reflection, shear. Composition of transformations = matrix multiplication. The kernel (null space) of T is the set of vectors mapped to zero; the image (column space) is the range. Rank-nullity theorem: dim(ker T) + dim(im T) = dim(V). Change of basis: B = P⁻¹AP.

Eigenvalues & Eigenvectors

For a square matrix A, if A⃗v = λ⃗v for non-zero ⃗v, then λ is an eigenvalue and ⃗v is an eigenvector. Found by solving det(A – λI) = 0 (characteristic polynomial). Properties: trace(A) = Σλᵢ, det(A) = Πλᵢ. Diagonalization : if A has n linearly independent eigenvectors, A = PDP⁻¹ where D is diagonal. Spectral theorem : real symmetric matrices have real eigenvalues and orthogonal eigenvectors. Applications: principal component analysis (PCA), Google’s PageRank, quantum mechanics (observables are Hermitian matrices), stability analysis of dynamical systems.

SVD & Applications in AI

Singular Value Decomposition : any m×n matrix A = UΣVᵀ, where U (m×m) and V (n×n) are orthogonal, Σ is diagonal with singular values σ₁ ≥ σ₂ ≥ … ≥ 0. SVD reveals the "geometry" of a linear transformation. Low-rank approximation (truncated SVD) is optimal in Frobenius norm (Eckart–Young theorem). Applications in AI/ML: dimensionality reduction, recommender systems (Netflix Prize), latent semantic analysis (NLP), image compression. In deep learning: weight initialization, loss landscape analysis, knowledge distillation. NumPy: np.linalg.svd(A). Machine learning relies heavily on linear algebra: gradient descent, backpropagation, attention mechanisms in transformers (Q, K, V matrices).

📚 Mathematics

All Mathematics articles

Home

Physics

Chemistry

Biology

Mathematics

Computer Science

🌐 Languages

🇺🇦 Українська

🇬🇧 English (current)

❓ Frequently Asked Questions

Why is linear algebra important in machine learning?

ML operates on high-dimensional data represented as vectors/matrices. Neural networks are chains of linear transformations (Wx + b) with nonlinear activations. Key operations: matrix multiplication (forward pass), gradient computation (backpropagation), SVD (dimensionality reduction), eigendecomposition (PCA), and attention (QKᵀ/√dₖ in transformers).

What are eigenvalues used for?

Eigenvalues/eigenvectors reveal fundamental properties of linear transformations: principal directions of stretching/compression (PCA), stability of systems (differential equations), page importance (Google PageRank), vibrational modes (structural engineering), and energy levels in quantum mechanics.

What is the determinant of a matrix?

The determinant is a scalar value that encodes several properties: det(A) = 0 means A is singular (no inverse, dependent columns). |det(A)| gives the volume scaling factor of the linear transformation. Sign indicates orientation (positive = preserved, negative = reversed). Computed via cofactor expansion, LU decomposition, or row reduction.

What is a vector space?

A vector space V over a field F is a set of vectors with two operations (addition, scalar multiplication) satisfying 8 axioms: closure, commutativity and associativity of addition, additive identity and inverse, scalar multiplication compatibility, distributivity, and identity element of multiplication. Examples: ℝⁿ, polynomial spaces, function spaces.

How does SVD relate to PCA?

PCA finds principal components (directions of maximum variance) of a data matrix X. Mathematically: compute X’s covariance matrix and its eigenvectors. Equivalently, SVD of the centered data matrix X = UΣVᵀ gives principal components as columns of V, and singular values encode explained variance. PCA = truncated SVD of centered data.

Try it live

Everything above runs in your browser — open Spirograph and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.