- Title: [[Steve Drunton - Singular Value Decomposition]] - Type: #source/video - Author: - Reference: https://www.youtube.com/playlist?list=PLMrJAkhIeNNSVjnsviglFoY2nXildDCcv - Published at: - Reviewed at: [[2021-10-13]] - Links: --- ## 1. Overview https://www.youtube.com/watch?v=gXbThCXjZFM&list=PLMrJAkhIeNNSVjnsviglFoY2nXildDCcv&index=1 SVD is a common first step in dimensionality reduction. Can think of SVD as a data-driven generalization of the [[Fast Fourier Transform]]. - "classical" mathematical modeling approaches used transforms like the FFT to map systems of interest into more useful coordinates spaces. - More complex behaviors can't be easily modeled using the more straight forward classical mathematical transforms (e.g. turbulent flow over a wing) - You need to "tailor" a specific coordinate space for the given data for the problem at hand. ^[data-driven space transformations pervade a lot of modern [[Machine Learning]] e.g. [[Embeddings]] or [[Transformers]] involve doing a data driven search of a highly dimensional target space] It's widely used because it's based on simple linear algebra, interpret-able, and scalable. ### SVD for solving linear systems of equations - Can solve for $Ax = b$ for non-square $A$. This is particularly useful for [[Linear regression]] models. e.g. if $A$ is risk factors and $b$ is some disease, you can find the best fit model $x$ using [[Least squares linear regression]]. - Can use it as the basis for [[Principal component analysis]] to find dominant correlations and patterns in a highly dimensional dataset. ## 2. Mathematical overview https://www.youtube.com/watch?v=nbBvuuNVfco&list=PLMrJAkhIeNNSVjnsviglFoY2nXildDCcv&index=2 How is the SVD computed Let $X$ be a matrix of $m$ column vectors $x_i$. e.g. each column can be an image of a face or a frame in a video. $ \mathbf X = \begin{pmatrix} x_{1} & \dots & x_{1n} \\ \vdots & \ddots \\ x_{m1} & \dots & x_{mn} \end{pmatrix} $