"Training of neural networks is a computationally intensive task. The significance of understanding and modeling the training dynamics is growing as increasingly larger networks are being trained. We propose a model based on the correlation of the parameters' dynamics, which dramatically reduces the dimensionality. We refer to our algorithm as Correlation Mode Decomposition (CMD). It splits the parameter space into groups of parameters (modes) which behave in a highly correlated manner through the epochs. We achieve a remarkable dimensionality reduction with this approach, where a network of 11M parameters like ResNet-18 can be modeled well using just a few modes. We observe each typical time profile of a mode is spread throughout the network in all layers. Moreover, retraining the network using our dimensionality reduced model induces a regularization which yields better generalization capacity on the test set. This is a joint work with Rotem Turjeman, Tom Berkov and Ido Cohen."