# Singular Learning Theory

This is the homepage of a seminar on Singular Learning Theory (SLT), a theory applying algebraic geometry to statistical learning theory founded by Sumio Watanabe. The seminar takes place at metauni every Thursday and all are welcome, for the future schedule see the metauni homepage.

*Image from Sumio Watanabe’s homepage*.

## References

The canonical references are Watanabe’s two textbooks:

**The gray book:**S. Watanabe “Algebraic geometry and statistical learning theory” 2009.**The green book:**S. Watanabe “Mathematical theory of Bayesian statistics” 2018.

Some other introductory references:

- Spencer Wong’s MSc thesis, May 2022, From Analytic to Algebraic: The Algebraic Geometry of Two Layer Neural Networks.
- Liam Carroll’s MSc thesis, October 2021, Phase transitions in neural networks.
- Tom Waring’s MSc thesis, October 2021, Geometric Perspectives on Program Synthesis and Semantics.
- S. Wei, D. Murfet, M. Gong, H. Li , J. Gell-Redman, T. Quella “Deep learning is singular, and that’s good” 2022.
- E. Lau’s blog Probably Singular.

## Schedule

Each week there is a main session and a supplementary session.

**13-1-22**(*Dan Murfet*): What is learning? Singularities and pendulums (video).**13-1-22**(*Edmund Lau*): The Fisher information matrix (video).**20-1-22**(*Edmund Lau*): Fisher information, KL-divergence and singular models (video).**20-1-22**(*Liam Carroll*): Markov Chain Monte Carlo (video).**27-1-22**(*Liam Carroll*): Neural networks and the Bayesian posterior (video)**27-1-22**(*Spencer Wong*): Rings, ideals and the Hilbert basis theorem (video).**3-2-22**(*Spencer Wong*): From analytic to algebraic I (video).**3-2-22**(*Ken Chan*): Resolution of singularities (video).**10-2-22**(*Dan Murfet*): Introduction to density of states (video, notes).**10-2-22**(*Spencer Wong*): Polynomial division (video).**17-2-22**(*Spencer Wong*): From analytic to algebraic II (video).**17-2-22**: Working session 1 (video).**24-2-22**(*Edmund Lau*): Free energy asymptotics (video)**24-2-22**: Working session 2 (video)**3-3-22**(*Spencer Wong*): From analytic to algebraic III (video).**3-3-22**: Working session 3 (video).**10-3-22**(*Tom Waring*): Regularly parametrised models (video).**17-3-22**(*Edmund Lau*): Bounding the partition function (video).**24-3-22**(*Edmund Lau*): The influence of sampling (video).**7-4-22**(*Edmund Lau*): Main Theorem 1 (video).**14-4-22**(*Edmund Lau*): Main Theorem 2 (video).**8-9-22**(*Matt Farrugia-Roberts*): Complexity of rank estimation (video, pocket).**15-9-22**(*Matt Farrugia-Roberts*): Piecewise-linear paths in equivalent networks (video, pocket).**22-9-22**(*various*) A minimal introduction to the geometry of tanh networks (video, pocket).**29-9-22**(*Dan Murfet*): Information theory I - entropy and KL divergence (video, pocket).**6-10-22**(*Zhongtian Chen*): The Kraft-McMillan theorem (video, pocket).**13-10-22**(*Edmund Lau*): On Watanabe’s paper: Asymptotic learning curve and renormalizable condition in statistical learning theory (video, pocket).**13-10-22**(*Dan Murfet*): Intro to blowing up (cross-posted from the Abstraction seminar, video, pocket).**20-10-22**(*Dan Murfet*): State of scaling laws 2022 (video, pocket).**27-10-22**(*Dan Murfet*): In-context learning (video, pocket).**3-11-22**(*Dan Murfet*): Open problems (video, pocket).**10-11-22**(*Edmund Lau*): Newton diagrams in singular learning theory (video, pocket)**17-11-22**(*Matt Farrugia-Roberts*): Overview of MSc thesis**24-11-22**(*Dan Murfet*): Jet schemes I**1-12-22**(*Matt Farrugia-Roberts*): Overview of MSc thesis Pt 2**8-12-22**(*Susan Wei*): TBD

## Background reading

Some rough handwritten notes:

- Deep Learning Theory 1: Why deep learning theory?
- Deep Learning Theory 2: Thermodynamics of Singular Learning Theory
- Deep Learning Theory 3: Phase transitions
- Singular Learning Theory 4: Local RLCT
- Singular Learning Theory 5: Symmetry and RLCT
- Singular Learning Theory 6: Generalisation and Power Laws
- Singular Learning Theory 8: Calculations for feedforward networks
- Singular Learning Theory 12: Density of states
- Singular Learning Theory 13: Asymptotics of the free energy