NOTE: Work in Progress

How much do the models we train depend on the path they follow through weight space?

Should we expect to always get the same models for a given choice of hyperparameters and dataset? Or do the outcomes depend highly on quirks of the training process, such as the weight initialization and batch schedule?

If models are highly path-dependent, it could make alignment harder: we'd have to keep a closer eye on our models during training for chance forays into deception. Or it could make alignment easier: if alignment is unlikely by default, then increasing the variance in outcomes increases our odds of success.

Vivek Hebbar and Evan Hubinger have already explored the implications of path-dependence for alignment here. In this post, I'd like to formalize "path-dependence" in the language of dynamical systems, and share the results of experiments inspired by this framing.

From these experiments, we find novel scaling laws relating choices of hyperparameters like model size, learning rate, and momentum to the trajectories of these models during training. We find functional forms that provide a good empirical fit to these trajectories across a wide range of hyperparameter choices, datasets, and architectures.

Finally, we study a set of toy models that match the observed scaling laws, which gives us insight into the possible mechanistic origins of these trends, and a direction towards building a theory of the dynamics of training.

Formalizing Path-dependence

Hebbar and Hubinger define path-dependence "as the sensitivity of a model's behavior to the details of the training process and training dynamics." Let's make that definition more precise. To start, let us formalize the learning process as a dynamical system over weight space.

In the context of machine learning, our goal is (usually) to model some "true" function $f^* : \mathcal X \to \mathcal Y$¹. To do this, we construct a neural network, $f: \mathcal X \times \mathcal W \to \mathcal Y$, which induces a model of $f^*$ upon fixing a choice of weights, $w \in \mathcal W \subset \mathbb R^D$. For convenience, we'll denote the resulting model with a subscript, i.e., $\mathcal F \ni f_{w}(x):= f(x, w)$.²

To find the weights, $w_\mathrm{opt}$, such that $f_\text{opt} := f_{w_\text{opt}}$ is as "close" as possible to $f^*$, we specify a loss function $L_n$, which maps a choice of parameters and a set of training examples, $\mathbf d_\text{train} = \{(x_{i},y_i)\}_{i=1}^n \in \mathcal Z^n$, to a scalar "loss"³:

For a fixed choice of dataset, we denote the empirical loss, $L_\text{train}(w) =L(w, \mathbf d_\text{train})$. Analogously, we can define a test loss, $L_\text{test}$ over a corresponding test set, and batch loss, $L^{(t)}_\mathbf{b}(w)$, for a batch $\mathbf b^{(t)} \subset \mathbf d_\text{train}$.

Next, we choose an optimizer, $\Phi$, which iteratively updates the weights (according to some variant of SGD),

The optimizer depends on some hyperparameters, $h \in \mathcal H$, and learning schedule $\mathbf b = \{\mathbf b^{(t)}\}_{t=1}^T$ of $T$ batches, $\mathbf b^{(t)} \in \boldsymbol{\mathcal B}$. Note: this learning schedule covers all $N_\text{epochs}$ epochs. In other words, if an epoch consists of $T_\text{epochs}$ batches, then $T/ T_\text{epochs} = N_\text{epochs}$. Within any given epoch, the batches are disjoint, but across epochs, repetition is allowed (though typically no batch will be repeated sample for sample).

If we take the learning schedule and hyperparameters as constant, we obtain a discrete-time dynamical system over parameter space, which we denote $\Phi_{\mathbf b, h}: \mathcal W \to \mathcal W$.

Two Kinds of Path Dependence

Given this dynamical system, $\Phi_{\mathbf b, h}$ over weight space, let us contrast two kinds of path dependence:

1. Global path dependence. Given some distribution over starting weights, $p(w^{(0)})$, hyperparameters, $p(h)$, or batches, $p(\mathbf b)$, what is the resulting distribution over final weights, $p(w^{(T)})$?
2. Local path dependence. For some choice of initial weights, $w^{0}$, hyperparameters, $h$, or batches, $\mathbf b$, and a small perturbation (of size $\epsilon \ll 1$) to one of these values, how "different" does the perturbed model end up being from the baseline unperturbed model? In the limit of infinitesimal perturbations, this reduces to computing derivatives of the kind $\frac{\partial w^{(T)}}{\partial{w^{(0)}}}$, $\frac{\partial w^{(T)}}{\partial{h}}$, and $\frac{\partial w^{(T)}}{\partial{b^{(t)}}}$.

Though we ultimately want to form global statements of path dependence, the distributions involved are generally intractable (which require us to resort to empirical approximations). In the local view, we rarely care about specific perturbations, so we end up studying the relation between a similarly intractable distribution over perturbations $p(\delta)$ and $p(w^{(T)})$. Still, this is much easier to explore empirically because of its smaller support.

Continuous vs. Discrete Hyperparameters

Below, is a table of the various hyperparameters we encounter in our model, weight initializer, optimizer, and dataloader. Here, we're taking "hyperparameter" in the broadest sense to incorporate even the dataset and choice of model architecture.

To study local perturbations, we're interested in those hyperparameters that we can vary continuously. E.g.: we can gradually increase the magnitude $\epsilon$ of the perturbation applied to our baseline model, as well as the momenta, regularization coefficients, and learning rates of the optimizer, but we can't smoothly vary the number of layers in our model or the type of optimizer we're using.

From weights to functions

The problem with studying dynamics over $\mathcal W$ is that we don't actually care about $w \in \mathcal W$. Instead, we care about the evolution in function space, $\mathcal F$, which results from the mapping $m: \mathcal W \ni w \mapsto f_{w}\in \mathcal F$.

We care about this evolution because the mapping to function space is non-injective; the internal symmetries of $m$ ensure that different choices of $w\in \mathcal W$ can map to the same function $f \in \mathcal F$.[1] As a result, distance in $\mathcal W$ can be misleading as to the similarity of the resulting function.

The difficulty with function space is that it is infinite-dimensional, so studying it is that much more intractable than studying weight space. In practice, then, we have to estimate where we are in function space over a finite number of samples of input-output pairs, for which we typically use the training and test sets.

However, even though we have full knowledge of $\mathcal W$, we can't resolve exactly where we are in $\mathcal F$ from any finite set of samples. At best, we can resolve the level set in $\mathcal F$ with a certain empirical performance (like training/test loss). This means our sampling procedure acts as a second kind of non-injective map from $\mathcal F\to \mathcal F_\text{obs} = (\mathcal X \times \mathcal Y)^n$.

A note on optimizer state: If we want to be exhaustive, let us recall that optimizers might have some internal state, $s \in \mathcal S$. E.g., for Adam, $s_{t}= (m_t, v_t)$, a running average of the first moment and second moment of the gradients, respectively. Then, the dynamical system over $\mathcal W$ is more appropriately a dynamical system over the joint space $\mathcal W \times \mathcal S$. Completing this extension will have to wait for another day; we'll ignore it for now for the sake of simplicity and because, in any case, $s_t$, tends to be a deterministic function of $\mathbf b$ and $h$.

Experiments

In section XX, we defined path dependence as about understanding the relation between $(P(B_t), P(W_0), P(H))$ and $(P(W_{T}), P(F_{T}))$. As mentioned, there are two major challenges:

1. In general, we have to estimate these distributions empirically, and training neural networks is already computationally expensive, so we're restricted to studying smaller networks and datasets.
2. The mapping $m:\mathcal W\to \mathcal F$ is non-straightforward because of the symmetries of $m$.

To make it easier for ourselves, let us restrict our attention to the narrower case of studying local path-dependence.

a (small) perturbation, $\epsilon$, to one of $(b_{t}, w_{0}, h)$. E.g., we'll study probability densities of the kind $p(\mathbf w_0)=\mathcal N(\mathbf w_{0}|\mathbf w_\text{baseline}, \epsilon \mathbf 1)$. For discrete variables (like the number of layers, network width, and batch schedule), there's a minimum size we can make $\epsilon$,

Within dynamical systems theory, there are two main ways to view dynamical systems:

1. The trajectory view studies the evolution of points like $(w_0, w_1, \dots, w_T)$ and $(f_1, f_2, \dots, f_T)$.
2. The probability view studies the evolution of densities like $(p_0(w), p_1(w), \dots, p_T(w))$ and $(p_1(f), p_2(f), \dots, p_T(f))$.

Both have something to tell us about path dependence:

1. In the trajectory view, sensitivity of outcomes becomes a question of calculating Lyapunov exponents, the rate at which nearby trajectories diverge/converge.
2. In the probability view, sensitivity of outcomes becomes a question of measuring autocorrelations, $\langle w_{i} w_{i+\tau}\rangle$ and $\langle f_{i} f_{i+\tau} \rangle$.

Tracking Trajectories

The experimental set-up involves taking some baseline model, $m_0$, then applying a Gaussian perturbation to the initial weights with norm $\epsilon$, to obtain a set of perturbed models $\{m_i\} _{i=1}^{n _\text{models}}$. We train these models on MNIST (using identical batch schedules for each model).

Over the course of training, we track how these models diverge via several metrics (see next subsection). For each of these, we study how the rate of divergence varies for different choices of hyperparameters: momentum $\beta$, weight decay $\lambda$, learning rate $\eta$, hidden layer width (for one-hidden-layer models), and number of hidden layers. Moving on from vanilla SGD, we compare adaptive variants like Adam and RMSProp.

TODO: Non-FC models. Actually calculate the rate of divergence (for the short exponential period at the start). Perform these experiments for several different batch schedules. Perturbations in batch schedule. Other optimizers besides vanilla SGD.

Measuring distance

• $d_\mathbf{w}^{(p)}(m_i, m_0)$: the $p$-norm between the weight vectors of each perturbed model and the baseline model. We'll ignore the $p$ throughout and restrict to the case $p=2$. This is the most straightforward notion of distance, but it's flawed in that it can be a weak proxy for distance in $\mathcal F$ because of the internal symmetries of the model.
• $\frac{d_{\mathbf w}(m_{i}, m_0)}{|\mathbf w_{0}|}$: the relative $p$-norm between weight vectors of each perturbed model.
• $L_\text{train,test}(m_i)$: the training/test losses
• $\delta L_\text{train, test}(m_{i}, m_0)$: the training/test loss relative (difference) to the baseline model $m_0$
• $f_\text{train, test}(m_i)$, $\delta f_\text{train, test}(m_i, m_0)$: the training/test set classification accuracy and relative classification accuracy.
• $L_\text{train cf., test cf.}$ and $f_\text{train cf., test cf.}$: same as above, but we take the predictions of the baseline model as the ground truth (rather than the actual labels in the test set).

A few more metrics to include:

• $d^\text{perm.}_{\mathbf w}$: the l2 norm after adjusting for permutation differences (as described in Ainsworth et al.)
• $\int_{\mathbf w_0}^{\mathbf w_{i}^\text{perm.}} L_\text{train (cf.), test (cf.)}(\mathbf w) d\mathbf w$. The loss integrated over the linear interpolation between two models' weights after correcting for permutations (as described in Ainsworth et al.)

Tracking Densities

TODO

Results

One-hidden-layer Models

What's remarkable about one-hidden-layer models is how little the model depends on weight initialization: almost all of the variance seems to be explained by the batch schedule. Even for initial perturbations of size $\epsilon=10$¹, the models appear to become almost entirely equivalent in function space across the entire training process. In the figure below, you can see that the differences in $L_\text{train}$ are imperceptible (both for a fixed $\epsilon$ and across averages over different $\epsilon$).

TODO: I haven't checked $\epsilon/|\mathbf w|$. I'm assuming this is >1 for $\epsilon=10$ (which is why I find this surprising), but I might be confused about PyTorch's default weight initialization scheme. So take this all with some caution.

The rate of growth for $d_\mathbf{w}$ has a very short exponential period (shorter than a single epoch), followed by a long linear period (up to 25 epochs, long past when the error has stabilized). In some cases, you'll see a slight bend upwards or downwards. I need more time to test over more perturbations (right now it's 10 perturbed models) to clean this up. Maybe these trends change over longer periods, and with a bit more time I'll test longer training runs and over more perturbations.

Hyperparameters

• Momentum: ($\beta=0.1, 0.5, 0.9$) The $d_\textbf w$ curves look slightly more curved upwards/exponential but maybe that's confirmation bias (I expect momentum to make divergence more exponential back when I expected exponential divergence). Small amounts of momentum appear to increase convergence (relative to other models equivalent up to momentum), while large amounts increase divergence (towards . For very small amounts, the effect appears to be negligible. Prediction: the $d_w$ curves curve down because in flat basins, momentum slows you down.
• Learning rate: Same for $\eta=10^{-3}, 10^{-2}, 10^{-1}$. (Not too surprising) TODO: Compare directly between different learning rates. Can we see $\eta$ back in the slope of divergence? Prediction: This should not make much of a difference. I expect that correcting for the learning rate should give nearly identical separation slopes.
• Weight Decay: TODO for $\lambda=10^{-3}, 10^{-2}, 10^{-1}$. Prediction: $d_w$ will shrink (because $|w|$ will shrink). The normalized $d_w/|w|$ will curve downwards because we break the ReLU-scaling symmetry which means the volume of our basins will shrink
• Width: TODO. Both this and depth require care. With different sizes of $\mathbf w$, we can't naively compare norms. Is it enough to divide by $\dim \mathbf w$? Prediction: I expect the slope of divergence to increase linearly with the width of a one-layer network (from modeling each connection as separating of its own linear accord).
• Depth: TODO. Prediction: I expect the shape of $d_w$ for each individual layer to become more and more exponential with depth (because the change will be the product of layers that diverge linearly independently). I expect this to dominate the overall divergence of the networks.
• Architecture: Prediction: I expect convolutional architectures to decrease the rate of separation (after correcting for the number of parameters).
• Optimizer:
  • Prediction: I expect adaptive techniques to make the rate of divergence exponential.

Datasets

• Computer Vision
  • MNIST
  • Fashion-MNIST
  • Imagenet: Prediction: I expect Imagenet to lead to the same observations as MNIST (after correcting for model parameter count).
• Natural Language
  • IMDb movie review database

Why Linear?

I'm not sure. My hunch going in was that I'd see either exponential divergence (indicating chaos) or square root divergence (i.e., Brownian noise). Linear surprises me, and I don't yet know what to make of it.

Maybe all of this is just a fact about one-hidden-layer networks. If each hidden layer evolves independently linearly, maybe this combines additively into something Brownian. Or maybe it's multiplicative (so exponential).

Deep Models

TODO

Appendix

Weight Initialization

For ReLU-based networks, we use a modified version of Kaiming (normal) initialization, which is based on the intuition of Kaiming initialization as sampling weights from a hyperspherical shell of vanishing thickness.

Kaiming initialization is sampling from a hyperspherical shell

Consider a matrix, $\mathbf w^{(l)}$, representing the weights of a particular layer $l$ with shape $(D_\mathrm{in}^{(l)}, D_\mathrm{out}^{(l+1)})$. $D_\mathrm{in}^{(l)}$ is also called the fan-in of the layer, and $D_\mathrm{in}^{(l+1)}$ the fan-out. For ease of presentation, we'll ignore the bias, though the following reasoning applies equally well to the bias.

We're interested in the vectorized form of this matrix, $\vec w^{(l)} \in \mathbb R^{D^{(l)}}$, where $D^{(l)} =D_\mathrm{in}^{(l)} \times D_\mathrm{out}^{(l+1)}$.

In Kaiming initialization, we sample the components, $w_i^{(l)}$, of this vector, i.i.d. from a normal distribution with mean 0 and variance $\sigma^2$ (where $\sigma^2 = \frac{2}{D_\mathrm{in}^{(l)}}$).

Geometrically, this is equivalent to sampling from a hyperspherical shell, $S^{D-1}$ with radius $\sqrt{D}\sigma$ and (fuzzy) thickness, $\delta$.

This follows from some straightforward algebra (dropping the superscript $l$ for simplicity):

and

So the thickness as a fraction of the radius is

where the last equality follows from the choice of $\sigma$ for Kaiming initialization.

This means that for suitably wide networks ($D_\mathrm{in}^{(l)} \to \infty$), the thickness of this shell goes to $0$.

Taking the thickness to 0

This suggests an alternative initialization strategy: sample directly from the boundary of a hypersphere with radius $\sqrt{D}\sigma$, i.e., modify the shell thickness to be $0$.

This can easily be done by sampling each component from a normal distribution with mean 0 and variance $1$ and then normalizing the resulting vector to have length $\sqrt{D}\sigma$ (this is known as the Muller method).

Perturbing Weight initialization

Naïvely, if we're interested in a perturbation analysis of the choice of weight initialization, we prepare some baseline initialization, $\mathbf w_0$, and then apply i.i.d. Gaussian noise, $\boldsymbol \delta$, to each of its elements, $\delta_i \sim \mathcal N(0, \epsilon^2)$.

The problem with this is that the perturbed weights $\mathbf w = \mathbf w_0 + \boldsymbol\delta$ are no longer sampled from the same distribution as the baseline weights. In terms of the geometric picture from the previous section, we're increasing the thickness of the hyperspherical shell in the vicinity of the baseline weights.

There is nothing wrong with this per se, but it introduces a possible confounder (the thickness).

The modification we made to Kaiming initialization was to sample directly from the boundary of a hypersphere, rather than from a hyperspherical shell. This is a more natural choice when conducting a perturbation analysis, because it makes it easier to ensure that the perturbed weights are sampled from the same distribution as the baseline weights.

Geometrically, the intersection of a hypersphere $S^D$ of radius $w_0=|\mathbf w_0|$ with a hypersphere $S^D$ of radius $\epsilon$ that is centered at some point on the boundary of the first hypersphere, is a lower-dimensional hypersphere $S^{D-1}$ of a modified radius $\epsilon'$. If we sample uniformly from this lower-dimensional hypersphere, then the resulting points will follow the same distribution over the original hypersphere.

This suggests a procedure to sample from the intersection of the weight initialization hypersphere and the perturbation hypersphere.

First, we sample from a hypersphere of dimension $D-1$ and radius $\epsilon'$ (using the same technique we used to sample the baseline weights). From a bit of trigonometry, see figure below, we know that the radius of this hypersphere will be $\epsilon' = w_0\cos \theta$, where $\theta = \cos^{-1}\left(1-\frac{\epsilon^2}{2w_0^2}\right)$.

Next, we rotate the vector so it is orthogonal to the baseline vector $\mathbf w_0$. This is done with a Householder reflection, $H$, that maps the current normal vector $\hat{\mathbf n} = (0, \dots, 0, 1)$ onto $\mathbf w_0$:

where

and $\hat{\mathbf w}_0 = \frac{\mathbf w_0}{|w_0|}$ is the unit vector in the direction of the baseline weights.

Implementation note: For the sake of tractability, we directly apply the reflection via:

Finally, we translate the rotated intersection sphere along the baseline vector, so that its boundary goes through the intersection of the two hyperspheres. From the figure above, we find that the translation has the magnitude $w_0' = w_0 \cos \theta$.

By the uniform sampling of the intersection sphere and the uniform sampling of the baseline vector, we know that the resulting perturbed vector will have the same distribution as the baseline vector, when restricted to the intersection sphere.

Dynamical Systems

Formally, a dynamical system is a tuple $(\mathcal T, \mathcal M, \Phi)$, where $\mathcal T$ is the "time domain" (some monoid), $\mathcal M$ is the phase space over which the evolution takes place (some manifold), and $\Phi$ is the evolution function, a map,

that satisfies, $\forall x \in \mathcal M, \forall t_{1}, t_{2}\in \mathcal T$,

⁴

Informally, we're usually interested in one of two perspectives:

1. Trajectories of individual points in $\mathcal M$, or
2. Evolution of probability densities over $\mathcal M$.

The former is described in terms of differential equations (for continuous systems) or difference equations (for discrete systems), i.e.,

or

The latter is described in terms of a transfer operator:

Both treatments have their advantages: the particle view admits easier empirical analysis (we just simulate a trajectory), while the latter

In terms of the former, path-sensitivity becomes the question of chaos: do nearby

Varying the Depth

For a fair comparison across network depths, we want to keep the total number of parameters $D$ constant as we increase depth.

Given $n_h$ hidden layers with biases that each have width, $w$, an input dimensionality of $n_\mathrm{in}$, and an output dimensionality of $n_\mathrm{out}$, the total number of parameters is

Lyapunov spectrum

The Lyapunov exponent $\lambda$ quantifies the rate of separation of infinitesimally close trajectories:

If the exponent is positive, then nearby trajectories will diverge, and the dynamics are chaotic. If the exponent is negative, then nearby trajectories will converge

For a $D$-dimensional system, there are $D$ Lyapunov exponents. We often focus exclusively on the maximal Lyapunov exponent because it dominates the overall rate of separation between two neighboring trajectories. However, the full spectrum contains valuable additional information like the rate of entropy production, the fractal dimension, the Hausdorff dimension, and the Lyapunov dimension.

The first major limitation is that the full Lyapunov spectrum is intractable for large $D$. The other major limitation is that the Lyapunov spectrum requires a suitable norm. We can use the l2 norm in $\mathcal W$, but as we already saws what we really care about is measuring distance in $\mathcal F$ with some metric $d : \mathcal F \times \mathcal F \to \mathbb R$.

One option would be to repurpose the loss function $\ell$:

Here, we treat $f_2(x_i)$ as the truth, and use the empirical risk as our distance metric. We could define an analogous distance over either the train or test set. As long as $\ell$ is a suitable metric (or easily converted into a metric through, e.g., symmetrization), $d$ will be too.

This suffers a major downside that two functions will be seen as identical as long as they have the same performance on the dataset in question. They can have totally different performance on unseen examples. Devising suitable distance metrics for neural networks requires more work.

Autocorrelations

The autocorrelation of a random process $\{X_t\}$ is the correlation between two values of that process at different times,

For stationary processes (where $\langle X_t\rangle$ is independent of $t$), this becomes a function of one variable, $\tau=t_2-t_1$,

In the case of training, the dynamics aren't stationary: learning rate schedules and the the "descent" of gradient descent ensures that these correlations will depend on our choice of starting time. However, we're not typically interested in correlations between later time steps. We care about autocorrelations relative to the starting point $t_1=0$.

In practice, the probabilistic and point-wise views are intimately connected: the relevant autocorrelation timescales can often be directly related to the maximal Lyapunov component. These two formulas are only a small sample of the available tooling.

$\mathcal F$ is the space of $p$-integrable functions, $f_w \in \mathcal F$ iff

which is equipped with a metric,

So we exchange evolution over a finite-dimensional $\mathcal W$ with an infinite-dimensional

Footnotes

1. For regression, $\mathcal Y = \mathbb R^N$, for classification, $\mathcal Y \subset \mathbb N$, for self-supervised tasks, we're often interested in $\mathcal Y = \mathcal X$, etc. ↩ ↩²

2. To start, we'll ignore the model hyperparameters. You can view this as either absorbing the hyperparameters into the weights, into the optimizer, or into our choice of $f$. Later, it'll be useful to separate out hyperparameters (i.e., $f: \mathcal X \times \mathcal Y \times \mathcal H \to \mathcal Y$). ↩

3. A quick note on notation: Often, you'll see the dataset denoted with a capital $D$. Here, we're using a lowercase because it'll be useful to treat $\mathbf d$ as an instance of a random variable $\mathbf D$. Similarly for $x$, $y$, and $w$ ($X$, $Y$, and $W$). Even though $x$, $y$, and $w$ are (typically) all vectors, we'll reserve boldface for sets. TODO: Maybe just bite the bullet and bold all of them. ↩

4. It's possible to generalize this further so that $\Phi : U \subseteq (\mathcal T \times \mathcal M) \to \mathcal M$, but this won't be necessary for us. ↩

NOTE: Work in Progress

Abstract

---

Introduction

Regular learning theory is lying to you: "overparametrized" models actually aren't overparametrized, and generalization is not just a question of broad basins.

The standard explanation for neural networks is that gradient descent settles in flat basins of the loss function. On the left, in a sharp minimum, the updates bounce the model around. Performance will vary wildly with new examples. On the right, in a flat minimum, the updates settle to zero. Performance is stable under small perturbations.

That's because loss basins actually aren't basins but valleys, and at the base of these valleys lie manifolds of constant, minimum loss. The higher the dimension of these "rivers", the lower the effective dimensionality of your model. Generalization is a balance between expressivity (more effective parameters) and simplicity (fewer effective parameters).

Singular directions lower the effective dimensionality of your model. In this example, a line of degenerate points effectively restricts the two-dimensional loss surface to one dimension.

These manifolds correspond to the internal symmetries of NNs: continuous variations of a given network that perform the same calculation. Many of these symmetries are predetermined by the architecture and so are always present. We call these "generic". The more interesting symmetries are non-generic symmetries, which the model can form or break during training.

In this light, part of the power of NNs is that they can vary their effective dimensionality (thus also expressivity). Generality comes from a kind of "forgetting" in which the model throws out unnecessary dimensions. At the risk of being elegance-sniped, SLT seems like a promising route to develop a better understanding of training dynamics (and phenomenon such as sharp left turns and path-dependence). If we're lucky, SLT may even enable us to construct a grand unified theory of scaling.

A lot still needs to be done (esp. in terms of linking the Bayesian presentation of singular learning theory to conventional machine learning), but, from an initial survey, singular learning theory feels meatier than other explanations of generalization.¹ So let me introduce you to the basics…

Singular Models

Maximum likelihood estimation is KL-divergence minimization.

We're aiming for a shallow introduction of questionable rigor. For full detail, I recommend Carroll's MSc thesis here (whose notation I am adopting).

The setting is Bayesian, so we'll start by translating the setup of "standard" regression problem to more appropriate Bayesian language.

We have some true distribution $q(y|x)$ and some model $p(y|x, w)$ parametrized by weights, $w \in W \subseteq \mathbb R^D$. Our aim is to learn the weights that make $p$ as "close" as possible to $q$.

Given a dataset $\mathcal D = {(x_i, y_i)}*{i=1}^n$, frequentist learning is usually formulated in terms of the empirical likelihood of our data (which assumes that each sample is i.i.d.):

The aim of learning is to find the weights that maximize this likelihood (hence "maximum likelihood estimator"):

That is: we want to find the weights which make our observations as likely as possible.

In practice, because sums are easier than products and because we like our bits to be positive, we end up trying to minimize the negative log likelihood instead of the vanilla likelihood. That is, we're minimizing average bits of information rather than maximizing probabilities:

If we define the empirical entropy, $S_n$, of the true distribution,

then, since $S_n$ is independent of $w$, we find that minimizing $L_n(w)$ is equivalent to minimizing the empirical Kullback-Leibler divergence, $K_n(w)$, between our model and the true distribution:

So maximizing the likelihood is not just some half-assed frequentist heuristic. It's actually an attempt to minimize the most straightforward information-theoretic "distance" between the true distribution and our model.

The advantage of working with the KL-divergence is that it's bounded: $K(w) \geq 0$ with equality iff $q(y|x, w) = p(y|x)$ almost everywhere.

In this frame, our learning task is not simply to minimize the KL-divergence, but to find the true parameters:

Note that it is not necessarily the case that a set of true parameters actually exists. If your model is insufficiently expressive, then the true model need not be realizable: your best fit may have some non-zero KL-divergence.

Still, from the perspective of generalization, it makes more sense to talk about true parameters than simply the KL-divergence-minimizing parameters. It's the true parameters that give us perfect generalization (in the limit of infinite data).

The Bayesian Information Criterion is a lie.

One of the main strengths of the Bayesian frame is that it lets enforce a prior $\varphi(w)$ over the weights, which you can integrate out to derive a parameter-free model:

One of the main weaknesses is that this integral is often almost always intractable. So Bayesians make a concession to the frequentists with a much more tractable Laplace approximation (i.e., you approximate your model as quadratic/gaussian in the vicinity of the maximum likelihood estimator (MLE), $w^{(0)}$):²

where $I(w)$ is the Fisher information matrix:

The Laplace approximation is a probability theorist's Taylor approximation.

From this approximation, a bit more math gives us the Bayesian information criterion (BIC):

The BIC (like the related Akaike information criterion) is a criterion for model selection that penalizes complexity. Given two models, the one with the lower BIC tends to overfit less (/"generalize better").

The problem with regular learning theory is that deriving the BIC invokes the inverse, $I^{-1}(w^{(0)})$, of the information matrix. If $I(w^{(0)})$ is non-invertible, then the BIC and all the generalization results that depend on it are invalid.

As it turns out, information matrices are pretty much never invertible for deep neural networks. So, we have to rethink our theory.

Singularities in the context of Algebraic Geometry

For an analytic function $K : W \to \mathbb R, x \in W$ is a critical point of $K$ if it has zero divergence, $\nabla K(x) = 0$. A singularity is a critical point that is also equal to zero, $K(x) = 0$.

Under these definitions, any true parameter $w^*$ is a singularity of the KL divergence. $K(w^*)=0$ follows from the definition of $w^*$, and $\nabla K(w^*)$ follows from the lower bound, $K(w) \geq 0$.

So another advantage of the KL divergence over the NLL is that it gives us a cleaner lower bound, under which $K(w^*)$ is a singularity for any true parameter $w^*$.

We are interested in degenerate singularities — singularities that occupy a common manifold. For degenerate singularities, there is some continuous change to $w^*$ which leaves $K(w^*)$ unchanged. That is, the surface is not locally parabolic.

Non-degenerate singularities are locally parabolic. Degenerate singularities are not.

In terms of $K$, this means that the Hessian at the singularity has at least one zero eigenvalue (equivalently, it is non-invertible). For the KL-divergence, the Hessian at a true parameter is precisely the Fisher information matrix we just saw.

Generic symmetries of NNs

Neural networks are full of symmetries. that let you change the model's internals without changing the overall computation. This is where our degenerate singularities come from.

The most obvious symmetry is that you can permute weights without changing the overall computation. Given any compatible two linear transformations, $A$ and $B$ (i.e., weight matrices), an element-wise activation function, $\phi$, and any permutation, $P$,

because permutations commute with $\phi$. The non-linearity of $\phi$ means this isn't the case for invertible transformations in general.

An example using the identity function for $\phi$.

At least for this post, we'll ignore this symmetry as it is discrete, and we're interested in continuous symmetries that can give us degenerate singularities.

A more promising continuous symmetry is the following (for models that use ReLUs):

For a ReLU layer, you can continuously scale the incoming pre-activation as long as you inversely scale the outgoing activation. Since this symmetry is present over the entire parameter space, $W$, nowhere in the weight space is safe from degeneracy.

As an exercise for the reader, can you think of any other generic symmetries in deep neural networks?³

Both of these symmetries are generic. They're an after-effect of the architecture choice, and are always active. The more interesting symmetries are non-generic symmetries — those that depend on $w$.

Non-Generic Symmetries

The key observation of singular learning theory for neural networks is that neural networks can vary their effective parameter count.

Real Log Canonical Threshold (RLCT)

Zeta function of $K(w)$

where $\forall w \in W: \phi(w)>0$.

Analytically continue this to the whole complex plane with a Laurent expansion, then the first (large) pole is the RLCT.

Missing good image of what is going on here. Yes, the pole is the location of a singularity ($\zeta(z) \to \infty$) and its multiplicity is the order of the polynomial you need to approximate the local behavior of the corresponding zero of $\zeta^{-1}$. But how do I actually interpret $z$?

Real log canonical threshold ($\lambda$)

• The RLCT is the volume co-dimension (the number of effective parameters near the most singular point $W_0$).
• For regular (non-singular) models, the RLCT is precisely $D/2$
• Why divide by two?

Orientation-reversing symmetries

Using ReLUs, we can imagine our network performing a kind of piece-wise high-dimensional splice approximation of the input function. For higher dimensions, we're looking at constant hypersurfaces.

The intersections of these surfaces are described by a linear equation of the parameters that sum to zero. That is, there's an orientation. If we reverse this orientation, we get the same line.

Degenerate nodes

In addition to these symmetries, when the model has more hidden nodes than truth, excess nodes are either degenerate or have the same activation boundary as another one.

NOTE: I'm still confused here.

INSERT comment on equivariance (link Distill article)

---

Glossary

• If these parameters exist at all (i.e., this set is non-empty, and there is some choice of weights $w_0$ for which $K(w_0) = 0$), we say our model is realizable. We'll assume this is the case from now on.
• When every choice of weights corresponds to a unique model (i.e., the map $\theta \mapsto q(y|x, \theta)$ is injective for all $x, y$), we say our model is identifiable.
• If a model is identifiable and its Fisher information matrix is positive definite, then a model is regular. Otherwise, the model is strictly singular.

Singular learning theory kicks into gear when our models are singular. When the true parameters are

If the Hessian is always strictly positive definite (it has no zero eigenvalues for any $\boldsymbol\theta$), then an identifiable model is called regular. A non-regular model is called strictly singular.

Example: Pendulum

Our object of study are triples of the kind $(p(y|x,w), q(y|x), \phi(w))$, where $p(y|x, w)$ is a model p,¹ of some unknown true model, $q(y|x)$, with a prior over the weights, $\phi(w)$.

The model itself is a regression model on $f$:

We have some probability distribution $p(y|x)$ and a model $q(y|x, \theta)$ parameterized by $\theta$. Our aim is to learn the weights $\theta$ so as to capture the true distribution, and we assume $p(x, y)$ is realizable

Let's start with an example to understand how learning changes when your models are singular.

You have a pendulum with some initial angular displacement $x_0$ and velocity $v_0$. Newton tells us that at time $t$, it'll be in position

The problem is that we live in the real world. Our measurement of $x_t$ is noisy:

What we'd like to do is to learn the "true" parameters $(x_0^*, v_0^*, g^*, t^*)$ from our observations $x_t$. That gives us a problem: For any set of true parameters, the following would output the same value:

That is: our map from parameters to models is non-injective. Multiple parameters determine the same model. We call these models strictly singular.

• Aim: Modeling a pendulum given a noisy estimate of $(x, y)$ at time $t$.

• The parameters of our model are $(\lambda, g, t)$ (initial velocity, gravitational acceleration, time of measurement), and the model is:

• The map from parameters to models is non-injective. That is, the function, $f$, is exactly the same after a suitable mapping (like $g \to 4g$, $\lambda \to 2 \lambda$, $t \to t/2$).
• You can reparameterize this model to get rid of the degeneracy ($\lambda' = \lambda/\sqrt{g}$, $t' = \sqrt g \cdot t$):

• But that may actually make the parameters less useful to reason about, and, in general, may make the "true" model harder to find.

(If you look at the $t - \lambda$ plane, you get straight line level sets, same for $t - \sqrt g$ plane).

Relation to Hessian of $K(w)$ (KL-divergence between a parameter and the true model)

…TODO

Connection to basin broadness

…TODO

Emphasize this early on.

Question

• Does SGD actually reach these solutions? For a given loss, if we are uniformly distributed across all weights with that loss, we should end up in simpler solutions, right? Does this actually happen though?
• Is part of the value of depth that you create more ReLU like symmetries. Can you create equally successful shallow, wide models if you hardcode additional symmetries?

Footnotes

1. E.g.: that explicit regularization enforces simpler solutions (weight decay is a Gaussian prior over weights), that SGD settles in broader basins that are more robust to changes in parameters (=new samples), that NNs have Solomonoff-like inductive biases [1], or that highly correlated weight matrices act as implicit regularizers [2]. ↩ ↩²

2. This is just a second-order Taylor approximation modified for probability distributions. That is, the Fisher information matrix gives you the curvature of the negative log likelihood: it tells you how many bits you gain (=how much less likely your dataset becomes) as you move away from the minimum in parameter space. ↩

3. Hint: normalization layers, the encoding/unencoding layer of transformers / anywhere else without a privileged basis. ↩

It's been three months since I decided to pivot to AI safety. Who knew three months can be such a long time.

FTX

I got an FTX future fund regrant for six months to help make the switch. Then FTX imploded, and it turns out my grant may be clawed back during the bankruptcy proceedings. Unfortunate. On the bright side¹, FTX seems to have been such a mess that it could take years for the process to get to me. So if you have short enough timelines…²

Courses

• The ARENA virtual program is going along smoothly.
• This month, I joined SERI MATS to get my hands dirty in research. I'm officially under Evan Hubinger's Deceptive AI stream (though I'm also participating in John Wentworth's workshops). Yes, I'm reaching the limits of what I can juggle. We're also working our way through the Alignment 201 curriculum.³
• When ARENA finishes up, I'm going to dedicate more of my attention to metauni (especially their track on singular learning theory (SLT)).
• I've also finished most of the miscellaneous Alignment Forum sequences I wanted to go through (as well as AXRP and The Inside View).

The SERI MATS research sprint is about to start, and I'll be in Berkeley from January to at least February to work on this in person. Safe to say, I have way too many ideas for research projects, but I'm planning to focus on Toy Models of Superposition.

Distillation

• I'm working on an introduction to SLT that should be out soon.
• The video I'm working on with Hoog on AI risk is a little delayed because OpenPhil funding was paused (and I'm a little overextended but don't want to admit it), but it is coming along.
• Next to all that, I've started working on an online, interactive AI safety textbook (very much a work in progress, more coming soon)⁴.

Textbooks

• Mathematics for Machine Learnings: Done. Great book. Highly recommend.
• Pattern Recognition and Machine Learning: Completed up through chapter 9. I have 5 chapters to go. Instead of trying to bang these out in the next month to meet my original deadline, I'm going to push back my deadline by two weeks, so I have more time during the research sprint.
• Cracking the Coding Interview: 6 more chapters to go. Like PRML, I'm going to push back my original deadline 2 weeks to mid-January.⁴
• Reinforcement Learning: I've already gone through 6/17 chapters ahead of schedule. I'm aiming to be done by April.
• Artificial Intelligence: A Modern Approach: Here too, I'm 3 chapters in. Ahead of when I originally planned to get this started. This is a big book, so my (self-enforced) deadline is May 1st.

Outreach

• I'm writing this on a plane to the EAGxBerkeley. Judging from EAGxRotterdam, EAGxBerkeley is going to be great (as long as they leave out the food poisoning part).
• I've started laying out some feelers for a longer term project of establishing an AI-safety-oriented company in mainland Europe. There's a lot of early-career interest among very smart students. Give them a few years, and mainland Europe will be ripe for a new organization like 3 Resources/Rationalia/EA/Topics/Anthropic, Redwood, or Conjecture.

Conclusion

The next month is going to be hectic. I'll be in the Bay area for a week and a half, then my parents' place in NY state for a week, then Michigan at my girlfriend's fiancée's dad's, then NY for another week. Oh yeah, did I mention? I got engaged!⁵ On the day I publish this, it's our 5-year anniversary. Robin, I love you, and I can't wait to spend the rest of our lives together. (However long that is, doomer.)

Footnotes

1. For me. I think it's safe to say the inconvenience for me is less than the inconvenience for people who lost their life savings. ↩

2. My timelines aren't actually that short. But I'm not worried about eventually being able to pay this back (even very soon with the SERI MATS stipend). ↩

3. I've come to conclude that I can safely skip Intro to ML Safety for now. Much of the content overlaps with these other programs/textbooks. ↩

4. I'm probably a bit newer to the field than would be ideal for this task, so I'm hoping to migrate to a more editorial role, delegating the bits that I can. I think my main strength here is more a kind of Olahian interactive distillation. That's an ability which seems to be pretty rare among active researchers. ↩ ↩²

5. I proposed with an Oura ring, which definitely says something about the kind of people we are. Now that I think about it, I should have probably asked for a sponsorship and gotten the whole wedding funded by a late-stage capitalism PR departments, but hey hindsight is 20/20. ↩

Two months ago, I decided to quit my company and dedicate myself full-force at AI safety. The problems I had been working on were not inspiring me, and the actual work left me feeling like my brain was shrinking. Something had to change.

So far, this feels like one of the best decisions I've ever made.

I received an FTX future fund regrant for six months to transition to research. My plan for this period rests on three pillars: (1) technical upskilling in ML, (2) theoretical upskilling in AI safety, and (3) networking/community outreach.

Concretely, my plan is to (1) read lots of textbooks and follow online courses, (2) read lots of alignment forum and go through curricula (like Richard Ngo's AGI Safety Fundamentals and Dan Hendrycks's Intro to ML Safety), and (3) travel to events, apply to different fellowships, and complete small research projects.

A month and a half has gone by since I really started, which turns to be quite a lot of time. Enough that it's a good moment for a progress report and forecast.

Technical Upskilling in ML

Textbooks

• Mathematics for Machine Learning by Deisenroth, Faisal, and Ong (2020).
  • This is a wonderful book. Clear, concise writing. Excellent visuals (color-coded with the corresponding formulas!). It hints at what Chris Olah might be able to do with the textbook genre if he got his hands on it.
  • I've completed up to chapter 9 (that's the first half plus one chapter of the second half). I'll finish the book this month.
• Pattern Recognition and Machine Learning by Bishop (2006).
  • This book is… okay. Sometimes. It leaves very much to be desired on the visualizing front, and in retrospect, I probably wouldn't recommend to it others. But it does provide a strong probabilistic supplement to a wider ML curriculum.
  • I've done up to chapter 5 and skipped ahead to do chapter 9. I plan to go through the rest of the book for completeness. Even if many methods are not immediately relevant to the DL paradigm, a broad basis in statistics and probability theory certainly is. I'm most looking forward to the chapters on causal models (8), sampling techniques (11) and hidden Markov models (13). This should be done by mid-December.
• Cracking the Coding Interview by McDowell (2015)
  • The widespread goodharting of leetcode is one of many reasons I'm afraid of AI. We just have to deal with it.
  • I've completed chapters 1-7, with 10(-ish) to go. I'm aiming to be done with this by January.

I couldn't help myself and got some more textbooks. When I finish MML, I'll move on to Sutton and Barto's Reinforcement Learning. In December, I'll start on to Russell and Norvig's Artificial Intelligence: A Modern Approach. Now that I think about it, I should probably throw Goodfellow's Deep Learning in the mix.

Courses

• Practical Deep Learning for Coders by Fast AI
  • I began following this course but was disappointed by it, mostly because its level was too basic, and its methods were too applied. So I stopped following the course.
• ARENA Virtual
  • Two weeks, a friend introduced me to ARENA Virtual, and I jumped on the opportunity. This program follows a curriculum based on Jacob Hilton's Curriculum, and it's much more my cup of tea. It assumes prior experience, goes much deeper, and is significantly higher-paced. It's also super motivating to work with others.
  • This goes until late December.

Once ARENA is done, I might pick and choose from other online courses like OpenAI's Spinning Up, NYU's Deep Learning, etc. But I don't expect this to be necessary anymore, and it may even be counterproductive. ARENA + textbooks is likely to be enough to learn what I need. Any extra time can probably best go towards actual projects.

Theoretical Upskilling in AI Safety

Courses

• AGI Safety Fundamentals by Richard Ngo
  • I'm going through this on my own and reading everything (the basics + supplementary material). I'm currently on week 7 of 8, so I'll finish this month.
• Intro to ML Safety by Dan Hendrycks
  • As soon as I finish AGISF, I'll move on to this course.

Once I'm done with Intro to ML Safety, I'll go on to work through AGI Safety 201. In the meantime, I've also gone through lots of miscellaneous sequences: Value Learning, Embedded Agency, Iterated Amplification, Risks from Learned Optimization, Shard Theory, Intro to Brain-Like-AGI Safety, Basic Foundations for Agent Models, etc. I'm also working my way through AXRP and The Inside View for an informal understanding of various researchers.

Over the last two months, I've actually found myself becoming less doomer and developing longer timelines.¹ In terms of where I see myself ending up: it's still interpretability with an uptick in interest for brain-flavored approaches (Shard Theory, Steven Byrnes). I picked up Evolutionary Psychology by David Buss and might pick up a neuroscience textbook one of these days. My ideal fit is still probably Anthropic.

Network & Outreach

Programs

• SERIMATS. The essay prompts were wonderful practice in honing my intuitions and clarifying my stance. I think my odds are good of getting in, and that this is the highest value thing I can currently do to speed up my transition into AI safety. The main downside is that SERIMATS includes an in-person component that will be in the Bay starting in January. That's sooner than I would move in an ideal world. But then I guess an ideal world has solved alignment. 🤷‍♂️
• REMIX (by Redwood). I'll be applying this week. This seems as good an opportunity as SERIMATS.

I received the advice to apply more often. To already send off applications to Anthropic, Redwood, etc. I think the attitude is right, but my current approach already sufficient. Let's check in when we hear back from these programs.

Research

• I've also put together a research agenda (email me if you want the link). In it, I've begun dissecting how the research I did during my masters on toy models from theoretical neuroscience could inform novel research directions for interpretability and alignment. I'm starting a few smaller experiments to better understand the path-dependence of training.
• I've also started a collaboration with Diego Dorn to review the literature on representation learning and how to measure distance/similarity between different trained models.

I've decided to hold off on publishing what I've written up in my research agenda until I have more results. Some of the experiments are really low-hanging fruit, yet helpful to ground the ideas, so I figure it's better to wait a little and immediately provide the necessary context.

Networking

• I attended an AI Safety retreat organized by EA NL, which was not only lots of fun, but introduced me to lots of awesome people.
• I'll be attending EAGxRotterdam next week, and EAGxBerkeley in December. Even more awesome people coming soon.

Miscellaneous

• As a final note, I'm working with Hoog on a video about AI safety. It's going to be excellent.

Footnotes

1. More on why in a future post. ↩

A lot has changed for me in the past month. My partner and I decided to close the business we had started together, and I've thrown myself full-force at AI safety.

We weren't seeing the traction we needed, I was nearing the edge of burnout (web development is not the thing for me¹), and, at the end of the day, I did not care enough about our users. It's hard to stay motivated to help a few patients today when you think there's a considerable risk that the world might end tomorrow. And I think the world might end soon — not tomorrow, but more likely than not in the next few decades.² At some point, I reached a point where I could no longer look away, and I had to do something.

So I reached out to the 80,000 hours team, who connected me to people studying AI safety in my area, and helped me apply to the FTX Future Fund Regranting Program for a six-month upskilling grant to receive $25,000 for kickstarting my transition to AI.

Now, I'm not a novice (my Bachelors and Masters theses applied techniques from statistical physics to understand neural networks), but I could definitely use the time to refresh & catch up on the latest techniques. A year is a long time in AI.

Next to "upskilling" in ML proper, I need the time to dive deep into AI safety: there's overlap with the conventional ML literature, but there's also a lot of unfamiliar material.

Finally, I need time to brush up my CV and prepare to apply to AI labs and research groups. My current guess is that I'll be best-suited to empirical/interpretability research, which I think is likely to be compute-constrained. Thus, working at a larger lab is crucial. That's not to mention the benefits of working alongside people smarter than you are. Unfortunately (for me), the field is competitive, and a "gap year" in an unrelated field after your masters is likely to be perceived as a weakness. There's a signaling game at hand, and it's play or be played. To sum, spending time on intangibles like "networking" and tangibles like "publications"³ will be a must.

To keep myself focused throughout the next half year, I'll be keeping track of my goals and progress here. To start, let's take a look at my current plan for the next half year.

Learning Plan

Like all good plans, this plan consists of three parts:

1. Mathematics/Theory of ML
2. Implementation/Practice of ML
3. AI Safety

There's also an overarching theme of "community-building" (i.e., attending EAGs and other events in the space) and of "publishing".

Resources

Textbooks

• Mathematics for Machine Learning by Deisenroth, Faisal, and Ong (2020).
  • I was told that this book is predominantly important for its first half, but I'm ready to consume it in full.
• Pattern Recognition and Machine Learning by Bishop (2006)
  • I was advised to focus on chapter 1-5 and 9, but I'm aiming to at least skim the entirety.
• Cracking the Coding Interview by McDowell (2015)
  • One specification I'm going to have to game is the interview. I'm also taking this as an opportunity to master Rust, as I think having a solid understanding of low-level systems programming is going to be an important enabler when working with large models.

ML/DL Courses

• Practical Deep Learning for Coders by Fast AI
  • This comes with an accompanying book
• Spinning Up by Open AI

There are a bunch more, but these are the only ones I'm currently committing to finishing. The rest can serve as supplementary material after.

AI Safety Courses

• AGI Safety Fundamentals by Richard Ngo
• Intro to ML Safety by Dan Hendrycks

Miscellaneous

• Podcasts: AXRP, Inside View, 80,000 hours
• Articles: AI Safety Papers, <Insert important ML papers here>, <Insert Alignment Forum best of here>
• AI Alignment Newsletter, Robert Miles Youtube channel⁴

Publishing

I'm not particularly concerned about publishing to prestigious journals, but getting content out there will definitely help. Most immediately, I'm aiming to convert / upgrade my Masters thesis to an AI Safety/Interpretability audience. I'm intrigued by the possibility that perspectives like the Lyapunov spectrum can help us enforce constraints like "forgetfulness" (which may be a stronger condition than myopia), analyze the path-dependence of training, and detect sensitivity to adversarial attacks / improbable inputs, that random matrix theory might offer novel ways to analyze the dynamics of training, and, more generally, that statistical physics is an un(der)tapped source of interpretability insight.

In some of these cases, I think it's likely that I can come to original results within the next half year. I'm going to avoid overcommitting to any particular direction just yet, as I'm sure my questions will get sharper with my depth in the field.

Next to this, I'm reaching out to several researchers in the field and offering myself up as a research monkey. I trust that insiders will have better ideas than I can form as of yet, but not enough resources to execute (in particular, I'm talking about PhD students), and that if I make myself useful, karma will follow.

Timeline

Over the next three months, my priority is input — to complete the textbooks and courses mentioned above (which means taking notes, making flashcards, doing exercises). Over the subsequent three months, my priority is output — to publish & apply.

Of course, this is simplifying; research is a continuous process: I'll start to produce output before the next three months is up & I'll continue to absorb lots of input when the three months is up. Still, heuristics are useful.

I'll be checking in here on a monthly basis — reviewing my progress over the previous month & updating my goals for the next month. Let's get the show off the road.

Month 1 (October)

Highlights

• Finish part 1 of MML
• Finish chapters 1-5 of PRML
• Finish intro & chapters 1-5 of Cracking the Coding Interview
• Finish lessons 1-5 of Practical Deep Learning for Coders
• Finish weeks 1-5 of AGI Safety Fundamentals
• ██ ██████████ ████████ ███ ████████ ████████ ██████

Footnotes

1. At least not as a full-time occupation. I like creating things, but I also like actually using my brain, and too much of web development is mindless twiddling (even post-Copilot). ↩

2. More on why I think this soon. ↩

3. Whether in formal journals or informal blogs. ↩

4. I'm including less formal / "easier" sources because I need some fallback fodder (for when my brain can no longer handle the harder stuff) that isn't Twitter or Hacker News. ↩