imatrix: calculate activation-based statistics for new format (GGUF) imatrices

[EAddario]

Following up from #9400 and #12718, I've started tinkering with activation-based statistics, in addition to what's currently available via --show-statistics.

At the moment, I'm exploring three options going from from easy to implement and OK approximation, to some assembly required but fairly accurate:

1. L2 norm of activation difference: where larger values would suggest the tensor has significantly transformed the input with respect to the previous layer.
2. KL Divergence reduction using a pre-computed logit file: using a similar approach as described by nostalgebraist in logit lens, and based on a pre-computed logit file (e.g. from a previous llama-perplexity --save-all-logits run)
3. Given that llama-imatrix already generates the actual logits to compute PPL, use Thông T. Nguyễn's logit prism approach to calculate the exact contribution of each layer to the final logit scores

Sharing with the readers, and in particular @compilade and @jukofyork, in case anyone's willing to double check assumptions and/or suggest alternative approaches I haven't considered.

[jukofyork]

L2 norm of activation difference: where larger values would suggest the tensor has significantly transformed the input with respect to the previous layer.

If we had access to some numerical linear algebra routines then it would likely be possible to get much more interesting stats from this.

If you think about it:

• The L2 norm of the activation difference is just measuring the Euclidean distance of the tip of the input vector vs the tip of the output vector.
• The mean of these norms probably isn't that interesting (but could be used to test if a quant is systematically biasing or scaling the activations).
• The variance of these norms is likely much more interesting and tells you about the "richness" of the transformation (indirectly - see below).

If instead of using the L2 norms of the differences, we construct the cross-covariance matrix of the paired samples, and then take the SVD of this:

• The "richness" of the transformation (measured indirectly above) is actually to do with the distribution of the singular values, eg: there are many sets of activation differences with the same L2-norm, but those with a flat(er) distribution of singular values (vs a couple of large singular values) are likely to be much more important and interesting.
• If you convert the SVD into a polar decomposition, then the scaling and rotational components will likely lead to other interesting insights, eg:

I suspect that the scaling part of the transformation is quite well handled by the current scaler quants, but the rotational component is likely not.

IIRC, some of the 1-2bit quants use vector quantization, and if so; these will likely handle the rotational components better and/or show quite different properties.

I'm on my phone ATM so can't easily link them, but there have been several papers showing:

1. Outlier activations in LLMs matter much more than simple rate–distortion theory would suggest/measure. This is likely related to the "flatness" of the singular values, where only rarely do some singular vector directions give a high dot-product with an input activation, but when they do; they add a significant/important contribution to the output.
2. LLMs are much more rotational than people first realised, eg: there was [IIRC] a Microsoft paper where they constrained everything to be on the surface of a unit ball, and there are several PEFT methods that purely alter the rotational directions via orthogonal transformations.

[jukofyork]

If it's any use, then there is code here to analyse the symmetrised cross-covaraince matrix I used for the control vectors:

https://github.com/jukofyork/control-vectors/blob/main/direction_analyzer.py

The symmetrised version deliberately gets rid of the rotational components as there can't be made use of if we are just looking for a single direction... You can actually do the same on the anti-symmetrised version (to look at the rotational components only), but Eigen-decompostion is less useful for this as it will return all complex vectors (hence why SVD makes more sense).

I should also add that from my experiments using SVD on the tensors (ie: ignoring the activations!) of LLMs, it often appears that the early/final tensors (which actually appear to be very important and are bumped in bits in the quant routines here!), actually tend to have a less flat distribution of singular values themselves! So when you ignore the distribution of input activations - they generally appear to be doing something inherently "lower dimensional" than the middle tensors!? It would be interesting to investigate this whilst also looking at the activations...

[EAddario]

I'd be lying if I were to claim I understand everything in there 🥴, but I think I got the gist.

Implementing the l2 norm seems straightforward without having to introduce additional 3rd party dependencies, but completely agree that a "light" BLAS lib will be a godsend.

For now, I'll focus on l2 norm, but will add activation variance as well (good shout!)

For a later version, I'd like to try the logit prism approach but that's for another day.

Thanks for the steer @jukofyork! more weekend reading 😁

[jukofyork]

I'd be lying if I were to claim I understand everything in there 🥴, but I think I got the gist.

Implementing the l2 norm seems straightforward without having to introduce additional 3rd party dependencies, but completely agree that a "light" BLAS lib will be a godsend.

For now, I'll focus on l2 norm, but will add activation variance as well (good shout!)

For a later version, I'd like to try the logit prism approach but that's for another day.

Thanks for the steer @jukofyork! more weekend reading 😁

If you want to learn more about Linear Algebra then Gilbert Strang's video lectures are amazing:

https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8

(IIRC, the first lecture only is bad resolution, so don't be put off by that!)

or if you like books:

https://www.amazon.co.uk/Practical-Linear-Algebra-Textbooks-Mathematics/dp/0367507846

(or one of the earlier editions of this same book)

gives a really solid foundation in terms of 2D and 3D.

The biggest problem breaking into it is for some reason American Universities decided to make it much more abstract and proof-based than it needs to be (probably to weed out potential math-majors!).

If you look at some much older pre-1980s books, or books not aimed at Westerners, then it's surprising how approachable it is:

https://mirtitles.org/?s=linear+algebra

[jukofyork]

but completely agree that a "light" BLAS lib will be a godsend.

I have tried to bring this up before:

#8831 (reply in thread)
#8831 (comment)

I think it would be fairly straightforward to port the non-complex routines and then open up all that GSL has to offer:

https://www.gnu.org/software/gsl/doc/html/linalg.html

instead of trying to rewrite numerical routines that have had 1000's and 1000's of thought and testing put into them! :)