Skip to content

LinearSolve::luc: Result for LinearSolve of badly conditioned matrix {{7.61902,0.0118411,0.00677937,0.10901,0.0815707,0.0166805,0.184858,0.0340636,0.0288654,0.0182311,<<246>>},<<9>>,<<246>>} may contain significant numerical errors. #7

@dbl001

Description

@dbl001

Thanks for creating these Mathematica packages.
When I run NMF on a GPT-2 embedding matrix, I get warnings from LinearSolve.
I tried the following RegularizationParameters: 0.01, 0.1, 1, 2.
I also tried different PrecisionGoals: 8, 16, 32, 64.
Any suggestions?

gpt2Model = NetModel["GPT2 Transformer Trained on WebText Data"]
embeddings = 
 NetExtract[gpt2Model, {"embedding", "embeddingtokens", "Weights"}]
{wMat, hMat} = 
  ResourceFunction["NonNegativeMatrixFactorization"][
   Normal[embeddings], 256, "RegularizationParameter" -> 2, 
   MaxSteps -> 300, PrecisionGoal -> 64];

Screenshot 2024-02-29 at 10 17 12 AM

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions