Great !!!

[yuedajiong]

I tidied standalone-qwen3.ipynb into qwen.py.txt

[rasbt]

Thanks! However, I would prefer not to add it at this time to reduce code duplication as it is also available here: https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py

[yuedajiong]

Thanks. (I did not notice that code.)

A question: No MoE in Qwen3?

[casinca]

Thanks. (I did not notice that code.)

A question: No MoE in Qwen3?

Their largest model and a 30B are sparse, the others are dense (the ones from the notebook).

You can find more info in the Qwen technical report, Sebastian included it in the notebook.

[rasbt]

A question: No MoE in Qwen3?

Good question. I focused on the dense models for now. I plan to add the MoE variants later. (I needed the smaller dense models for another project but more on that soon...)

[yuedajiong]

There are an overwhelming number of emerging technologies in the general direction of LLMs, and many seem worth tracking:

1. The techniques (including data) used by various SOTA large models.

2. Alternative approaches beyond Transformers.
   Attention+
   FlashAttention： #LinearAttention+LocalWindow>Manba
   xformers
   RetNet/Retention：
   Mixed:
   RWKV: RNN+Attention
   Mega:Recurrent+Attention
   GatedRecurrentAttention
   Griffin/Hawk：Linear Recurrent Unit + Local Attention
   GateLoop/HGR：
   Recurrent:
   SSD/StateSpaceDuality
   SSM/StateSpaceModels:
   Mamba:SelectiveStateSpaces -> (SSM+Attention)/(Jamba/Zamba)
   S4:StructuredStateSpaceSequence (SeletiveSSM)
   HiPPO 
   H3/HungryHungryHippos
   HyenaHierarchy/Striped Hyena
   xLSTM
   ARMT/RMT/Associative/RecurrentMemoryTransformers
   NoRecurrentNoAttention:
   Yan
   Liquid:
   Liquid/LFM/LiquidFoundationModels (AdaptiveParameters)
   Diffusion:
   Mercury Coder (Diffusion)
   Other:
   SANA/Sparse Activations, Not Attention

   And here: https://github.com/erogol/BlaGPT

3. The use of reinforcement learning for post-training/alignment.
   And here: https://magazine.sebastianraschka.com/p/the-state-of-llm-reasoning-model-training

4. Technologies built on top of LLMs, such as RAG, agents, and others.

Too many ... :-)

[yuedajiong]

@rasbt @casinca

You guys are so kind.
So, share with you a python trick: dict-like config. :-)

cfg = type('',(), dict(para1='A', para2=123))()
cfg.para1 #better than cfg['para1'], without any import.
cfg.para2

[yuedajiong]

SmolLM3-3B is 'FUUUUUUUUULL' open. :-)

[rasbt]

Thanks for sharing! I like the trick:

d = dict(para1='A', para2=123)
cfg = type('',(), d)()

However, how would you print all the keys for reference, like in d.keys()?