Table 1:

Effects of per-sample gradient clipping on gradient flow. Here “Yes/No” means guaranteed or not and the loss refers to the training set. “Loss convergence” is conditioned on $\mathbf{H}\left(t\right)\succ 0$.

Clipping type	NTK matrix	Symmetric NTK	Positive in quadratic form	Positive in eigenvalues	Loss convergence	Monotone loss decay	To zero loss
No clipping	$\text{H}\equiv {\sum }_r{\text{H}}_r$	✓	✓	✓	✓	✓	✓
Batch clipping	$c\text{H}\equiv c{\sum }_r{\text{H}}_r$	✓	✓	✓	✓	✓	✓
Large R clipping (Flat & layerwise)	$\text{H}\equiv {\sum }_r{\text{H}}_r$	✓	✓	✓	✓	✓	✓
Small R clipping (Flat)	$\text{HC}$	✗	✗	✓	✗	✗	✓
Small R clipping (Layerwise)	${\sum }_r{\text{H}}_r{\text{C}}_r$	✗	✗	✗	✗	✗	✗