英文字典中文字典Word104.com

中文字典辭典英文字典 a b c d e f g h i j k l m n o p q r s t u v w x y z

安裝中文字典英文字典辭典工具!

安裝中文字典英文字典辭典工具!

[2404. 02948] PiSSA: Principal Singular Values and Singular Vectors . . .
Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance
GitHub - MuLabPKU PiSSA: PiSSA: Principal Singular Values and Singular . . .
The PiSSA-initialized models are shared on Models for easy reuse They retain the same input and output as the original models but are split into residual models and PiSSA adapters for more effective fine-tuning
NeurIPS 2024 Spotlight | PiSSA披萨收敛快 - 知乎
本文提出了一种名为PiSSA的参数高效微调方法，微调模型中最重要的参数。 PiSSA使用与LoRA [1]完全相同的架构，但是不同的初始化方式。相比LoRA，PiSSA在多个模型和任务上，收敛更快，效果更好。此外结合 4bit量化和PiSSA，相比QLoRA还能减小量化误差 10%-50%
PiSSA: Principal Singular Values and Singular Vectors Adaptation of . . .
Comparative experiments of PiSSA and LoRA across 11 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups
PiSSA: Principal Singular Values and Singular Vectors Adaptation of . . .
PiSSA provides a novel direction for research in PEFT by identifying and fine-tuning the principal components within the model, analogous to slicing and re-baking the richest slice of a pizza
《PISSA》原理及应用_pissa的公式-CSDN博客
二、PiSSA PiSSA全称Principal Singular Values and Singular Vectors Adaptation，主奇异值和奇异向量自适应方法。从名称可以知道，PiSSA类似之前的PCA（主成分分析），利用矩阵分解，找到矩阵的主成分。从下图可以看出结构和LoRA是一致的。 PiSSA的示意图如下：
Pan India Solar Sector Association (PISSA)
Pan India Solar Sector Association (PISSA)
改变LoRA的初始化方式，北大新方法PiSSA显著提升微调效果
PiSSA 的发音类似 “披萨” (pizza)--- 如果把整个大模型类比为一个完整的披萨，PiSSA 切掉其中一角，而且是馅料最丰富的一角（主奇异值、奇异向量），重新烘焙（在下游任务上微调）成喜欢的口味。