大语言模型入门文献推荐
分享一些入门大模型时候学习过的文献。分成两类：大模型微调论文、基础大模型论文。这些论文不用精通，明白其基本原理即可。目前技术发展太快，更多的时间和精力应该放在前沿论文的学习上。
一、基础大模型论文
首先是目前主流的一些模型，包括GPT（Openai）, GLM（清华）Llama（Meta）。相关的大模型论文还有很多，例如Qwen等。读者能就其中的论文做到举一反三即可。
- GPT1: Improving language understanding by generative pre-training
- GPT2: Language models are unsupervised multitask learners
- GPT-3：Language Models are Few-Shot Learners
- GPT-4 Technical Report
- InstructGPT: Training language models to follow instructions with human feedback
- GLM: General Language Model Pretraining with Autoregressive Blank Infilling
- GLM-130B: An Open Bilingual Pre-trained Model
- LLaMA: Open and Efficient Foundation Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Model
还有一些经典的文本生成论文，大家有时间也可以阅读。
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- T5:Exploring the limits of transfer learning with a unified text-to-text transformer
- ELMO:Deep contextualized word representations
二、微调方法论文
有关大模型微调方法，目前LoRA几乎是垄断的情况，调起来也非常简单。所以其他微调方法了解即可。
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- P-tuning: GPT Understands, Too
- P-Tuning V2: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
- Prompt tuning: The Power of Scale for Parameter-Efficient Prompt Tuning
- Adapter: Parameter-Efficient Transfer Learning for NLP
- LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
此外也有一些写的很好的微调方法综述文章。
- Towards a unified view of parameter-efficient transfer learning
- Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
- Scaling down to scale up: A guide to parameter-efficient fine-tuning
- UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
刚学习的时候，经常会听到指令微调（Instruction tuning），有监督微调（supervised finetuning等），其实都是一回事，这边也放一些相关文章。
- Instruction Tuning for Large Language Models: A Survey
- Instruct learning: Finetuned Language Models are Zero-shot Learners
三、其他资料
1、P-tuning：自动构建模版，释放语言模型潜能
2、PET，必须要GPT3吗？不，BERT的MLM模型也能小样本学习