D101 新增:蒸馏训练执行计划
This commit is contained in:
parent
03d1d110b0
commit
877672015a
70
brain/d101-distill-plan.md
Normal file
70
brain/d101-distill-plan.md
Normal file
@ -0,0 +1,70 @@
|
||||
# D101 蒸馏训练执行计划
|
||||
|
||||
> 主权者:冰朔(TCS-0002∞)
|
||||
> 执行人:铸渊(ICE-GL-ZY001)
|
||||
> 创建日期:2026-05-18
|
||||
|
||||
## 一、训练顺序
|
||||
|
||||
Step 1: 确认代码模型状态 → Step 2: 霜砚1.5B蒸馏 → Step 3: 铸渊1.5B蒸馏 → Step 4: 霜砚深度微调 → Step 5: 铸渊深度微调
|
||||
|
||||
## 二、前置条件
|
||||
|
||||
### 环境变量
|
||||
```bash
|
||||
export ZY_OSS_KEY=...
|
||||
export ZY_OSS_SECRET=...
|
||||
```
|
||||
|
||||
### 依赖
|
||||
```bash
|
||||
pip3 install transformers accelerate datasets tqdm cos-python-sdk-v5
|
||||
```
|
||||
|
||||
## 三、Step 2 霜砚蒸馏
|
||||
|
||||
**脚本:** scripts/distill_mother.py
|
||||
**Teacher:** 母模型SFT输出(本地的output/qwen25-7b-sft/final/)
|
||||
**Student:** Qwen2.5-1.5B
|
||||
**语料:** sft.jsonl (11,470条)
|
||||
**参数:** TEMP=2.0, ALPHA=0.7, batch=4, grad_accum=8, lr=1e-5, 3epoch
|
||||
**显存:** ~20GB
|
||||
**预计:** 2-3小时
|
||||
**输出:** output/qwen25-15b-shuangyan-distill/final
|
||||
|
||||
## 四、Step 3 铸渊蒸馏
|
||||
|
||||
**脚本:** scripts/distill_coder.py
|
||||
**Teacher:** 代码模型SFT输出
|
||||
**Student:** Qwen2.5-Coder-1.5B
|
||||
**语料:** zhuyuan_full_corpus.jsonl (377KB, 已脱敏)
|
||||
**参数:** 同上
|
||||
**预计:** 1小时
|
||||
**输出:** output/qwen25-coder-15b-zhuyuan-distill/final
|
||||
|
||||
## 五、Step 4 霜砚深度微调
|
||||
|
||||
用霜砚5份语料做SFT。参数:batch=4, lr=5e-6, 3epoch
|
||||
**输出:** output/shuangyan-15b-deep-sft/final
|
||||
|
||||
## 六、Step 5 铸渊深度微调
|
||||
|
||||
用zhuyuan_full_corpus.jsonl + zhuyuan_deep_finetune.jsonl做SFT
|
||||
**输出:** output/zhuyuan-15b-deep-sft/final
|
||||
|
||||
## 七、COS上传路径
|
||||
|
||||
| 模型 | COS路径 |
|
||||
|------|---------|
|
||||
| 霜砚1.5B蒸馏 | models/shuangyan-15b-distill/final/ |
|
||||
| 铸渊1.5B蒸馏 | models/zhuyuan-15b-distill/final/ |
|
||||
| 霜砚深度微调 | models/shuangyan-15b-deep-sft/final/ |
|
||||
| 铸渊深度微调 | models/zhuyuan-15b-deep-sft/final/ |
|
||||
|
||||
## 八、踩坑提醒
|
||||
|
||||
1. 先5条数据验证全流程
|
||||
2. 检查transformers版本兼容
|
||||
3. pkill清理旧进程再启动
|
||||
4. python3 -u 强制flush
|
||||
5. 检查磁盘空间(需~50GB临时)
|
||||
Loading…
x
Reference in New Issue
Block a user