test/ModelLink2
1
0
Fork 0
mirror of https://gitee.com/ascend/ModelLink.git synced 2024-12-05 05:17:40 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

更新llama-7B、llama-13B README.md;新增llama-13B 16p参数脚本

Browse Source 
更新llama-7B、llama-13B README.md;新增llama-13B 16p参数脚本

更新llama-7B、llama-13B README.md;新增llama-13B 16p参数脚本

更新llama-7B、llama-13B README.md;新增llama-13B 16p参数脚本
This commit is contained in:
19952409173 2023-11-23 16:27:53 +08:00
parent 210f698b6c
commit f36b2414dc
12 changed files with 428 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@ -139,19 +139,19 @@ LLaMA2-13B/34B, Aquila-7B, Baichuan2-7B/13B 等模型即将上线...
<td>7B</td>
<td>1x8</td>
<td>FP16</td>
<td> 2862 </td>
<td> 2859 </td>
<td> <a href="sources/images/llama7b_loss.png">Loss</a> </td>
<td> 3689 </td>
<td> 3788 </td>
<td> <a href="sources/images/llama7b-loss-with-weight.png">Loss</a> </td>
<td> <a href="examples/llama/pretrain_llama_7B_zero_8p.sh">训练</a> </td>
</tr>
<tr>
<td>13B</td>
<td>1x8</td>
<td>FP16</td>
<td> 1800 </td>
<td> 1734 </td>
<td> <a href="sources/images/llama13b_loss.png">Loss</a> </td>
<td> <a href="examples/llama/pretrain_llama_13B_zero_8p.sh">训练</a> </td>
<td> 1958 </td>
<td> 2012 </td>
<td> <a href="sources/images/llama13b-loss-with-weight.png">Loss</a> </td>
<td> <a href="examples/llama/pretrain_llama_13B_ptd_8p.sh">训练</a> </td>
</tr>
<tr>
<td>33B</td>

8
README_en.md
View File

@ -138,8 +138,8 @@ Coming soon ...
<td>7B</td>
<td>1x8</td>
<td>FP16</td>
<td> 2862 </td>
<td> 2859 </td>
<td> 3689 </td>
<td> 3788 </td>
<td> <a href="sources/images/llama7b_loss.png">Loss</a> </td>
<td> <a href="examples/llama/pretrain_llama_7B_zero_8p.sh">Train</a> </td>
</tr>
@ -147,8 +147,8 @@ Coming soon ...
<td>13B</td>
<td>1x8</td>
<td>FP16</td>
<td> 1800 </td>
<td> 1734 </td>
<td> 1958 </td>
<td> 2012 </td>
<td> <a href="sources/images/llama13b_loss.png">Loss</a> </td>
<td> <a href="examples/llama/pretrain_llama_13B_zero_8p.sh">Train</a> </td>
</tr>

154
examples/llama/README.md
View File

@ -14,6 +14,9 @@
- [性能](#性能)
- [吞吐](#吞吐)
- [精度](#精度)
- [推理](#推理)
- [脚本](#脚本)
- [使用基线数据集进行评估](#使用基线数据集进行评估)
- [LLaMA-33B/65B](#LLaMA-33B/65B)
- [训练](#训练)
- [数据集](#数据集)
@ -27,15 +30,16 @@
LLaMA-7B/13B 训练的硬件配置如下:
| 硬件 | 配置 |
|:---:|:---------------:|
| NPU | 8 x Ascend NPUs |
| 硬件 | 设备 | 配置 |
|:----:|:------:|:----------------:|
| NPU | A+K | 8 x Ascend NPUs |
| NPU | A+X | 16 x Ascend NPUs |
LLaMA-7B/13B 训练的软件配置如下:
| 软件 | 配置 |
|:-------------------------:|:------------------------------------------------------------------------------------------------------------:|
| python | 3.7.16 |
| python | 3.7.5 |
| driver | [package](https://support.huawei.com/enterprise/zh/ascend-computing/atlas-900-pod-a2-pid-254184911/software) |
| firmware | [package](https://support.huawei.com/enterprise/zh/ascend-computing/atlas-900-pod-a2-pid-254184911/software) |
| CANN | [package](https://support.huawei.com/enterprise/zh/ascend-computing/cann-pid-251168373/software) |
@ -158,6 +162,7 @@ python $SCRIPT_PATH \
LLaMA-13B
```shell
# 单机八卡
mkdir model_weights
SCRIPT_PATH=./tools/ckpt_convert/llama/convert_weights_from_huggingface.py
python $SCRIPT_PATH \
@ -166,6 +171,16 @@ python $SCRIPT_PATH \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 8 \
--type 13B
# 单机16卡
mkdir model_weights
SCRIPT_PATH=./tools/ckpt_convert/llama/convert_weights_from_huggingface.py
python $SCRIPT_PATH \
--input-model-dir ./model_from_hf/llama-13b/ \
--output-model-dir ./model_weights/llama-13b \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 2 \
--type 13B
```
6. 配置 LLaMA-7B/13B 预训练脚本
@ -185,12 +200,15 @@ CHECKPOINT=./model_weights/
LLaMA-7B
```shell
bash examples/intern/pretrain_llama_7B_zero_8p.sh
bash examples/llama/pretrain_llama_7B_zero_8p.sh
```
LLaMA-13B
```shell
bash examples/intern/pretrain_llama_13B_ptd_8p.sh
# 单机8卡
bash examples/llama/pretrain_llama_13B_ptd_8p.sh
# 单机16卡
bash examples/llama/pretrain_llama_13B_ptd_16p.sh
```
### 性能
@ -199,12 +217,14 @@ bash examples/intern/pretrain_llama_13B_ptd_8p.sh
LLaMA-7B/13B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
| 设备 | 模型 | 迭代数 | 样本吞吐 (samples/p/s) | token吞吐 (tokens/p/s) | 单步迭代时间 (s/step) | 浮点计算数 (TFLOPs/s) |
|----|-----|-----|--------------------|----------------------|-----------------|------------------|
| NPUs | LLaMA-7B | 2048 | 1.398 | 2862 | 5.725 | 162.2 |
| 参考 | LLaMA-7B | 2048 | 1.395 | 2859 | 5.73 | 161.8 |
| NPUs | LLaMA-13B | 2048 | 0.879 | 1800 | 18.20 | 146.1 |
| 参考 | LLaMA-13B | 2048 | 0.847 | 1734 | 18.89 | 141.0 |
| 设备 | 硬件 | 模型 | 迭代数 | 样本吞吐 (samples/p/s) | token吞吐 (tokens/p/s) | 单步迭代时间 (s/step) | 浮点计算数 (TFLOPs/s) |
|------|-----------|-----------|------|------------------|----------------------|-----------------|------------------|
| NPUs | 910 1*8p | LLaMA-7B | 2048 | 1.80 | 3686 | 4.44 | 156.5 |
| 参考 | - | LLaMA-7B | 2048 | 1.85 | 3788 | 4.31 | 161.1 |
| NPUs | 910 1*8p | LLaMA-13B | 2048 | 0.956 | 1958 | 16.70 | 212.25 |
| NPUs | 910 1*16p | LLaMA-13B | 2048 | 0.88 | 1800 | 36.32 | 195.58 |
| 参考 | - | LLaMA-13B | 2048 | 0.98 | 2012 | 16.33 | 217.37 |
@ -218,7 +238,115 @@ LLama-13b NPU vs 参考 loss.
![NPU-Loss-with-weight-and-Relative-Error](../../sources/images/llama/llama13b-loss-with-weight.png)
## 推理
我们支持使用 LLaMA-7B 和 LLaMA-13B 进行文本生成的推理。
推理与预训练不同，比如我们需要加载预训练权重和输出样本的长度：
配置LLaMA-7B推理脚本`examples/llama/generate_llama_7B_deepspeed.sh`和LLaMA-13B推理脚本`examples/llama/generate_llama_13B_tp8_pp1.sh`。
```shell
# 修改模型权重路径和分词器路径
CHECKPOINT=<checkpoint-path>
VOCAB_FILE=<vocabfile-path>
```
LLaMA-7B:
```shell
bash ./examples/llama/generate_llama_7B_deepspeed.sh
```
LLaMA-13B:
```shell
bash ./examples/llama/generate_llama_13B_tp8_pp1.sh
```
部分推理样本如下：
LLaMA-7B:
![llama-7B_generate.png](../../sources/images/llama/llama-7B_generate.png)
LLaMA-13B:
![llama-13B_generate.png](../../sources/images/llama/llama-13B_generate.png)
## 使用基线数据集进行评估
我们使用 BBH benchmark 来评估我们的模型。Benchmark下载[此处](https://huggingface.co/datasets/lukaemon/bbh)。
配置LLaMA-7B评估脚本：
```shell
CHECKPOINT=./llama-7b-tp4-pp2/
VOCAB_FILE=./llama-7b-hf/
# 配置任务和数据路径
DATA_PATH="./bbh/data/test/"
TASK="bbh"
# 配置生成参数
python -m torch.distributed.launch $DISTRIBUTED_ARGS ./tasks/evaluation/evaluation.py \
--task-data-path $DATA_PATH \
--task $TASK\
--seq-length 2048 \
--max-new-tokens 32 \
--max-position-embeddings 2048 \
--tensor-model-parallel-size 4 \
--pipeline-model-parallel-size 2 \
--num-layers 32 \
--hidden-size 4096 \
--ffn-hidden-size 11008 \
--load ${CHECKPOINT} \
--num-attention-heads 32 \
--tokenizer-type PretrainedFromHF \
--tokenizer-name-or-path $VOCAB_FILE \
--tokenizer-not-use-fast \
--fp16 \
--micro-batch-size 1 \
--seed 42 | tee logs/evaluation.log
```
配置LLaMA-13B评估脚本：
```shell
CHECKPOINT=./llama-13b-tp1-pp8/
VOCAB_FILE=./llama-13b-hf/
# 配置任务和数据路径
DATA_PATH="./bbh/data/test/"
TASK="bbh"
# 配置参数
python -m torch.distributed.launch $DISTRIBUTED_ARGS ./tasks/evaluation/evaluation.py \
--task-data-path $DATA_PATH \
--task $TASK\
--seq-length 2048 \
--max-new-tokens 32 \
--max-position-embeddings 2048 \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 8 \
--num-layers 40 \
--hidden-size 5120 \
--ffn-hidden-size 13824 \
--load ${CHECKPOINT} \
--num-attention-heads 40 \
--tokenizer-type PretrainedFromHF \
--tokenizer-name-or-path $VOCAB_FILE \
--tokenizer-not-use-fast \
--fp16 \
--micro-batch-size 1 \
--seed 42 | tee logs/evaluation.log
```
```shell
# 开始评估
bash tasks/evaluation/eval.sh
```
LLaMA-7B/13B在**Ascend NPU**中的评测表现：
| 任务 | 模型 | 昇腾值 | 社区值 |
|-----------------------------------------------------|-----------|------|------|
| [BBH](https://huggingface.co/datasets/lukaemon/bbh) | LLaMA-7B | 33.7 | [33.5](https://opencompass.org.cn/dataset-detail/BBH) |
| [BBH](https://huggingface.co/datasets/lukaemon/bbh) | LLaMA-13B | 38.7 | [37.9](https://opencompass.org.cn/dataset-detail/BBH) |
# LLaMA-33B/65B
@ -450,4 +578,4 @@ LLaMa-65B
Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave*, Guillaume Lample*},
journal={arXiv preprint arXiv:2302.13971},
year={2023}}
```
```

158
examples/llama/README_en.md
View File

@ -13,6 +13,9 @@
- [Performance](#performance)
- [Machine performance](#machine-performance)
- [Accuracy of the loss](#accuracy-of-the-loss)
- [Inference](#Inference)
- [Script](#script)
- [Evaluation with Numerous Benchmarks](#Evaluation-with-Numerous-Benchmarks)
- [LLaMA-33B/65B](#llama-65b)
- [Training](#pre-training)
- [Datasets](#datasets)
@ -26,16 +29,17 @@
Here's a hardware summary of pre-training LLaMA-7B/13B:
| Hardware | Value |
| :------: | :---------------------------------------------: |
| NPU | 8 x Ascend NPUs |
| Hardware | Device | Value |
|:--------:|:------:|:----------------:|
| NPU | A+K | 8 x Ascend NPUs |
| NPU | A+X | 16 x Ascend NPUs |
Here's a software summary of pre-training LLaMA-7B/13B:
| Software | Version |
| :-----------------------: |:-----------:|
| Python | 3.7.16 |
| Software | Version |
| :-----------------------: |:------------------------------------------------------------------------------------------------------------:|
| Python | 3.7.5 |
| driver | [package](https://support.huawei.com/enterprise/zh/ascend-computing/atlas-900-pod-a2-pid-254184911/software) |
| firmware | [package](https://support.huawei.com/enterprise/zh/ascend-computing/atlas-900-pod-a2-pid-254184911/software) |
| CANN | [package](https://support.huawei.com/enterprise/zh/ascend-computing/cann-pid-251168373/software) |
@ -162,6 +166,7 @@ python $SCRIPT_PATH \
LLaMA-13B
```shell
# Single machine with 8p
mkdir model_weights
SCRIPT_PATH=./tools/ckpt_convert/llama/convert_weights_from_huggingface.py
python $SCRIPT_PATH \
@ -170,6 +175,16 @@ python $SCRIPT_PATH \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 8 \
--type 13B
# Single machine with 16p
mkdir model_weights
SCRIPT_PATH=./tools/ckpt_convert/llama/convert_weights_from_huggingface.py
python $SCRIPT_PATH \
--input-model-dir ./model_from_hf/llama-13b/ \
--output-model-dir ./model_weights/llama-13b \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 2 \
--type 13B
```
6. Config LLaMA-7B/13B pre-training script.
@ -193,7 +208,10 @@ bash examples/llama/pretrain_llama_7B_zero_8p.sh
LLaMA-13B
```shell
# 8p
bash examples/llama/pretrain_llama_13B_ptd_8p.sh
# 16p
bash examples/llama/pretrain_llama_13B_ptd_16p.sh
```
### Performance
@ -202,12 +220,13 @@ bash examples/llama/pretrain_llama_13B_ptd_8p.sh
The performance of LLaMA-7B/13B in **Ascend NPU** and **Reference**:
| Device | Model | total Iterations | throughput rate (samples/s/p) | throughput rate (tokens/s/p) | single-step time (s/step) | floating point operation (TFLOPs/s) |
| ------ |--------------| ---------------- |-------------------------------|------------------------------|---------------------------|-------------------------------------|
| NPUs | LLaMA-7B | 2048 | 1.398 | 2862 | 5.725 | 162.2 |
| Reference | LLaMA-7B | 2048 | 1.395 | 2859 | 5.73 | 161.8 |
| NPUs | LLaMA-13B | 2048 | 0.879 | 1800 | 18.20 | 146.1 |
| Reference | LLaMA-13B | 2048 | 0.847 | 1734 | 18.89 | 141.0 |
| Device | Hardware | Model | total Iterations | throughput rate (samples/s/p) | throughput rate (tokens/s/p) | single-step time (s/step) | floating point operation (TFLOPs/s) |
|-----------|----------|-----------|------------------|-------------------------------|------------------------------|---------------------------|-------------------------------------|
| NPUs | 910 1*8p | LLaMA-7B | 2048 | 1.80 | 3686 | 4.44 | 156.5 |
| Reference | - | LLaMA-7B | 2048 | 1.85 | 3788 | 4.31 | 161.1 |
| NPUs | 910 1*8p | LLaMA-13B | 2048 | 0.956 | 1958 | 16.70 | 212.25 |
| NPUs | 910 1*16p | LLaMA-13B | 2048 | 0.88 | 1800 | 36.32 | 195.58 |
| Reference | - | LLaMA-13B | 2048 | 0.98 | 2012 | 16.33 | 217.37 |
@ -215,10 +234,120 @@ The performance of LLaMA-7B/13B in **Ascend NPU** and **Reference**:
LLama-7b with huggingface weights NPU vs GPU loss.
![NPU-Loss-with-weight-and-Relative-Error](../../sources/images/llama/llama7b-loss-with-weight.png)
LLama-13b with huggingface weights NPU vs GPU loss.
![NPU-Loss-with-weight-and-Relative-Error](../../sources/images/llama/llama13b-loss-with-weight.png)
## Inference
We support AscendSpeed Inference for text generation with LLaMA-7B and LLaMA-13B.
Inference different from pre-training, such as we need to Load pre-training checkpoint and the length of the output samples:
Config LLaMA-7B inference script `examples/llama/generate_llama_7B_deepspeed.sh` and LLaMA-13B inference script `examples/llama/generate_llama_13B_tp8_pp1.sh`.
```shell
# modify the model weight path and tokenizer path
CHECKPOINT=<checkpoint-path>
VOCAB_FILE=<vocabfile-path>
```
LLaMA-7B:
```shell
bash ./examples/llama/generate_llama_7B_deepspeed.sh
```
LLaMA-13B:
```shell
bash ./examples/llama/generate_llama_13B_tp8_pp1.sh
```
Some inference samples are as follows:
LLaMA-7B:
![llama-7B_generate.png](../../sources/images/llama/llama-7B_generate.png)
LLaMA-13B:
![llama-13B_generate.png](../../sources/images/llama/llama-13B_generate.png)
## Evaluation with Numerous Benchmarks
We use bbh benchmark to evaluate our model. Benchmark Download [here](https://huggingface.co/datasets/lukaemon/bbh).
Config LLaMA-7B evaluation script:
```shell
CHECKPOINT=./llama-7b-tp4-pp2/
VOCAB_FILE=./llama-7b-hf/
# configure task and data path
DATA_PATH="./bbh/data/test/"
TASK="bbh"
# configure generation parameters
python -m torch.distributed.launch $DISTRIBUTED_ARGS ./tasks/evaluation/evaluation.py \
--task-data-path $DATA_PATH \
--task $TASK\
--seq-length 2048 \
--max-new-tokens 32 \
--max-position-embeddings 2048 \
--tensor-model-parallel-size 4 \
--pipeline-model-parallel-size 2 \
--num-layers 32 \
--hidden-size 4096 \
--ffn-hidden-size 11008 \
--load ${CHECKPOINT} \
--num-attention-heads 32 \
--tokenizer-type PretrainedFromHF \
--tokenizer-name-or-path $VOCAB_FILE \
--tokenizer-not-use-fast \
--fp16 \
--micro-batch-size 1 \
--seed 42 | tee logs/evaluation.log
```
Config LLaMA-13B evaluation script:
```shell
CHECKPOINT=./llama-13b-tp1-pp8/
VOCAB_FILE=./llama-13b-hf/
# configure task and data path
DATA_PATH="./bbh/data/test/"
TASK="bbh"
# configure generation parameters
python -m torch.distributed.launch $DISTRIBUTED_ARGS ./tasks/evaluation/evaluation.py \
--task-data-path $DATA_PATH \
--task $TASK\
--seq-length 2048 \
--max-new-tokens 32 \
--max-position-embeddings 2048 \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 8 \
--num-layers 40 \
--hidden-size 5120 \
--ffn-hidden-size 13824 \
--load ${CHECKPOINT} \
--num-attention-heads 40 \
--tokenizer-type PretrainedFromHF \
--tokenizer-name-or-path $VOCAB_FILE \
--tokenizer-not-use-fast \
--fp16 \
--micro-batch-size 1 \
--seed 42 | tee logs/evaluation.log
```
```shell
# start evaluation
bash tasks/evaluation/eval.sh
```
The evaluation performance of LLaMA-7B/13B in **Ascend NPU**:
| Task | Model | NPU | Benchmark |
|---------|-----------|------|-----------|
| [BBH](https://huggingface.co/datasets/lukaemon/bbh) | LLaMA-7B | 33.7 | [33.5](https://opencompass.org.cn/dataset-detail/BBH) |
| [BBH](https://huggingface.co/datasets/lukaemon/bbh) | LLaMA-13B | 38.7 | [37.9](https://opencompass.org.cn/dataset-detail/BBH) |
# LLaMA-33B/65B
@ -240,7 +369,7 @@ Here's a hardware summary of training llama:
Here's a software summary of training llama:
|
| Software | Version |
| :-----------------------: |:-----------:|
| Python | 3.7 |
@ -457,6 +586,3 @@ You may also consider original work in your reference:
journal={arXiv preprint arXiv:2302.13971},
year={2023}}
```
\
\
<font size=1>If the download of the file fails using 'wget' , you can download it manually while ensuring website security.</font>

55
examples/llama/generate_llama_7B_deepspeed.sh Normal file
View File

@ -0,0 +1,55 @@
#!/bin/bash
export TOKENIZERS_PARALLELISM=false
NNODES=1
NPUS_PER_NODE=8
CHECKPOINT="your megatron checkpoint path"
VOCAB_FILE="your vocab path"
ZERO_STAGE=0
MICRO_BATCH_SIZE=1
config_json="./ds_config.json"
cat <<EOT > $config_json
{
"train_micro_batch_size_per_gpu": $MICRO_BATCH_SIZE,
"gradient_clipping": 1.0,
"zero_optimization": {
"stage": $ZERO_STAGE
},
"fp16": {
"enabled": true,
"loss_scale": 0,
"loss_scale_window": 1000,
"hysteresis": 2,
"min_loss_scale": 1,
"initial_scale_power": 8
},
"steps_per_print": 2000,
"wall_clock_breakdown": false
}
EOT
deepspeed --num_nodes $NNODES --num_gpus $NPUS_PER_NODE \
./tasks/inference/inference_llama.py \
--no-contiguous-buffers-in-local-ddp \
--tensor-model-parallel-size 8 \
--num-layers 32 \
--hidden-size 4096 \
--ffn-hidden-size 11008 \
--num-attention-heads 32 \
--max-position-embeddings 2048 \
--tokenizer-type PretrainedFromHF \
--load "${CHECKPOINT}" \
--tokenizer-name-or-path "$VOCAB_FILE" \
--tokenizer-not-use-fast \
--fp16 \
--micro-batch-size ${MICRO_BATCH_SIZE} \
--seq-length 2048 \
--max-new-tokens 64 \
--seed 42 \
--deepspeed \
--deepspeed_config ${config_json} \
--no-pipeline-parallel \

62
examples/llama/pretrain_llama_13B_ptd_16p.sh Normal file
View File

@ -0,0 +1,62 @@
# This is an example: train llama using PTD.
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib:/root/miniconda3/lib:$LD_LIBRARY_PATH
export HCCL_CONNECT_TIMEOUT=1200
export COMBINED_ENABLE=1
source /usr/local/Ascend/ascend-toolkit/set_env.sh
# Change for multinode config
MASTER_ADDR=localhost
MASTER_PORT=6001
NNODES=1
NODE_RANK=0
NPUS_PER_NODE=16
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
DATA_PATH=<data-path>
LOAD_CHECKPOINT_PATH=<origin-ckpt-path>
SAVE_CHECKPOINT_PATH=<ckpt-path>
TOKENIZER_PATH=<tokenizer-path>
DISTRIBUTED_ARGS="--nproc_per_node $NPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT"
# Main script
python -m torch.distributed.launch $DISTRIBUTED_ARGS \
pretrain_llama.py \
--DDP-impl local \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 2 \
--num-layers 40 \
--hidden-size 5120 \
--ffn-hidden-size 13824 \
--num-attention-heads 40 \
--micro-batch-size 1 \
--global-batch-size 512 \
--seq-length 2048 \
--max-position-embeddings 2048 \
--train-iters 5000 \
--lr-decay-iters 5000 \
--load $LOAD_CHECKPOINT_PATH \
--data-path $DATA_PATH \
--tokenizer-name-or-path $TOKENIZER_PATH \
--tokenizer-not-use-fast \
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--data-impl mmap \
--split 949,50,1 \
--distributed-backend nccl \
--lr 1.0e-6 \
--lr-decay-style cosine \
--min-lr 1.0e-7 \
--weight-decay 1e-2 \
--clip-grad 1.0 \
--lr-warmup-fraction .01 \
--log-interval 1 \
--save-interval 10000 \
--eval-interval 1000 \
--eval-iters 10 \
--initial-loss-scale 4096.0 \
--checkpoint-activations \
--use-fused-rotary-pos-emb \
--use-flash-attn \
--use-distributed-optimizer \
--fp16 | tee logs/train_llama_13B.log

1
examples/llama/pretrain_llama_13B_ptd_8p.sh
View File

@ -55,4 +55,5 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--triangle-attn \
--use-fused-rotary-pos-emb \
--release-fp32-grad \
--use-flash-attn \
--fp16 | tee logs/train_13B.log

28
examples/llama/pretrain_llama_7B_zero_8p.sh
View File

@ -19,7 +19,7 @@ CHECKPOINT=./ckpt
DS_CONFIG=deepspeed_config_7B.json
ZERO_STAGE=2
GLOBAL_BATCH=64
MICRO_BATCH=8
MICRO_BATCH=2
cat <<EOT > $DS_CONFIG
{
@ -46,7 +46,7 @@ cat <<EOT > $DS_CONFIG
"contiguous_gradients": true
},
"gradient_accumulation_steps": 1,
"gradient_accumulation_steps": 4,
"train_batch_size": $GLOBAL_BATCH,
"train_micro_batch_size_per_gpu":$MICRO_BATCH,
"zero_allow_untested_optimizer": true
@ -73,26 +73,30 @@ deepspeed pretrain_llama.py \
--seq-length 2048 \
--max-position-embeddings 2048 \
--train-iters 500000 \
--lr-decay-iters 320000 \
--save $CHECKPOINT \
--data-path $DATA \
--tokenizer-name-or-path ./dataset/llama/ \
--tokenizer-not-use-fast \
--data-impl mmap \
--split 949,50,1 \
--attention-softmax-in-fp32 \
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--init-method-std 0.01 \
--split 900,50,50 \
--distributed-backend nccl \
--lr 0.000015 \
--lr 1.0e-6 \
--lr-decay-style cosine \
--lr-warmup-fraction .01 \
--min-lr 1.0e-6 \
--weight-decay 1e-2 \
--clip-grad 1.0 \
--lr-warmup-fraction .01 \
--checkpoint-activations \
--adam-beta1 0.9 \
--adam-beta2 0.95 \
--log-interval 1 \
--save-interval 10000 \
--eval-interval 1000 \
--eval-iters 10 \
--triangle-attn \
--use-fused-rotary-pos-emb \
--no-load-optim \
--no-load-rng \
--no-bias-gelu-fusion \
--no-query-key-layer-scaling \
--use-flash-attn \
$ds_args \
--fp16 | tee logs/train_7B.log

BIN
sources/images/llama/llama-13B_generate.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

BIN
sources/images/llama/llama-7B_generate.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.3 KiB

BIN
sources/images/llama/llama13b-loss-with-weight.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 305 KiB

BIN
sources/images/llama/llama7b-loss-with-weight.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 42 KiB

After

Width:  |  Height:  |  Size: 261 KiB