31 Commits

Author	SHA1	Message	Date
fengliangjun	7a21f0bf58	up	2023-09-26 14:30:47 +08:00
yangyuan667	be5d413ec6	Won't Import ascendspeed.ops if unneccssary	2023-08-16 09:35:10 +08:00
xuqiang	8e0f2e9b1a	sync parallel_state.py file	2023-08-15 10:45:26 +08:00
yangyuan667	ff9d62bcc7	Add gcc compiler args	2023-08-11 11:44:28 +08:00
machangjun	8e436e3a9a	add ffts mode ... del torch_trans del torch_trans and resove bloom ckpt and add bloom ffts+ add ffts mode del torch_trans del torch_trans and resove bloom ckpt and add bloom ffts+ replace fused_adam to adam del unused code	2023-07-25 14:14:28 +08:00
yangyuan667	2ae07e63ee	[New]add FlashAttention adpater	2023-08-10 11:11:24 +08:00
kingsleyandher	e4bee6c48a	Layer Fusion for LLama	2023-07-25 15:25:55 +08:00
fengliangjun	260e8eea8f	create megatron core	2023-07-24 15:00:57 +08:00
fengliangjun	b559dc6385	set log level	2023-07-20 22:23:33 +08:00
chenzomi	92c27d5e2a	add a llama2 brach.	2023-07-21 15:20:25 +08:00
fengliangjun	db9c25bdd9	llama modify	2023-07-19 10:20:40 +08:00
machangjun	de85201818	modify baddbmm to bmm to accelerate	2023-07-14 15:41:00 +08:00
simon717	fedb2127c0	1. llama_model.py attention实现回退 ... 2. huggingface llama权重转换脚本 3. llama并行训练策略改变后权重转换脚本	2023-07-05 15:30:44 +08:00
liulinfeng	243bfe5cfa	Bloom适配SP代码	2023-07-14 14:18:07 +08:00
chenzomi	937791fa6d	format some code.	2023-07-21 01:42:26 +08:00
kingsleyandher	31cf1ecdd0	merge code	2023-07-11 20:12:39 +08:00
liulinfeng	85e23be9f2	删除保存初始权重的代码	2023-07-10 11:16:43 +08:00
liulinfeng	8aa62e1049	修复codecheck问题，删除多余的空行	2023-07-10 10:54:13 +08:00
liulinfeng	36f787bc89	Author：刘林峰 ... 修改说明： 1、提交权重加载、推理生成文本的代码实现 2、修改codecheck问题 3、修复断点续训卡死的问题	2023-07-07 15:03:48 +08:00
kingsleyandher	3afb525a97	提交SP算法	2023-07-10 14:44:42 +08:00
kingsleyandher	b5a0fc04a2	Optimizer Pipeline parallel ... Author: 李冰聪	2023-07-07 11:41:45 +08:00
wiyr	d87e921410	added trick	2023-06-30 11:00:38 +08:00
kingsleyandher	bc2a4a33d5	提交VP算法 ... Author: 李冰聪/张梦阳	2023-06-29 10:22:17 +08:00
wiyr	6304cab765	remove useless code	2023-06-20 16:54:12 +08:00
machangjun	2d8c6fee9d	add bloom st and adapt new data load method ... modify bloom st run modify bloom st run modify times add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add new pretrain_bloom.py add st	2023-06-17 17:36:17 +08:00
simon717	7047d75663	Llama模型结构与huggingface对齐, 前向推理结果与huggingface对齐	2023-06-20 11:27:15 +08:00
kingsleyandher	4e3b7cd992	LlamaTokenizer适配及预训练脚本更改	2023-06-13 12:33:33 +08:00
wiyr	2f826f7351	can run with bloom7b and pass ci	2023-06-12 14:42:29 +08:00
wiyr	89bfaf64c2	added fused softmax, tokenizer, language, utils for bloom	2023-06-12 11:50:46 +08:00
wiyr	6d07c029fe	added func code	2023-06-12 10:39:51 +08:00
chenzomi	37cc0b949d	change megatron to ascendspeed	2023-06-10 21:26:01 +08:00