test/fastNLP
1
0
Fork 0
mirror of https://gitee.com/fastnlp/fastNLP.git synced 2024-12-04 05:07:44 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,208 Commits 3 Branches 13 Tags 25 MiB
e813aeaa6f
Commit Graph

259 Commits

Author SHA1 Message Date
x54-729
cf19062fb2 set_dist_repro_dataloader测试例的完善 2022-04-15 09:06:22 +00:00
x54-729
cca265f99c Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-15 08:10:19 +00:00
x54-729
d0f26c7c34 将validate_step替换为evaluate_step 2022-04-15 08:10:01 +00:00
yh_cc
7c70874b4a 删除core.sampelrs.sampler.py；增加torch的clipgradient和warmupcallback 2022-04-15 16:04:43 +08:00
MorningForest
3ea74b52d2 Merge remote-tracking branch 'refs/remotes/origin/dev0.8.0' into dev0.8.0 2022-04-15 14:00:39 +08:00
yh_cc
f27d53261c 修改部分测试用例中validate_dataloader为evaluate_dataloader 2022-04-15 12:30:20 +08:00
x54-729
f6f489dc90 多卡 set_dist_repro_dataloader 的测试例 2022-04-14 16:17:20 +00:00
x54-729
98644d2d0b fix conflict 2022-04-14 16:05:53 +00:00
x54-729
b97962b8dd 简化paddle trainer的单卡测试例 2022-04-14 16:04:56 +00:00
x54-729
5c29cd384a 调整num_consumed_batches在driver save中的逻辑 2022-04-14 16:04:12 +00:00
yh_cc
a4b2e0fac5 修复若干bug 2022-04-15 00:01:29 +08:00
YWMditto
2924e2117f 删除了 driver 中的 **_step，使用 model_call 和 get_model_call_fn 来代替；删除了 driver 中的所有 dataloaders 2022-04-14 23:34:35 +08:00
MorningForest
4cad7f548d Merge remote-tracking branch 'refs/remotes/origin/dev0.8.0' into dev0.8.0 2022-04-14 19:37:02 +08:00
MorningForest
4be26c5620 修改torch fdl 2022-04-14 19:36:50 +08:00
yh_cc
16a467393c 1.montior允许传入callable的对象进行选择; 2.解决Sampler中存在的循环引用问题 2022-04-14 16:02:41 +08:00
YWMditto
2f23d80ccc 修改了 trainer 中的 validate 的调用的逻辑 2022-04-14 00:45:17 +08:00
YWMditto
8c0b0b8cd0 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-13 19:09:34 +08:00
YWMditto
b9b0b53430 将 Events 修改为小写 2022-04-13 19:09:27 +08:00
yh_cc
31c6d5d02a merge 2022-04-13 17:05:16 +08:00
yh_cc
9d71170bef 解决Trainer在断点重训的时候无法实现准确load和保存的问题 2022-04-13 17:04:33 +08:00
YWMditto
1ac9e75c50 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-13 15:37:15 +08:00
YWMditto
3ee6fc66f5 添加了 on_after_optimizers_step 和 on_after_zero_grad 的callback接口 2022-04-13 15:37:08 +08:00
x54-729
c2575ab357 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-13 07:33:40 +00:00
x54-729
3ab93b2fae paddle driver单卡和utils的pytest测试，添加了断点重训的测试 2022-04-13 07:33:27 +00:00
YWMditto
eb8d761a05 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-13 14:27:08 +08:00
YWMditto
76a1e69022 little change 2022-04-13 14:27:01 +08:00
x54-729
5acaeabae4 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-13 05:27:09 +00:00
yh_cc
e8d11cd5a9 1. 修复torch 分布式在不同版本中group参数default值不一样的问题; 2. torch修复多卡时只有batchsampler evaluate会遇到bug的问题; 3。logger增加warning_once接口;4.增加callback相关文档 2022-04-13 12:55:28 +08:00
x54-729
c61f28ce8e Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-12 20:48:47 +00:00
YWMditto
8c22d0b1f6 修改了 Trainer.on 的错误提示 2022-04-12 22:47:39 +08:00
x54-729
a589cb5ddc Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-12 09:24:16 +00:00
YWMditto
1443065cbd Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-12 17:00:18 +08:00
YWMditto
5419b6a042 填了了关闭参数匹配的逻辑；添加了 trainer 中获取 driver 参数的接口 2022-04-12 17:00:07 +08:00
x54-729
e08ae33bfd Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-12 08:48:16 +00:00
MorningForest
7a68f90b56 增加ClassifyF1PreRecMetric的多卡测试用例 2022-04-12 16:06:25 +08:00
x54-729
fcd5125009 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-12 07:34:44 +00:00
MorningForest
ab3b66715e Merge remote-tracking branch 'refs/remotes/origin/dev0.8.0' into dev0.8.0 2022-04-12 13:52:13 +08:00
MorningForest
c7616f8ba8 修改ClassifyF1PreRecMetric及对应的测试用例 2022-04-12 13:51:54 +08:00
yh_cc
a6ff3f8cc3 1. 将带有Monitor的callback都抽象为HasMonitorCallback，并由这个父类进行monitor的设置和检验的; 2.支持从Trainer中设置monitor给所有的Callback使用;3.新增EarlyStopCallback. 2022-04-12 13:49:03 +08:00
yh_cc
92b00d825f 修改fastnlp的dist功能 2022-04-12 02:19:28 +08:00
x54-729
00b5baf67a 整理PaddleSingleDriver的部分测试例 2022-04-11 17:39:19 +00:00
x54-729
3a3c38a44e fix conflict 2022-04-11 14:16:39 +00:00
x54-729
d1a589147a 修复测试例 2022-04-11 14:08:25 +00:00
yh_cc
14fffcb36c 新增SortedSampler和SequentialSampler 2022-04-11 21:44:53 +08:00
MorningForest
ffcf3ddcd3 修改测试用例unittest为pytest 2022-04-11 19:39:53 +08:00
MorningForest
36149a57b0 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-11 18:44:31 +08:00
MorningForest
f81d2e0552 合并冲突 2022-04-11 18:43:30 +08:00
yh_cc
9216c6117e 新增UnrepeatedSortedSampler 2022-04-11 16:44:08 +08:00
yh_cc
607367588c 1.增加DataSet的__setitem__方法，使得其可以直接random.shuffle(dataset); 2.优化部分log输出显示 2022-04-11 15:21:45 +08:00
yh_cc
ce835212e6 将CheckpointCallback拆分为ModelCheckpointCallback和TrainerCheckpointCallback，修改了部分实现 2022-04-11 14:34:11 +08:00
yh_cc
e8b6298fb1 完善SpanFPreRecMetric即测试 2022-04-10 23:56:55 +08:00
MorningForest
9c62e300a7 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-10 23:10:44 +08:00
MorningForest
0cab4dc526 修改F1RecPreMetric测试用例 2022-04-10 23:10:35 +08:00
x54-729
ebfa118ff2 PaddleDriver的测试例调整 2022-04-10 15:07:52 +00:00
x54-729
193c04c9e2 initialize_paddle_driver的测试例 2022-04-10 14:57:55 +00:00
yh_cc
817f6d6ad6 修改fastnlp_torch_all_gather函数，使得它gather后的tensor都在当前device 2022-04-10 22:28:48 +08:00
x54-729
08ee15fa10 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-10 12:49:38 +00:00
yh_cc
12d3e08568 增加BucketedBatchSampler的测试 2022-04-10 20:42:37 +08:00
x54-729
447e673afa Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-10 06:55:17 +00:00
x54-729
791580797c paddle 分布式的测试例 2022-04-10 06:55:05 +00:00
yh_cc
983c587cba Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-10 14:36:52 +08:00
yh_cc
20b8ca9a92 实现了BucketedBatchSampler; 创建了reproducible_batch_sampler.py 2022-04-10 14:36:47 +08:00
YWMditto
a376eea776 重新修改了断点重训的逻辑，主要修改了 trainer.save/load 和 driver.save 和 load 函数 2022-04-10 12:56:49 +08:00
YWMditto
770b2eaf4b Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-10 11:47:01 +08:00
yh_cc
0e7961cd9f Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-10 00:08:24 +08:00
yh_cc
8e4abf2aa5 删除了driver的replace_sampler替换为set_dist_repro_dataloader; 同时修改 driver.load/driver.save 函数 2022-04-10 00:08:19 +08:00
x54-729
680421fe86 paddle分布式训练 2022-04-09 12:52:19 +00:00
YWMditto
a8597602d0 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-09 15:28:19 +08:00
YWMditto
929abc3953 加入了 test_logger.py 2022-04-09 15:28:13 +08:00
x54-729
6ef912675b 新的paddle测试数据集 2022-04-09 14:37:55 +08:00
x54-729
fd3b82c5b0 Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-08 15:28:34 +00:00
yh_cc
4781178a5a 修复测试 2022-04-08 22:36:52 +08:00
x54-729
2d68c952d3 调整paddle_backend 2022-04-08 22:34:17 +08:00
yh_cc
40c0a712dd 移动env的位置为顶级目录 2022-04-08 22:29:22 +08:00
YWMditto
e3ce87271e Merge branch 'dev0.8.0' of github.com:fastnlp/fastNLP into dev0.8.0 2022-04-08 21:48:46 +08:00
YWMditto
fe270ef619 更新 callbacks, controllers, drivers, samplers 以及 tests 中的部分内容 2022-04-08 21:48:29 +08:00
MorningForest
8cf855ab47 添加了metric测试用例 2022-04-08 21:37:12 +08:00
MorningForest
6960eb56b6 添加了dataset测试用例 2022-04-08 21:36:41 +08:00
MorningForest
9ed4df6f86 添加了dataloaders测试用例 2022-04-08 21:36:12 +08:00
MorningForest
65c621db78 添加了collators测试用例 2022-04-08 21:35:39 +08:00
x54-729
3a580edb99 提交tests/core/utils/ 2022-04-08 12:16:05 +00:00
x54-729
05ca80bb6f 提交tests/modules/ 2022-04-08 12:15:33 +00:00
x54-729
6170976b7f 提交tests/core/drivers/torch_paddle_driver 2022-04-08 12:14:33 +00:00
x54-729
23502a0753 提交tests/core/drivers/paddle_driver 2022-04-08 12:14:20 +00:00
x54-729
96a565c763 提交tests/core/drivers/jittor_driver 2022-04-08 12:14:03 +00:00
x54-729
d8ce9a7915 提交/tests/core/drivers/ 2022-04-08 12:13:11 +00:00
yh_cc
259cbf80db 提交log/envs 2022-04-08 19:23:08 +08:00
yh_cc
916a2d2c30 init dev0.8.0 2022-04-08 01:21:26 +08:00
hw
c0aa5bd932
增加Mr, R8, R52, Ohsumed, 20ng数据集的loader ,pipe (#382) 
* 增加Mr, R8, R52, Ohsumed, 20ng数据集的loader ,pipe

* "更改_20ng为NG20"
 2021-09-23 14:40:07 +08:00
yh_cc
c72bc070a9 1.修复yxg同学发现的DataSet copy Field导致的bug; 2. 修改DistTrainer中update_every的bug; 3.Trainer和DistTrainer修改pin_memory默认值为True 2021-09-18 00:00:38 +08:00
WillQvQ
d9e0450ae7 Merge branch 'master' of github.com:fastnlp/fastNLP into dev 2021-04-12 10:30:28 +08:00
yh_cc
3e9330d250 解决https://github.com/fastnlp/fastNLP/issues/355 的问题 2021-03-30 11:12:26 +08:00
sleepy-owl
2b91830c26
Flaky test fix (#352) 2021-02-23 19:04:05 +08:00
yh_cc
04ad8e604e DataSet apply的时候可以传入use_tqdm和tqdm_desc 2021-01-05 23:45:15 +08:00
yh_cc
dc280fa1cb 删除对spacy的依赖 2020-12-11 14:03:10 +08:00
yh_cc
7ca8d045e7 update for basemodel bug 2020-12-11 13:42:18 +08:00
yh_cc
50d7cfdd7a 在dataset中添加concat函数，支持将两个dataset concat起来 2020-12-09 19:45:55 +08:00
yunfan
f711d3070a dist_trainer for fp16 2020-12-06 03:48:08 +00:00
yh_cc
2e8350cd16 Trainer支持fp16 2020-12-05 22:30:23 +08:00
willqvq
350772f518 修改读取测试数据的位置 2020-11-23 13:34:05 +08:00
willqvq
fc8f8f6913 把测试文件夹 test 改名为 tests 2020-11-23 13:00:31 +08:00
FengZiYjun
32652407df restructure files & add "modules" directory & add CRF.py 2018-07-01 10:39:36 +08:00
FengZiYjun
3e1d995b3c update file structures 2018-06-25 14:16:47 +08:00
FengZiYjun
58127d3c4e start building word seg (generally seq2seq) model 2018-05-30 22:28:22 +08:00
FengZiYjun
52b1337e8b READY TO GO: test_charlm tested 2018-05-25 17:40:28 +08:00
FengZiYjun
3081a57ef9 optimize trainer logic & prepare charlm test 2018-05-24 10:45:01 +08:00
FengZiYjun
c049b2fe1f finish basic logic of trainer & tester, wait for unittest 2018-05-23 23:03:51 +08:00
FengZiYjun
cd7b6b1e8d add toy model to check data flow all right 2018-05-22 16:28:33 +08:00
FengZiYjun
7b46f422c7 add base methods for model.base_model 2018-05-22 11:15:27 +08:00