Commit Graph

7328 Commits

Author SHA1 Message Date
Jiajie Zhong
a86f4e2693
[fix][python] Task switch branch not show in webui (#12120)
when create task switch in python api, switch branch not show correctly,
due to we add some unnecessary attribute to switch task, this patch we
add `_task_ignore_attr` in class `Task` to ignore those unnecessary
attribute from `_task_default_attr`
2022-09-23 19:41:05 +08:00
Jiajie Zhong
2531c222de
[Fix] Also auto formatter workflow instance if location is null (#12080)
* [fix] Also auto formatter workflow instance if location is null

In #11535(096fae77) and #11681(4dca488c), we already formatter
workflow definition, but I find out we forget formatter workflow
instance with definition's location is null, this patch also
formatter workflow instance.

* remove console log
2022-09-23 18:20:39 +08:00
JieguangZhou
6eb1eb722a
fix python home conflict (#12112) 2022-09-23 09:40:18 +08:00
Amy0104
e11e129cda
[Improvement][UI] Add filterable for src_table and target_table in the data quality task. (#12099) 2022-09-22 20:43:34 +08:00
chenrj
b1b57c8fb7
[feat][python] new resource plugin GitHub and gitlab (#11831)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: JieguangZhou <jieguang_zhou@163.com>
Co-authored-by: Devosend <devosend@gmail.com>
Co-authored-by: juzimao <578961953@qq.com>
2022-09-22 14:43:00 +08:00
jackfanwan
6e03a7e557
[BUG]Delete Setting ID (#12096)
* Delete Setting ID

* modify id

Co-authored-by: jackfanwan <fanwanlong@kezaihui.com>
2022-09-22 12:12:43 +08:00
Lyle Shaw
cc492c3e13
[Feature] Add CURD to the project/tenant/user section of the python-DS (#11162)
- Add CURD in project
- Add CURD in tenant
- Add CURD in user
- Add test in user

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-22 11:03:27 +08:00
Wenjun Ruan
fba5a8eaa0
Fix insert command error due to the id is not null (#12092) 2022-09-22 09:52:32 +08:00
冯剑
9b3b4e22b4
[Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported (#12059)
* [Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported

Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
2022-09-22 09:39:02 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
JieguangZhou
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use (#12071)
optimize code
2022-09-21 10:45:05 +08:00
Eric Gao
b52da64001
[Improvement][UT] Upgrade jUnit to 5.+ (#10976) (#11332)
* [Improvement][UT] Upgrade jUnit to 5.+ (#10976)

* Refactor AlertServerTest with jUnit-5 as an example
2022-09-21 10:15:21 +08:00
jackfanwan
7d80ea34ac
[Improvement] Add Set cluster name (#12074) 2022-09-21 10:05:15 +08:00
limaiwang
ed1d1e8856
add(dependent node's add unprivileged project list): dependent node's add unprivileged project list (#12073) 2022-09-21 09:28:22 +08:00
Wenjun Ruan
cb4ef54a0e
Script cannot contains ''' in params (#12067) 2022-09-21 08:52:51 +08:00
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center (#11708)
* Add support for Alibaba Cloud OSS as storage of resource center

* Add dependency licenses

* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
Wenjun Ruan
26dad5e2a8
Fix update TaskDefinition error (#12060) 2022-09-20 22:23:16 +08:00
jackfanwan
0bf639b851
[improvement]Add Set cluster name (#12058)
* Add Set cluster name

* add unit test

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-09-20 21:26:05 +08:00
insist777
955e8ebe0f
[Improve] Fix support test tasks UT (#12062) 2022-09-20 15:46:29 +08:00
金鑫
649015bfff
[Bug][ApiServer] When modifying the task definition, inserting the task definition log table will report the bug of primary key conflict #12037
Co-authored-by: 金鑫 <jinyanhui@huansi.net>
2022-09-20 14:42:21 +08:00
Eric Gao
e1b55db72a
[Improvement][Test] Give an example on replacing Powermock with Mockito (#11588)
* Give an example on replacing powermock with mockito

* Remove redundant comments

* Refactoring related UTs for better readability.
2022-09-20 09:48:02 +08:00
rickchengx
b3bbf69b2e
[Improvement-11291][sql] Improvement the log output while executing multiple sql statements (#11292) 2022-09-20 09:30:27 +08:00
songjianet
8cddb10773
[Feat][UI] Update package. (#12046)
* [Feat][UI] Update package.

* [Feat][UI] Update package.
2022-09-19 20:58:41 +08:00
Kerwin
89d55bc9b7
Fix link errors in release documentation. (#12049) 2022-09-19 20:01:23 +08:00
juzimao
c8884e4f2f
and task instance date range search (#11963) 2022-09-19 17:50:24 +08:00
JieguangZhou
ad683c3c42
[feat][python] Support MLflow task in python api (#11962) 2022-09-18 16:28:18 +08:00
limfff
c24ad9adb6
[Fix] Fixed an error importing project Maven (#12026) 2022-09-18 10:11:20 +08:00
JieguangZhou
bbc228e93f
[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
* add dvc task in pyds

* add BaseDVC class
2022-09-17 16:55:51 +08:00
JieguangZhou
e50a32d9cc
[feat][python] Support OpenMLDB task in python api (#11944)
Using the whole word zookeeper instead of short cut zk
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-17 15:24:17 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task (#11670)
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
Eric Gao
a44817fc46
[Doc][Style] Fix doc format once for all (#12006) 2022-09-17 11:33:32 +08:00
sgw
1a7c6eabf3
[Improvement-11773][api] Optimize the log printing of the api module according… (#11782)
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Resolve conflict.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- fix ProcessDefinitionServiceImpl.java dependency

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Accept proposed change in ResourcePermissionCheckServiceImpl.java

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Update the UT case in TenantServiceTest.java, UsersServiceTest.java, TaskGroupServiceTest.java.
- Update log printing in TaskGroupServiceImpl.java.
- Fix NPE of log parameter in ExecutorServiceImpl.java when call python api.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix NPE of log in WorkerGroupServiceImpl.java.
- Fix log parameter missing bug in ResourcesServiceImpl.java.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix bugs and handle some vulnerability codes.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix bugs and handle some vulnerability codes.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix e2e User test.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix codeQL check & SonarCloud Code Analysis.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Resolve import conflict.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix codeQL check & SonarCloud Code Analysis.
2022-09-17 11:28:04 +08:00
JieguangZhou
e05f14ba23
[feat][python] Add sagemaker yaml example (#11925) 2022-09-17 10:34:06 +08:00
JieguangZhou
5202e5cfc6
[feat][python] Support Pytorch task in python api (#11975)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-16 20:24:38 +08:00
Wenjun Ruan
864a90820d
Fix insertOrUpdate plugin may failed due to concurrent operation (#11471) 2022-09-16 15:39:04 +08:00
Wenjun Ruan
d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
jackfanwan
71b4087421
Example Change the regular expression of the data source password (#11459) 2022-09-16 13:34:22 +08:00
kezhenxu94
5811b84fcc
Add validations of possible malicious keys (#11966) 2022-09-16 13:32:59 +08:00
Wenjun Ruan
4ad34483a5
Fix alert_send_status may throw duplicate key exception and add limit for query alert (#11953) 2022-09-16 13:26:33 +08:00
rickchengx
5b6bebf7d5
[fix][doc] Update the registry related configuration in values.yaml (#11444) 2022-09-16 10:38:27 +08:00
ZhaoGuodong
86291bb5e6
[Feature-10273][Datasource] Add dataSource for AWS Athena (#11157) 2022-09-16 09:59:20 +08:00
caishunfeng
925e2fa551
[Bug][Workflow Definition] fix ordinary users can not create depend task (#11961) 2022-09-16 09:12:33 +08:00
kezhenxu94
b73a9511a7
Remove sensitive configs from dolphinscheduler_env.sh (#11917) 2022-09-15 19:03:47 +08:00
itzhang89
511149f644
[fix-10938]: use dot to replace source and make the default env work in shell (#11625)
* [fix-10938]: use the dot to replace source and make the default env work in the sh
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-09-15 15:05:09 +08:00
Wenjun Ruan
9813ded981
Add debug logs for properties read from common.properties (#11910) 2022-09-15 10:53:19 +08:00
Eric Gao
a88cb02ee8
[improve][Doc] Put default task parameters in a new file (#11776)
* Put default task parameters in a new file
* Add links to task plugin default parameters appendix docs
* Update Chinese docs
2022-09-15 10:32:10 +08:00
Stalary
bd0fa79e37
[Bug][Dependent]: Id also clone due to duplicate when use dependent mode. (#11929) 2022-09-15 10:00:38 +08:00
Yann Ann
e938fdbe96
[Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 (#11864)
* [Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838
2022-09-15 09:46:30 +08:00
xuhhui
dec6197a67
[Feature][Api] Refactor&Add api.controller.QueueV2Controller (#11389)
* [Feature][Api] Refactor&Add api.controller.QueueV2Controller

* update review
2022-09-15 09:37:12 +08:00
caishunfeng
6868876a29
fix workflow keep running when task fail (#11930) 2022-09-15 09:06:10 +08:00