Commit Graph

7437 Commits

Author SHA1 Message Date
冯剑
779a0b9c75
[Improvement-#11608][task-plugin] New time variables are added to facilitate business development (#11667)
Add new time expressions for users' convenience

Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
Co-authored-by: David <dailidong66@gmail.com>
2022-09-26 16:01:32 +08:00
caishunfeng
151c2d2c70
fix upgrade sql (#12141) 2022-09-26 15:21:52 +08:00
Tq
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts (#12126)
* fix htrace dependency conflicts

* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:08 +08:00
zhuxt2015
e6832220c3
[Improvement][SQL Task]use default sql segment separator (#10869) 2022-09-26 11:08:57 +08:00
labbomb
64f4cb4f3e
refactor: dag-menu refactor (#11965)
* refactor: dag-menu refactor

* feat: add i18n support for the dag-menu

* feat: add e2e class

Co-authored-by: devosend <devosend@gmail.com>
Co-authored-by: Tq <tianqitobethefirst@gmail.com>
2022-09-26 11:06:39 +08:00
Yann Ann
289e1ecdc6
[Bug] [Common] update IPv4 Pattern (#11762)
* [Bug] [Common] IPv4 Pattern is wrong. #11758
2022-09-26 09:10:22 +08:00
Yann Ann
ada7cf71d5
[fix#12000]Cannot remove the WorkerGroup from the master service (#12050)
* [Bug] [Master] Cannot remove the WorkerGroup from the master service. #12000

* remove unnecessary locks

* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/ServerNodeManager.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-24 18:57:20 +08:00
Eric Gao
064224696f
[Doc][Improvement] Add instructions for process execution type and data complement (#12121)
* Add instructions for process execution type and data complement

* Fix and polish the docs
2022-09-24 09:59:02 +08:00
Jiajie Zhong
82ddd72e4a
[Feature][API] New restful API for workflow and schedule (#11912)
* [feat] New restful API for workflow and schedule

CURD for workflow and schedule, different with exists
API, this new restful api only operate single resource
in each request, and return the latest. For example,
previous workflow should also need to post tasks definition
and tasks relation definition, but this patch will allow
you to create workflow without task relate information

* use checkProjectAndAuthThrowException, and fix CI error

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ScheduleV2Controller.java

* change method name
from createProcessDefinitionV2 to createSingleProcessDefinition
from updateProcessDefinitionV2 to updateSingleProcessDefinition

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-23 22:25:19 +08:00
Jiajie Zhong
a86f4e2693
[fix][python] Task switch branch not show in webui (#12120)
when create task switch in python api, switch branch not show correctly,
due to we add some unnecessary attribute to switch task, this patch we
add `_task_ignore_attr` in class `Task` to ignore those unnecessary
attribute from `_task_default_attr`
2022-09-23 19:41:05 +08:00
Jiajie Zhong
2531c222de
[Fix] Also auto formatter workflow instance if location is null (#12080)
* [fix] Also auto formatter workflow instance if location is null

In #11535(096fae77) and #11681(4dca488c), we already formatter
workflow definition, but I find out we forget formatter workflow
instance with definition's location is null, this patch also
formatter workflow instance.

* remove console log
2022-09-23 18:20:39 +08:00
JieguangZhou
6eb1eb722a
fix python home conflict (#12112) 2022-09-23 09:40:18 +08:00
Amy0104
e11e129cda
[Improvement][UI] Add filterable for src_table and target_table in the data quality task. (#12099) 2022-09-22 20:43:34 +08:00
chenrj
b1b57c8fb7
[feat][python] new resource plugin GitHub and gitlab (#11831)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: JieguangZhou <jieguang_zhou@163.com>
Co-authored-by: Devosend <devosend@gmail.com>
Co-authored-by: juzimao <578961953@qq.com>
2022-09-22 14:43:00 +08:00
jackfanwan
6e03a7e557
[BUG]Delete Setting ID (#12096)
* Delete Setting ID

* modify id

Co-authored-by: jackfanwan <fanwanlong@kezaihui.com>
2022-09-22 12:12:43 +08:00
Lyle Shaw
cc492c3e13
[Feature] Add CURD to the project/tenant/user section of the python-DS (#11162)
- Add CURD in project
- Add CURD in tenant
- Add CURD in user
- Add test in user

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-22 11:03:27 +08:00
Wenjun Ruan
fba5a8eaa0
Fix insert command error due to the id is not null (#12092) 2022-09-22 09:52:32 +08:00
冯剑
9b3b4e22b4
[Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported (#12059)
* [Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported

Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
2022-09-22 09:39:02 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
JieguangZhou
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use (#12071)
optimize code
2022-09-21 10:45:05 +08:00
Eric Gao
b52da64001
[Improvement][UT] Upgrade jUnit to 5.+ (#10976) (#11332)
* [Improvement][UT] Upgrade jUnit to 5.+ (#10976)

* Refactor AlertServerTest with jUnit-5 as an example
2022-09-21 10:15:21 +08:00
jackfanwan
7d80ea34ac
[Improvement] Add Set cluster name (#12074) 2022-09-21 10:05:15 +08:00
limaiwang
ed1d1e8856
add(dependent node's add unprivileged project list): dependent node's add unprivileged project list (#12073) 2022-09-21 09:28:22 +08:00
Wenjun Ruan
cb4ef54a0e
Script cannot contains ''' in params (#12067) 2022-09-21 08:52:51 +08:00
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center (#11708)
* Add support for Alibaba Cloud OSS as storage of resource center

* Add dependency licenses

* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
Wenjun Ruan
26dad5e2a8
Fix update TaskDefinition error (#12060) 2022-09-20 22:23:16 +08:00
jackfanwan
0bf639b851
[improvement]Add Set cluster name (#12058)
* Add Set cluster name

* add unit test

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-09-20 21:26:05 +08:00
insist777
955e8ebe0f
[Improve] Fix support test tasks UT (#12062) 2022-09-20 15:46:29 +08:00
金鑫
649015bfff
[Bug][ApiServer] When modifying the task definition, inserting the task definition log table will report the bug of primary key conflict #12037
Co-authored-by: 金鑫 <jinyanhui@huansi.net>
2022-09-20 14:42:21 +08:00
Eric Gao
e1b55db72a
[Improvement][Test] Give an example on replacing Powermock with Mockito (#11588)
* Give an example on replacing powermock with mockito

* Remove redundant comments

* Refactoring related UTs for better readability.
2022-09-20 09:48:02 +08:00
rickchengx
b3bbf69b2e
[Improvement-11291][sql] Improvement the log output while executing multiple sql statements (#11292) 2022-09-20 09:30:27 +08:00
songjianet
8cddb10773
[Feat][UI] Update package. (#12046)
* [Feat][UI] Update package.

* [Feat][UI] Update package.
2022-09-19 20:58:41 +08:00
Kerwin
89d55bc9b7
Fix link errors in release documentation. (#12049) 2022-09-19 20:01:23 +08:00
juzimao
c8884e4f2f
and task instance date range search (#11963) 2022-09-19 17:50:24 +08:00
JieguangZhou
ad683c3c42
[feat][python] Support MLflow task in python api (#11962) 2022-09-18 16:28:18 +08:00
limfff
c24ad9adb6
[Fix] Fixed an error importing project Maven (#12026) 2022-09-18 10:11:20 +08:00
JieguangZhou
bbc228e93f
[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
* add dvc task in pyds

* add BaseDVC class
2022-09-17 16:55:51 +08:00
JieguangZhou
e50a32d9cc
[feat][python] Support OpenMLDB task in python api (#11944)
Using the whole word zookeeper instead of short cut zk
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-17 15:24:17 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task (#11670)
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
Eric Gao
a44817fc46
[Doc][Style] Fix doc format once for all (#12006) 2022-09-17 11:33:32 +08:00
sgw
1a7c6eabf3
[Improvement-11773][api] Optimize the log printing of the api module according… (#11782)
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Resolve conflict.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- fix ProcessDefinitionServiceImpl.java dependency

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Accept proposed change in ResourcePermissionCheckServiceImpl.java

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Update the UT case in TenantServiceTest.java, UsersServiceTest.java, TaskGroupServiceTest.java.
- Update log printing in TaskGroupServiceImpl.java.
- Fix NPE of log parameter in ExecutorServiceImpl.java when call python api.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix NPE of log in WorkerGroupServiceImpl.java.
- Fix log parameter missing bug in ResourcesServiceImpl.java.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix bugs and handle some vulnerability codes.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix bugs and handle some vulnerability codes.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix e2e User test.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix codeQL check & SonarCloud Code Analysis.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Resolve import conflict.

* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix codeQL check & SonarCloud Code Analysis.
2022-09-17 11:28:04 +08:00
JieguangZhou
e05f14ba23
[feat][python] Add sagemaker yaml example (#11925) 2022-09-17 10:34:06 +08:00
JieguangZhou
5202e5cfc6
[feat][python] Support Pytorch task in python api (#11975)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-16 20:24:38 +08:00
Wenjun Ruan
864a90820d
Fix insertOrUpdate plugin may failed due to concurrent operation (#11471) 2022-09-16 15:39:04 +08:00
Wenjun Ruan
d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
jackfanwan
71b4087421
Example Change the regular expression of the data source password (#11459) 2022-09-16 13:34:22 +08:00
kezhenxu94
5811b84fcc
Add validations of possible malicious keys (#11966) 2022-09-16 13:32:59 +08:00
Wenjun Ruan
4ad34483a5
Fix alert_send_status may throw duplicate key exception and add limit for query alert (#11953) 2022-09-16 13:26:33 +08:00
rickchengx
5b6bebf7d5
[fix][doc] Update the registry related configuration in values.yaml (#11444) 2022-09-16 10:38:27 +08:00
ZhaoGuodong
86291bb5e6
[Feature-10273][Datasource] Add dataSource for AWS Athena (#11157) 2022-09-16 09:59:20 +08:00