test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-11-30 03:08:01 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
7,167 Commits 30 Branches 57 Tags 276 MiB
147e0ae8d7
Commit Graph

163 Commits

Author SHA1 Message Date
kezhenxu94
147e0ae8d7
Remove duplicate Spring version management (#11899) 2022-09-11 20:50:07 +08:00
W.XING
1164cb776d
Add chunjun in dolphinscheduler-task-all/pom.xml (#11375) 2022-09-11 19:14:05 +08:00
juzimao
5d9331f4ee
[Feature-10683][Task Plugin] Add Java Task Plugin. (#10689) 2022-09-09 10:37:24 +08:00
fuchanghai
e101a5cb2c
[fix-#11815] fix ck column names contain special characters (#11818) 
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
 2022-09-08 15:09:17 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model (#11767) 2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
Kengo Seki
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810) 
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
 2022-09-07 13:48:09 +08:00
fuchanghai
f8b9aad239
[feature-#11689] change replaceAll to replace in method of tryExecuteSqlResolveColumnNames (#11696) 2022-09-05 09:54:18 +08:00
caishunfeng
f95cdd350a
reduce task dependencies (#11731) 2022-09-01 17:04:07 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701) 
* Fix kill job may cause worker oom
 2022-08-31 15:25:01 +08:00
Eric Gao
4a37345436
[Feature][Task Plugin] Support hive cli task plugin (#11651) 
* Add hive cli task plugin

* Add docs for hive-cli task plugin

* Fix hive-cli task related front-end nits and checkstyle

* Fix exception handling logic
 2022-08-30 10:45:51 +08:00
Wenjun Ruan
1b120e3a59
Refactor worker execute task process (#11540) 
* Refactor worker execute task process
 2022-08-26 13:33:51 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory (#11614) 
* Add spi priority factory

* Add doc

* Add override log

* Use lombok

* Add comment
 2022-08-24 19:51:47 +08:00
Wenjun Ruan
a41c6824fc
Add dependent task instance log (#11541) 
* Add dependent task instance log

* Optimize log

* Fix dependent task initialize failed will throw exception
 2022-08-23 09:55:38 +08:00
kezhenxu94
13e4720aa2
Upgrade spring boot and swagger (#11574) 2022-08-22 10:27:40 +08:00
fuchanghai
d8d5d392a7
[feature-11408] add hive option (#11493) 2022-08-20 08:25:53 +08:00
JieguangZhou
a974ba74ab
[Feature] New task plugin Pytorch (#11498) 
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
 2022-08-18 10:20:37 +08:00
FangYin Cheng
11d74fd7e8
[test-11432][HttpTask] Add unit tests to http task (#11453) 
* [test][HttpTask] Add unit tests to http task

* [test][HttpTask] fix http task test code format problem

* [test][HttpTask] fix http task test code replace http code to constants
 2022-08-17 22:02:52 +08:00
lugela
e5d06c40a5
[Feature-10182][task-datax] (#11387) 
In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase
 2022-08-17 10:01:20 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map (#11402) 2022-08-16 20:25:17 +08:00
Eric Gao
896fef6c98
[Feature][Style] Enable spotless to manage imports (#11458) 2022-08-13 10:05:03 +08:00
Kerwin
b79bad19e5
[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… (#11099) 2022-08-11 09:28:40 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status (#11390) 2022-08-11 09:25:53 +08:00
Stalary
6345a13cdc
[Fix-10392][SqlTask]: Fix list param error when use sql task (#11285) 
* FIX: fix list param in.

* FIX: code bug

* FIX: fix one value

* FIX: header

* FIX: ci

* FIX: fix ci

* MOD: for review

* MOD: for review
 2022-08-11 09:24:44 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task (#11350) 
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
 2022-08-10 21:44:43 +08:00
Assert
b3919d30e7
[Improvement-11394][Task Plugin] Remove Passed Parameter --queue in Seatunnel Task Plugin (#11395) 2022-08-10 20:17:16 +08:00
Assert
03582f2e70
[Fix-11376][task plugin]Seatunnel Task cannot save successfully when … (#11377) 2022-08-10 12:54:09 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
Eric Gao
6a02870926
[Feature][style] Add spotless maven plugin for automatic style fix. (#11272) 
* [Feature][style] Add spotless maven plugin for automatic style fix (#10963)

* Fix spotless ratchet configuration

* Remove license-check and decrease line length threshold value

* Update related docs

* Remove checkstyle and add pre-commit hook

* Test updated pre-commit hook

* Replace checkstyle with spotless in CI

* Remove reviewdog
 2022-08-06 19:29:23 +08:00
microeastcowboy
9a53c6ac34
fix work oom when task logs's size is very large (#11224) 2022-08-01 15:28:06 +08:00
xuhhui
bfff3a7c5d
fix error (#11206) 2022-07-30 18:20:20 +08:00
xuhhui
513f336015
[Feature] Add new task type chunjun (#10937) 2022-07-28 19:25:29 +08:00
caishunfeng
818648df7d
[Bug-11071] fix local params (#11184) 
* fix local params

* add CI case
 2022-07-28 15:07:04 +08:00
Wenjun Ruan
052ceaacd0
Add is success in loop task status (#11169) 2022-07-28 12:12:01 +08:00
Assert
cdfe115247
[Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter (#10960) 
* [Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter

* checkstyle
 2022-07-26 17:41:13 +08:00
Wenjun Ruan
c7789bf049
Add base loop task execotor and http template parser (#11137) 
* Add base loop task execotor and http template parser

* Add JsonPathUtils
 2022-07-26 14:51:50 +08:00
JieguangZhou
af7b359333
[Feature][Task Plugin] Add SageMaker Pipeline task plugin for MLOps scenario (#10826) (#10935) 
* add sagemaker pipeline task plugin

[DOC] add Sagemaker task plugin document

change license and fix dependencies

* [fix] Optimize the code

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerParameters.java

Optimization parameter judgment

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

* [fix] nips

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
 2022-07-21 18:16:58 +08:00
Eric Gao
f689220290
[Feature][Task Plugin] Increase zeppelin task stability in production (#11010) 
* [Feature][Task Plugin] Increase zeppelin task stability in production (#10584)

* Add front-end and update docs for the production mode of zeppelin task

* Fix minor front-end bug of zeppelin task plugin

* Refactor ZeppelinParameters with lombok

* Fix formatting

* Replace @Data with @Getter, @Setter, @ToString to avoid decrease in test coverage
 2022-07-20 14:16:47 +08:00
xuhhui
35f32cd16e
[Improvement][Task]Improve seatunnel task command log (#11027) 2022-07-20 11:06:10 +08:00
Eric Gao
2f0d907bcb
[Improvement][Task Plugin]Refactor JupyterParameters.java with lombok (#11045) 
* [Improvement][Task Plugin]Refactor JupyterParameters.java with lombok (#11040)
 2022-07-20 11:02:47 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
JieguangZhou
11a354927d
[fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms (#10986) (#10987) 
* Optimize the download method of preset project

* fix checkstyle
 2022-07-15 15:38:13 +08:00
zhuxt2015
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters (#10950) 
* datax task plugin support custom parameters
 2022-07-15 09:27:23 +08:00
Eric Gao
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#10925) 
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814)
 2022-07-15 09:26:28 +08:00
chuxing
427c58546c
[fix][plugin]: fix spark sql use custom parma. (#10821) 
* [fix][plugin]: fix spark sql use custom parma.

* [fix][plugin] roll back part of the code

Co-authored-by: huaangcg <huangcg@getui.com>
 2022-07-13 11:44:13 +08:00
JieguangZhou
02b2de2a1e
[fix] mlflow project success status (#10897) 
close: #10902
 2022-07-12 20:58:53 +08:00
Tq
499e5b1307
[bug][task-plugins]fix switch class name and add param check to the switch task plugin (#10894) 
* fix switch class name and add param check to the switch task plugin
 2022-07-12 16:41:15 +08:00
ZhaoGuodong
8eaf5a2309
[Feature-10219][EMR] EMR supports use <add-Steps> to add steps to an existing cluster (#10657) 
* Add the ProgramType parameter to distinguish task types
* EmrAddStepsTask supports Add-Steps
* UI supports Add-Steps
* EmrTask modify the name of the class to EmrJobFlowTask
* add ERM Task abstract base class AbstractEmrTask
* add testcase for EmrAddStepsTask
* Modifying help Documents
 2022-07-10 10:15:25 +08:00
Eric Gao
71f0168510
[Feature] Enable users to create python env from requirements.txt (#10658) 2022-07-06 15:56:39 +08:00
Kerwin
bd1be6cf36
[Task] Added Apache SeaTunnel 2.x task node (#10480) 2022-07-05 19:21:30 +08:00