kezhenxu94
13e4720aa2
Upgrade spring boot and swagger ( #11574 )
2022-08-22 10:27:40 +08:00
fuchanghai
d8d5d392a7
[feature-11408] add hive option ( #11493 )
2022-08-20 08:25:53 +08:00
JieguangZhou
a974ba74ab
[Feature] New task plugin Pytorch ( #11498 )
...
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2022-08-18 10:20:37 +08:00
FangYin Cheng
11d74fd7e8
[test-11432][HttpTask] Add unit tests to http task ( #11453 )
...
* [test][HttpTask] Add unit tests to http task
* [test][HttpTask] fix http task test code format problem
* [test][HttpTask] fix http task test code replace http code to constants
2022-08-17 22:02:52 +08:00
lugela
e5d06c40a5
[Feature-10182][task-datax] ( #11387 )
...
In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase
2022-08-17 10:01:20 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2022-08-16 20:25:17 +08:00
Eric Gao
896fef6c98
[Feature][Style] Enable spotless to manage imports ( #11458 )
2022-08-13 10:05:03 +08:00
Kerwin
b79bad19e5
[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… ( #11099 )
2022-08-11 09:28:40 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2022-08-11 09:25:53 +08:00
Stalary
6345a13cdc
[Fix-10392][SqlTask]: Fix list param error when use sql task ( #11285 )
...
* FIX: fix list param in.
* FIX: code bug
* FIX: fix one value
* FIX: header
* FIX: ci
* FIX: fix ci
* MOD: for review
* MOD: for review
2022-08-11 09:24:44 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Assert
b3919d30e7
[Improvement-11394][Task Plugin] Remove Passed Parameter --queue in Seatunnel Task Plugin ( #11395 )
2022-08-10 20:17:16 +08:00
Assert
03582f2e70
[Fix-11376][task plugin]Seatunnel Task cannot save successfully when … ( #11377 )
2022-08-10 12:54:09 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
Eric Gao
6a02870926
[Feature][style] Add spotless maven plugin for automatic style fix. ( #11272 )
...
* [Feature][style] Add spotless maven plugin for automatic style fix (#10963 )
* Fix spotless ratchet configuration
* Remove license-check and decrease line length threshold value
* Update related docs
* Remove checkstyle and add pre-commit hook
* Test updated pre-commit hook
* Replace checkstyle with spotless in CI
* Remove reviewdog
2022-08-06 19:29:23 +08:00
microeastcowboy
9a53c6ac34
fix work oom when task logs's size is very large ( #11224 )
2022-08-01 15:28:06 +08:00
xuhhui
bfff3a7c5d
fix error ( #11206 )
2022-07-30 18:20:20 +08:00
xuhhui
513f336015
[Feature] Add new task type chunjun ( #10937 )
2022-07-28 19:25:29 +08:00
caishunfeng
818648df7d
[Bug-11071] fix local params ( #11184 )
...
* fix local params
* add CI case
2022-07-28 15:07:04 +08:00
Wenjun Ruan
052ceaacd0
Add is success in loop task status ( #11169 )
2022-07-28 12:12:01 +08:00
Assert
cdfe115247
[Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter ( #10960 )
...
* [Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter
* checkstyle
2022-07-26 17:41:13 +08:00
Wenjun Ruan
c7789bf049
Add base loop task execotor and http template parser ( #11137 )
...
* Add base loop task execotor and http template parser
* Add JsonPathUtils
2022-07-26 14:51:50 +08:00
JieguangZhou
af7b359333
[Feature][Task Plugin] Add SageMaker Pipeline task plugin for MLOps scenario ( #10826 ) ( #10935 )
...
* add sagemaker pipeline task plugin
[DOC] add Sagemaker task plugin document
change license and fix dependencies
* [fix] Optimize the code
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerParameters.java
Optimization parameter judgment
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [fix] nips
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-07-21 18:16:58 +08:00
Eric Gao
f689220290
[Feature][Task Plugin] Increase zeppelin task stability in production ( #11010 )
...
* [Feature][Task Plugin] Increase zeppelin task stability in production (#10584 )
* Add front-end and update docs for the production mode of zeppelin task
* Fix minor front-end bug of zeppelin task plugin
* Refactor ZeppelinParameters with lombok
* Fix formatting
* Replace @Data with @Getter, @Setter, @ToString to avoid decrease in test coverage
2022-07-20 14:16:47 +08:00
xuhhui
35f32cd16e
[Improvement][Task]Improve seatunnel task command log ( #11027 )
2022-07-20 11:06:10 +08:00
Eric Gao
2f0d907bcb
[Improvement][Task Plugin]Refactor JupyterParameters.java with lombok ( #11045 )
...
* [Improvement][Task Plugin]Refactor JupyterParameters.java with lombok (#11040 )
2022-07-20 11:02:47 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
JieguangZhou
11a354927d
[fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms ( #10986 ) ( #10987 )
...
* Optimize the download method of preset project
* fix checkstyle
2022-07-15 15:38:13 +08:00
zhuxt2015
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters ( #10950 )
...
* datax task plugin support custom parameters
2022-07-15 09:27:23 +08:00
Eric Gao
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks ( #10925 )
...
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814 )
2022-07-15 09:26:28 +08:00
chuxing
427c58546c
[fix][plugin]: fix spark sql use custom parma. ( #10821 )
...
* [fix][plugin]: fix spark sql use custom parma.
* [fix][plugin] roll back part of the code
Co-authored-by: huaangcg <huangcg@getui.com>
2022-07-13 11:44:13 +08:00
JieguangZhou
02b2de2a1e
[fix] mlflow project success status ( #10897 )
...
close : #10902
2022-07-12 20:58:53 +08:00
Tq
499e5b1307
[bug][task-plugins]fix switch class name and add param check to the switch task plugin ( #10894 )
...
* fix switch class name and add param check to the switch task plugin
2022-07-12 16:41:15 +08:00
ZhaoGuodong
8eaf5a2309
[Feature-10219][EMR] EMR supports use <add-Steps> to add steps to an existing cluster ( #10657 )
...
* Add the ProgramType parameter to distinguish task types
* EmrAddStepsTask supports Add-Steps
* UI supports Add-Steps
* EmrTask modify the name of the class to EmrJobFlowTask
* add ERM Task abstract base class AbstractEmrTask
* add testcase for EmrAddStepsTask
* Modifying help Documents
2022-07-10 10:15:25 +08:00
Eric Gao
71f0168510
[Feature] Enable users to create python env from requirements.txt ( #10658 )
2022-07-06 15:56:39 +08:00
Kerwin
bd1be6cf36
[Task] Added Apache SeaTunnel 2.x task node ( #10480 )
2022-07-05 19:21:30 +08:00
JieguangZhou
7d79a2165e
[Optimization]Optimize some details of MLFlow task plugin #10740 ( #10739 )
...
* Optimize some details of MLFlow task plugin
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-mlflow/src/main/java/org/apache/dolphinscheduler/plugin/task/mlflow/MlflowParameters.java
* fix some nips
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-07-05 11:05:20 +08:00
aiwenmo
5b0347e8da
[Task] Add Dinky task to better support the development and execution of FlinkSQL ( #10640 )
2022-07-05 10:07:24 +08:00
Paul Zhang
4a3c3e7797
[feature][task-flink] Support Flink application mode ( #9577 )
2022-07-01 12:20:38 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2022-06-30 22:45:25 +08:00
zhuxt2015
7e7df1d8ff
[Bug] [Task] SQL task no exception stack trace in task log ( #10656 )
2022-06-29 09:57:05 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2022-06-28 16:08:35 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
JieguangZhou
36e20cdfc8
[Feature][Task Plugin] Add DVC task plugin for MLops scenario ( #10372 ) ( #10407 )
2022-06-21 13:13:46 +08:00
Eric Gao
64cee03fed
[Feature] Enable users to switch and install conda env in jupyter task ( #10337 )
2022-06-20 14:47:44 +08:00
Eric Gao
4be3b877c3
[feature][task] Enable zeppelin schedule a whole zeppelin note ( #10434 )
2022-06-18 13:34:24 +08:00
guodong
2c227ab384
[Feature-10363][server] S3 Resource center supports bucket customization ( #10433 )
...
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify code review comments
2022-06-18 10:20:01 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
JieguangZhou
3258438f6e
[Feature] [MLOps] support mlflow deploy with docker compose ( #10217 )
...
* [Feature] [MLOps] support mlflow deploy with docker compose
fix doc
Update docs/docs/en/guide/task/mlflow.md
fix doc
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
revert cancel modification
fix ENV name and docker compose command
* fix doc image link
* fix testModelsDeployDockerCompose
* add docker compose container health check and fix mlflow bug
* update docker compose healthcheck timeout
2022-06-10 10:20:50 +08:00
xiangzihao
bbba20d39a
change to common lang3 and fix checkstyle ( #10380 )
2022-06-09 15:56:42 +08:00
