Rick Cheng
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2023-02-13 16:38:26 +08:00
fuchanghai
37939add0c
[fix-13492]add single quotes ( #13534 )
...
* [fix-13492]add single quotes
* [improve-#13201] fix test error
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-02-09 19:02:27 +08:00
JieguangZhou
16b193454b
[Improvement][Task Plugin] Support Task Plugin output response as a parameter ( #13529 )
...
* support http output response as param
* add UT
* fix sqoop UT
2023-02-08 23:03:21 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
shenyun
8fa90c2d8f
feature flinksql support standalone deploy mode ( #13495 )
2023-02-05 20:37:56 +08:00
Wenjun Ruan
5a9cb7ee64
Fix task log might delay until task finished due to batch flush ( #13502 )
2023-02-02 21:48:03 +08:00
Tq
78ba71bdbd
[Feature][task]add azure datafactory task type ( #13417 )
...
* init data factory
* add task-type-config.yaml
* add data factory ui
* submit doc
* submit license
---------
Co-authored-by: devosend <devosend@gmail.com>
2023-01-30 11:15:20 +08:00
LucasClt
a1f5675012
[Bug][TaskPlugin] Task execution error occurs when the Zeppelin task zeppelinNoteId parameter value is an empty string ( #13162 )
...
* [Bug] [TaskPlugin-Zeppelin] Task execution error occurs when the Zeppelin task zeppelinNoteId parameter value is an empty string. #13161
2023-01-06 19:29:05 +08:00
caishunfeng
ba0a253f09
fix k8s task run fail ( #13348 )
2023-01-06 09:44:12 +08:00
Wenjun Ruan
6634ff98af
Add source env list in common.properties ( #13317 )
2023-01-04 20:53:52 +08:00
JieguangZhou
df491f78cc
fix sagemaker sync status ( #13264 )
2022-12-22 20:17:17 +08:00
Wenjun Ruan
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
Wenjun Ruan
b2336b0ce9
Remove datasource in procedure task ( #13198 )
2022-12-19 17:14:59 +08:00
youzipi
042ec74a24
[refactor] cache compiled regex pattern ( #13208 )
...
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
2022-12-18 11:24:11 +08:00
HuangWei
43e6ac32dd
[Fix-13187] refactor OpenmldbParameters ( #13190 )
2022-12-14 20:13:45 +08:00
Wenjun Ruan
838a0df783
Remove logic task plugin module ( #13171 )
2022-12-13 16:30:44 +08:00
Wenjun Ruan
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
Wenjun Ruan
ab96a3d0c0
Fix datasource jdbc url param doesn't work ( #13098 )
2022-12-12 11:46:40 +08:00
Aaron Wang
560d232cbd
[Bug][Worker] Fix duplicated key bug when have out parameters of same name ( #12930 )
2022-12-12 11:07:23 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2022-12-10 23:50:19 +08:00
Wenjun Ruan
169168ef34
Add plugin-all module ( #13079 )
2022-12-02 23:19:08 +08:00
Eric Gao
c2ae76c0bd
Expand all static imports and add CI check to block wildcard static imports ( #13067 )
2022-12-01 14:15:28 +08:00
Wenjun Ruan
32a8241ef2
Don't source env file in task, this will override tenant's environment ( #13028 )
2022-11-28 21:09:50 +08:00
Kerwin
ae670e6e1c
[Bug-13010] [Task] The Flink SQL task page selects the pre-job deployment mode, but the task executed by the worker is the Flink local mode ( #13011 )
2022-11-28 10:09:49 +08:00
Kerwin
d480c271ec
[Bug-12994] [Worker] Fix kill process does not take effect ( #12995 )
2022-11-27 14:27:48 +08:00
Wenjun Ruan
a2ff140f43
Fix sql task will send alert if we don't choose the send email ( #12984 )
2022-11-24 21:26:52 +08:00
rickchengx
38b876733c
[Feature-10498] Mask the password in the log of sqoop task ( #11589 )
2022-11-24 14:54:54 +08:00
JieguangZhou
6e224d08a2
[fix] Python task can not pass the parameters to downstream task. ( #12961 )
2022-11-22 16:11:45 +08:00
JieguangZhou
7c90bf01bc
[Improvement][Task Plugin] Improvement Kubeflow task plugin ( #12928 )
...
add example
check phase in status:conditions
2022-11-17 15:59:44 +08:00
Assert
2dbc79693e
[feature][task plugin] Add new task plugin for apache linkis ( #12693 )
2022-11-17 10:14:36 +08:00
Wenjun Ruan
92a9b7e243
Use System#lineSeparator to replace /n ( #12889 )
2022-11-15 13:57:45 +08:00
JieguangZhou
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario ( #12843 )
2022-11-11 16:08:38 +08:00
simsicon
43d69ae157
Upgrade clickhouse jdbc driver ( #12639 )
2022-11-11 13:38:58 +08:00
Kerwin
c4fb555883
Modify shell env to bash ( #12742 )
2022-11-07 11:21:11 +08:00
Wenjun Ruan
2c3cc31a4c
Fix flink sql cannot run due to missing main jar ( #12705 )
2022-11-04 23:02:00 +08:00
Wenjun Ruan
eaef68856b
Change command file permission to 755 ( #12678 )
...
(cherry picked from commit 95484d930762712f4ab68ece27ca3b65a2073ab2)
2022-11-03 22:51:37 +08:00
JieguangZhou
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2022-11-03 16:33:39 +08:00
amao
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init ( #11759 )
...
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2022-10-31 22:35:11 +08:00
rickchengx
3c31ddfd9f
[Improvement-12536][k8s] Support the command for the container in k8s task plugin ( #12538 )
2022-10-31 19:06:41 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
HanayoZz
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2022-10-20 14:26:56 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
insist777
1e11034049
[Fix][Task] Fix dependent task can not predicate the status of the corresponding task correctly ( #12253 )
...
* [Fix][Task] dependent task can not predicate the status of the corresponding task correctly
2022-10-19 10:47:08 +08:00
simsicon
ba538067f2
[fix] Change the default unix shell executor from sh to bash ( #12180 )
...
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-10-18 12:57:37 +08:00
Yann Ann
3bef85f546
fix datax NPE issue ( #12388 )
2022-10-18 09:03:16 +08:00
Kerwin
55004bebe0
[Improvement-12391][API] Workflow definitions that contain logical task nodes support the copy function ( #12392 )
...
* Workflow definitions that contain logical task nodes support the copy function.
2022-10-17 10:28:58 +08:00
fuchanghai
b7bd8d7803
[Improvement] add presto in datax ( #12371 )
2022-10-16 17:19:29 +08:00
Yann Ann
7aa8a77fb2
[Migrate][Test] Migrate other UT cases from jUnit 4 to 5, change @Ignore to @Disable
2022-10-16 12:49:11 +08:00
rickchengx
7b44612f28
[Improvement-12372][k8s] Update the deprecated k8s api ( #12373 )
2022-10-14 16:18:35 +08:00
Eric Gao
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
rickchengx
05099bd8f7
[Improvement-12296][Test] Close static mock ( #12300 )
2022-10-11 16:13:05 +08:00
Eric Gao
875682d267
[Improvement][Test] Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example ( #12299 )
...
* Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example
2022-10-11 11:54:38 +08:00
Eric Gao
a865be09b6
Remove powermock from DatasyncTaskTest ( #12270 )
2022-10-09 12:38:03 +08:00
Tq
c7435a51ac
[Feature][task-plugins] aws datasync task plugins ( #11865 )
...
* submit datasync task
* add aws software package license
2022-10-09 10:47:01 +08:00
FlechazoW
0687697869
[Improvement][Task Plugin] Modify the comment of 'deployMode'. ( #12163 )
2022-10-07 21:21:13 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
JieguangZhou
308f632fcc
[Feature][TASK-PLUGIN] AWS Database Migration Service (DMS) ( #11868 )
...
* add aws dms
2022-09-30 10:16:16 +08:00
Stalary
aa5526ef5d
[Improvement][Test] Remove Powermock in dolphinscheduler-task-plugin ( #12153 )
...
* Remove the usage of powermock in task-plugin module
2022-09-28 14:13:23 +08:00
冯剑
779a0b9c75
[Improvement-#11608][task-plugin] New time variables are added to facilitate business development ( #11667 )
...
Add new time expressions for users' convenience
Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
Co-authored-by: David <dailidong66@gmail.com>
2022-09-26 16:01:32 +08:00
zhuxt2015
e6832220c3
[Improvement][SQL Task]use default sql segment separator ( #10869 )
2022-09-26 11:08:57 +08:00
JieguangZhou
6eb1eb722a
fix python home conflict ( #12112 )
2022-09-23 09:40:18 +08:00
冯剑
9b3b4e22b4
[Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported ( #12059 )
...
* [Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported
Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
2022-09-22 09:39:02 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task ( #11860 )
2022-09-21 20:27:28 +08:00
JieguangZhou
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use ( #12071 )
...
optimize code
2022-09-21 10:45:05 +08:00
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center ( #11708 )
...
* Add support for Alibaba Cloud OSS as storage of resource center
* Add dependency licenses
* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
Eric Gao
e1b55db72a
[Improvement][Test] Give an example on replacing Powermock with Mockito ( #11588 )
...
* Give an example on replacing powermock with mockito
* Remove redundant comments
* Refactoring related UTs for better readability.
2022-09-20 09:48:02 +08:00
rickchengx
b3bbf69b2e
[Improvement-11291][sql] Improvement the log output while executing multiple sql statements ( #11292 )
2022-09-20 09:30:27 +08:00
JieguangZhou
bbc228e93f
[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 ( #11941 )
...
* add dvc task in pyds
* add BaseDVC class
2022-09-17 16:55:51 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
itzhang89
511149f644
[fix-10938]: use dot to replace source and make the default env work in shell ( #11625 )
...
* [fix-10938]: use the dot to replace source and make the default env work in the sh
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-09-15 15:05:09 +08:00
Wenjun Ruan
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2022-09-14 16:18:08 +08:00
caishunfeng
0deafba12c
fix procedure task param pass ( #11919 )
2022-09-14 13:44:01 +08:00
limaiwang
25b78a8003
[Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql ( #11721 )
...
select different versions of spark-sql to execute different versions of spark-sql
2022-09-12 10:35:27 +08:00
kezhenxu94
147e0ae8d7
Remove duplicate Spring version management ( #11899 )
2022-09-11 20:50:07 +08:00
W.XING
1164cb776d
Add chunjun in dolphinscheduler-task-all/pom.xml ( #11375 )
2022-09-11 19:14:05 +08:00
juzimao
5d9331f4ee
[Feature-10683][Task Plugin] Add Java Task Plugin. ( #10689 )
2022-09-09 10:37:24 +08:00
fuchanghai
e101a5cb2c
[fix-#11815] fix ck column names contain special characters ( #11818 )
...
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-09-08 15:09:17 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2022-09-08 09:05:05 +08:00
Kengo Seki
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
fuchanghai
f8b9aad239
[feature-#11689] change replaceAll to replace in method of tryExecuteSqlResolveColumnNames ( #11696 )
2022-09-05 09:54:18 +08:00
caishunfeng
f95cdd350a
reduce task dependencies ( #11731 )
2022-09-01 17:04:07 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2022-08-31 15:25:01 +08:00
Eric Gao
4a37345436
[Feature][Task Plugin] Support hive cli task plugin ( #11651 )
...
* Add hive cli task plugin
* Add docs for hive-cli task plugin
* Fix hive-cli task related front-end nits and checkstyle
* Fix exception handling logic
2022-08-30 10:45:51 +08:00
Wenjun Ruan
1b120e3a59
Refactor worker execute task process ( #11540 )
...
* Refactor worker execute task process
2022-08-26 13:33:51 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2022-08-24 19:51:47 +08:00
Wenjun Ruan
a41c6824fc
Add dependent task instance log ( #11541 )
...
* Add dependent task instance log
* Optimize log
* Fix dependent task initialize failed will throw exception
2022-08-23 09:55:38 +08:00
kezhenxu94
13e4720aa2
Upgrade spring boot and swagger ( #11574 )
2022-08-22 10:27:40 +08:00
fuchanghai
d8d5d392a7
[feature-11408] add hive option ( #11493 )
2022-08-20 08:25:53 +08:00
JieguangZhou
a974ba74ab
[Feature] New task plugin Pytorch ( #11498 )
...
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2022-08-18 10:20:37 +08:00
FangYin Cheng
11d74fd7e8
[test-11432][HttpTask] Add unit tests to http task ( #11453 )
...
* [test][HttpTask] Add unit tests to http task
* [test][HttpTask] fix http task test code format problem
* [test][HttpTask] fix http task test code replace http code to constants
2022-08-17 22:02:52 +08:00
lugela
e5d06c40a5
[Feature-10182][task-datax] ( #11387 )
...
In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase
2022-08-17 10:01:20 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2022-08-16 20:25:17 +08:00
Eric Gao
896fef6c98
[Feature][Style] Enable spotless to manage imports ( #11458 )
2022-08-13 10:05:03 +08:00
Kerwin
b79bad19e5
[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… ( #11099 )
2022-08-11 09:28:40 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2022-08-11 09:25:53 +08:00
Stalary
6345a13cdc
[Fix-10392][SqlTask]: Fix list param error when use sql task ( #11285 )
...
* FIX: fix list param in.
* FIX: code bug
* FIX: fix one value
* FIX: header
* FIX: ci
* FIX: fix ci
* MOD: for review
* MOD: for review
2022-08-11 09:24:44 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Assert
b3919d30e7
[Improvement-11394][Task Plugin] Remove Passed Parameter --queue in Seatunnel Task Plugin ( #11395 )
2022-08-10 20:17:16 +08:00
Assert
03582f2e70
[Fix-11376][task plugin]Seatunnel Task cannot save successfully when … ( #11377 )
2022-08-10 12:54:09 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
Eric Gao
6a02870926
[Feature][style] Add spotless maven plugin for automatic style fix. ( #11272 )
...
* [Feature][style] Add spotless maven plugin for automatic style fix (#10963 )
* Fix spotless ratchet configuration
* Remove license-check and decrease line length threshold value
* Update related docs
* Remove checkstyle and add pre-commit hook
* Test updated pre-commit hook
* Replace checkstyle with spotless in CI
* Remove reviewdog
2022-08-06 19:29:23 +08:00
microeastcowboy
9a53c6ac34
fix work oom when task logs's size is very large ( #11224 )
2022-08-01 15:28:06 +08:00
xuhhui
bfff3a7c5d
fix error ( #11206 )
2022-07-30 18:20:20 +08:00
xuhhui
513f336015
[Feature] Add new task type chunjun ( #10937 )
2022-07-28 19:25:29 +08:00
caishunfeng
818648df7d
[Bug-11071] fix local params ( #11184 )
...
* fix local params
* add CI case
2022-07-28 15:07:04 +08:00
Wenjun Ruan
052ceaacd0
Add is success in loop task status ( #11169 )
2022-07-28 12:12:01 +08:00
Assert
cdfe115247
[Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter ( #10960 )
...
* [Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter
* checkstyle
2022-07-26 17:41:13 +08:00
Wenjun Ruan
c7789bf049
Add base loop task execotor and http template parser ( #11137 )
...
* Add base loop task execotor and http template parser
* Add JsonPathUtils
2022-07-26 14:51:50 +08:00
JieguangZhou
af7b359333
[Feature][Task Plugin] Add SageMaker Pipeline task plugin for MLOps scenario ( #10826 ) ( #10935 )
...
* add sagemaker pipeline task plugin
[DOC] add Sagemaker task plugin document
change license and fix dependencies
* [fix] Optimize the code
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerParameters.java
Optimization parameter judgment
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [fix] nips
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-07-21 18:16:58 +08:00
Eric Gao
f689220290
[Feature][Task Plugin] Increase zeppelin task stability in production ( #11010 )
...
* [Feature][Task Plugin] Increase zeppelin task stability in production (#10584 )
* Add front-end and update docs for the production mode of zeppelin task
* Fix minor front-end bug of zeppelin task plugin
* Refactor ZeppelinParameters with lombok
* Fix formatting
* Replace @Data with @Getter, @Setter, @ToString to avoid decrease in test coverage
2022-07-20 14:16:47 +08:00
xuhhui
35f32cd16e
[Improvement][Task]Improve seatunnel task command log ( #11027 )
2022-07-20 11:06:10 +08:00
Eric Gao
2f0d907bcb
[Improvement][Task Plugin]Refactor JupyterParameters.java with lombok ( #11045 )
...
* [Improvement][Task Plugin]Refactor JupyterParameters.java with lombok (#11040 )
2022-07-20 11:02:47 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
JieguangZhou
11a354927d
[fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms ( #10986 ) ( #10987 )
...
* Optimize the download method of preset project
* fix checkstyle
2022-07-15 15:38:13 +08:00
zhuxt2015
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters ( #10950 )
...
* datax task plugin support custom parameters
2022-07-15 09:27:23 +08:00
Eric Gao
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks ( #10925 )
...
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814 )
2022-07-15 09:26:28 +08:00
chuxing
427c58546c
[fix][plugin]: fix spark sql use custom parma. ( #10821 )
...
* [fix][plugin]: fix spark sql use custom parma.
* [fix][plugin] roll back part of the code
Co-authored-by: huaangcg <huangcg@getui.com>
2022-07-13 11:44:13 +08:00
JieguangZhou
02b2de2a1e
[fix] mlflow project success status ( #10897 )
...
close : #10902
2022-07-12 20:58:53 +08:00
Tq
499e5b1307
[bug][task-plugins]fix switch class name and add param check to the switch task plugin ( #10894 )
...
* fix switch class name and add param check to the switch task plugin
2022-07-12 16:41:15 +08:00
ZhaoGuodong
8eaf5a2309
[Feature-10219][EMR] EMR supports use <add-Steps> to add steps to an existing cluster ( #10657 )
...
* Add the ProgramType parameter to distinguish task types
* EmrAddStepsTask supports Add-Steps
* UI supports Add-Steps
* EmrTask modify the name of the class to EmrJobFlowTask
* add ERM Task abstract base class AbstractEmrTask
* add testcase for EmrAddStepsTask
* Modifying help Documents
2022-07-10 10:15:25 +08:00
Eric Gao
71f0168510
[Feature] Enable users to create python env from requirements.txt ( #10658 )
2022-07-06 15:56:39 +08:00
Kerwin
bd1be6cf36
[Task] Added Apache SeaTunnel 2.x task node ( #10480 )
2022-07-05 19:21:30 +08:00
JieguangZhou
7d79a2165e
[Optimization]Optimize some details of MLFlow task plugin #10740 ( #10739 )
...
* Optimize some details of MLFlow task plugin
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-mlflow/src/main/java/org/apache/dolphinscheduler/plugin/task/mlflow/MlflowParameters.java
* fix some nips
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-07-05 11:05:20 +08:00
aiwenmo
5b0347e8da
[Task] Add Dinky task to better support the development and execution of FlinkSQL ( #10640 )
2022-07-05 10:07:24 +08:00
Paul Zhang
4a3c3e7797
[feature][task-flink] Support Flink application mode ( #9577 )
2022-07-01 12:20:38 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2022-06-30 22:45:25 +08:00
zhuxt2015
7e7df1d8ff
[Bug] [Task] SQL task no exception stack trace in task log ( #10656 )
2022-06-29 09:57:05 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2022-06-28 16:08:35 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
JieguangZhou
36e20cdfc8
[Feature][Task Plugin] Add DVC task plugin for MLops scenario ( #10372 ) ( #10407 )
2022-06-21 13:13:46 +08:00
Eric Gao
64cee03fed
[Feature] Enable users to switch and install conda env in jupyter task ( #10337 )
2022-06-20 14:47:44 +08:00
Eric Gao
4be3b877c3
[feature][task] Enable zeppelin schedule a whole zeppelin note ( #10434 )
2022-06-18 13:34:24 +08:00
guodong
2c227ab384
[Feature-10363][server] S3 Resource center supports bucket customization ( #10433 )
...
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify code review comments
2022-06-18 10:20:01 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
JieguangZhou
3258438f6e
[Feature] [MLOps] support mlflow deploy with docker compose ( #10217 )
...
* [Feature] [MLOps] support mlflow deploy with docker compose
fix doc
Update docs/docs/en/guide/task/mlflow.md
fix doc
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
revert cancel modification
fix ENV name and docker compose command
* fix doc image link
* fix testModelsDeployDockerCompose
* add docker compose container health check and fix mlflow bug
* update docker compose healthcheck timeout
2022-06-10 10:20:50 +08:00
xiangzihao
bbba20d39a
change to common lang3 and fix checkstyle ( #10380 )
2022-06-09 15:56:42 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
2022-06-08 19:42:05 +08:00
He Zhao
da3c25dc67
[improve] Using create or replace function in sql task ( #10170 )
2022-06-06 10:33:24 +08:00
Wenjun Ruan
0bf4043d5f
Move datasource params to datasouce plugin module ( #10353 )
...
* Move datasource params to datasouce plugin module
2022-06-03 12:15:53 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2022-05-31 15:22:41 +08:00
Eric Gao
a8a5da367a
[Feature][Task Plugin] Add support for dynamic form for zeppelin task plugin ( #9977 ) ( #10269 )
2022-05-31 11:19:41 +08:00
HuangWei
c5961aed25
[feat][task] Add OpenMLDB task plugin ( #10198 )
2022-05-31 11:00:21 +08:00
JieguangZhou
54ab2651bd
[Feature][MLops] Add MLflow Projects (custom project) component ( #10175 )
...
* Add MLflow Projects (custom project) component
* fix something
* update document
* update document
* Supplementary test cases
2022-05-24 09:42:46 +08:00
JieguangZhou
e11373b963
[Feature][MLops] Support MLflow Models to deploy model service (MLflow models serve and Docker) ( #10150 )
...
* Add backend of MLflow Moldes component
* Add front end of MLflow Moldes component
* [DOC] update mlflow document
* revert unnecessary
* fix doc image url
* Update docs/docs/en/guide/task/mlflow.md
Discard the lr abbreviation
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-20 19:41:22 +08:00
JieguangZhou
9ab3bebc90
[Feature][Task Plugin] Add mlflow task plugin for MLops scenario. ( #10036 )
...
* [Feature][Task Plugin] Add mlflow task plugin (#9725 )
* [Feature][Task Plugin] Add mlflow task plugin UI (#9725 )
* [Feature][Task Plugin] fix license header (#9725 )
* [Feature][Task Plugin] fix license header (#9725 )
* [Feature][Task Plugin] revert unnecessary
* [Feature][Task Plugin] add auto ml to mlflow task plugin
* [DOC] add mlflow document
* [DOC] fix mlflow docs imgs
* [DOC] fix dead link localhost:5000
* [DOC] fix dead link localhost:5000
* [DOC] remove dead link localhost:5000
* Update docs/docs/en/guide/task/mlflow.md
* Update docs/docs/zh/guide/task/mlflow.md
* [DOC] format ui code
* [DOC] remove dead link localhost:5000
* [Feature][Task Plugin] revert unnecessary
* fix some nits
* Move the personal repository to the public repository.
* Run the command directly instead of saving the file
* fix paramsMap initialize
* revert unnecessary
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-19 09:40:31 +08:00
He Zhao
5bb1eb04fc
[Feature][plugin] Add k8s task in task plugin ( #9425 )
...
* [Feature][plugin] Add k8s task in task plugin
* [Feature][plugin] fix dos and code problems
* [Feature][plugin] refactor some code based on sonar
* [Feature][UI] front-end for k8s task plugin
* [Feature][plugin] delete some front files
* [Feature][plugin] update document
* Update docs/docs/zh/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* [Feature][UI] front-end change from review comments
* [Feature][UI] replace get namespace list api
* [Feature][plugin] change file name
* Add kubernetes to zh task list
Co-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-18 17:23:42 +08:00
