Wenjun Ruan
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
youzipi
042ec74a24
[refactor] cache compiled regex pattern ( #13208 )
...
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
2022-12-18 11:24:11 +08:00
Wenjun Ruan
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
Aaron Wang
560d232cbd
[Bug][Worker] Fix duplicated key bug when have out parameters of same name ( #12930 )
2022-12-12 11:07:23 +08:00
Wenjun Ruan
32a8241ef2
Don't source env file in task, this will override tenant's environment ( #13028 )
2022-11-28 21:09:50 +08:00
Kerwin
d480c271ec
[Bug-12994] [Worker] Fix kill process does not take effect ( #12995 )
2022-11-27 14:27:48 +08:00
Wenjun Ruan
92a9b7e243
Use System#lineSeparator to replace /n ( #12889 )
2022-11-15 13:57:45 +08:00
JieguangZhou
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario ( #12843 )
2022-11-11 16:08:38 +08:00
simsicon
43d69ae157
Upgrade clickhouse jdbc driver ( #12639 )
2022-11-11 13:38:58 +08:00
Kerwin
c4fb555883
Modify shell env to bash ( #12742 )
2022-11-07 11:21:11 +08:00
Wenjun Ruan
eaef68856b
Change command file permission to 755 ( #12678 )
...
(cherry picked from commit 95484d930762712f4ab68ece27ca3b65a2073ab2)
2022-11-03 22:51:37 +08:00
JieguangZhou
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2022-11-03 16:33:39 +08:00
amao
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init ( #11759 )
...
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2022-10-31 22:35:11 +08:00
rickchengx
3c31ddfd9f
[Improvement-12536][k8s] Support the command for the container in k8s task plugin ( #12538 )
2022-10-31 19:06:41 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2022-10-20 14:26:56 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
insist777
1e11034049
[Fix][Task] Fix dependent task can not predicate the status of the corresponding task correctly ( #12253 )
...
* [Fix][Task] dependent task can not predicate the status of the corresponding task correctly
2022-10-19 10:47:08 +08:00
simsicon
ba538067f2
[fix] Change the default unix shell executor from sh to bash ( #12180 )
...
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-10-18 12:57:37 +08:00
Kerwin
55004bebe0
[Improvement-12391][API] Workflow definitions that contain logical task nodes support the copy function ( #12392 )
...
* Workflow definitions that contain logical task nodes support the copy function.
2022-10-17 10:28:58 +08:00
Yann Ann
7aa8a77fb2
[Migrate][Test] Migrate other UT cases from jUnit 4 to 5, change @Ignore to @Disable
2022-10-16 12:49:11 +08:00
rickchengx
7b44612f28
[Improvement-12372][k8s] Update the deprecated k8s api ( #12373 )
2022-10-14 16:18:35 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
Eric Gao
875682d267
[Improvement][Test] Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example ( #12299 )
...
* Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example
2022-10-11 11:54:38 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
冯剑
779a0b9c75
[Improvement-#11608][task-plugin] New time variables are added to facilitate business development ( #11667 )
...
Add new time expressions for users' convenience
Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
Co-authored-by: David <dailidong66@gmail.com>
2022-09-26 16:01:32 +08:00
zhuxt2015
e6832220c3
[Improvement][SQL Task]use default sql segment separator ( #10869 )
2022-09-26 11:08:57 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task ( #11860 )
2022-09-21 20:27:28 +08:00
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center ( #11708 )
...
* Add support for Alibaba Cloud OSS as storage of resource center
* Add dependency licenses
* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
itzhang89
511149f644
[fix-10938]: use dot to replace source and make the default env work in shell ( #11625 )
...
* [fix-10938]: use the dot to replace source and make the default env work in the sh
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-09-15 15:05:09 +08:00
Wenjun Ruan
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2022-09-14 16:18:08 +08:00
caishunfeng
0deafba12c
fix procedure task param pass ( #11919 )
2022-09-14 13:44:01 +08:00
juzimao
5d9331f4ee
[Feature-10683][Task Plugin] Add Java Task Plugin. ( #10689 )
2022-09-09 10:37:24 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2022-09-08 09:05:05 +08:00
caishunfeng
f95cdd350a
reduce task dependencies ( #11731 )
2022-09-01 17:04:07 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2022-08-31 15:25:01 +08:00
Wenjun Ruan
1b120e3a59
Refactor worker execute task process ( #11540 )
...
* Refactor worker execute task process
2022-08-26 13:33:51 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2022-08-24 19:51:47 +08:00
Wenjun Ruan
a41c6824fc
Add dependent task instance log ( #11541 )
...
* Add dependent task instance log
* Optimize log
* Fix dependent task initialize failed will throw exception
2022-08-23 09:55:38 +08:00
kezhenxu94
13e4720aa2
Upgrade spring boot and swagger ( #11574 )
2022-08-22 10:27:40 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2022-08-16 20:25:17 +08:00
Kerwin
b79bad19e5
[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… ( #11099 )
2022-08-11 09:28:40 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2022-08-11 09:25:53 +08:00
Stalary
6345a13cdc
[Fix-10392][SqlTask]: Fix list param error when use sql task ( #11285 )
...
* FIX: fix list param in.
* FIX: code bug
* FIX: fix one value
* FIX: header
* FIX: ci
* FIX: fix ci
* MOD: for review
* MOD: for review
2022-08-11 09:24:44 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
microeastcowboy
9a53c6ac34
fix work oom when task logs's size is very large ( #11224 )
2022-08-01 15:28:06 +08:00
