test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-02 04:08:31 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
8,404 Commits 27 Branches 59 Tags 276 MiB
ec6a3e0727
Commit Graph

531 Commits

Author SHA1 Message Date
旺阳
61637cc0a3
[Bug] [Cron] Parse corn expression error (#13841) 
* fix-cron

* update catch exception
 2023-04-09 12:50:42 +08:00
Wenjun Ruan
41a8ba9aab
Support alert server HA (#13865) 
* Support alert server cluster

* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
 2023-04-05 17:31:50 +08:00
Rick Cheng
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target (#13654) 2023-03-30 12:07:58 +08:00
caishunfeng
9d9ae9ad54
[improvement] support self-dependent (#13818) 2023-03-29 18:18:03 +08:00
Wenjun Ruan
68660ec96b
Refactor remote command (#13809) 
* Refactor remote command

* Rename Command to Message
 2023-03-29 17:54:57 +08:00
Wenjun Ruan
d1b6e6f02c
Take over task instance in master failover (#13798) 2023-03-28 16:02:57 +08:00
Aaron Wang
8895921d87
[Improvement-13651] Remove useless resources_task (#13721) 2023-03-23 22:52:23 +08:00
Wenjun Ruan
c9066e8de9
Use MDC to filter task instance log (#13673) 
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
 2023-03-06 17:44:23 +08:00
Aaron Wang
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes (#13550) 2023-02-21 23:26:21 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger (#13509) 2023-02-07 20:32:53 +08:00
Wenjun Ruan
6300a9b8de
Fix serial wait will go into block when there is no task need to submit (#13486) 2023-02-02 11:10:37 +08:00
Wenjun Ruan
12dd60fa46
Fix task wake up failed will block the event (#13466) 2023-02-01 15:31:04 +08:00
qianli2022
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId (#13184) 
* add sql

* add mapper

* add dao

* add excutor


Co-authored-by: qianl4 <qianl4@cicso.com>
 2023-01-18 17:58:32 +08:00
Wenjun Ruan
01f4fb26f0
Change remove task instance log to async request (#13399) 2023-01-16 15:49:23 +08:00
Wenjun Ruan
98dfd64444
Clear the task group data when delete a project/workflowInstance (#13383) 2023-01-12 21:51:51 +08:00
Wenjun Ruan
8a479927f3
Add projectCode in t_ds_process_instance and t_ds_task_instance to remove join (#13284) 2023-01-03 09:52:28 +08:00
Wenjun Ruan
52134277a3
Fix task group cannot release when kill task (#13314) 2023-01-03 09:52:03 +08:00
Aaron Wang
8503ee0eed
[Improvement][Worker] Startup parameter should have the highest priority (#13274) 2022-12-29 11:05:00 +08:00
Wenjun Ruan
fd3afd84ad
Remove alert when delete workflow instance (#13281) 2022-12-28 11:40:12 +08:00
Wenjun Ruan
de70421a1e
Remove the log path check (#13280) 2022-12-26 22:39:08 +08:00
Wenjun Ruan
14ec4a2398
Remove dao module in worker (#13242) 2022-12-22 12:25:29 +08:00
JieguangZhou
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks (#13194) 
* Supports task instance cache operation

* add task plugin cache

* use SHA-256 to generate key

* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>

* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>

* Optimizing database Scripts

* Optimize clear cache operation

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
 2022-12-18 18:17:09 +08:00
JieguangZhou
e4b9b67255
Allow execute task in workflow instance (#13103) 2022-12-13 16:43:44 +08:00
Wenjun Ruan
838a0df783
Remove logic task plugin module (#13171) 2022-12-13 16:30:44 +08:00
Wenjun Ruan
f5801a0e1c
Fix task group might be rob by multiple task (#13115) 2022-12-11 22:58:21 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 (#13002) 2022-12-10 23:50:19 +08:00
Wenjun Ruan
8a152aebc7
Fix task log file might not be clear (#13102) 2022-12-06 11:23:20 +08:00
Wenjun Ruan
fed1e85fb8
Fix the invalidate task instance never be deleted (#13091) 2022-12-04 18:38:54 +08:00
rickchengx
38b876733c
[Feature-10498] Mask the password in the log of sqoop task (#11589) 2022-11-24 14:54:54 +08:00
Kerwin
c916c60853
fix NPE while retry task (#12903) 2022-11-16 10:31:52 +08:00
Wenjun Ruan
d99ba29b66
Fix master cluster may loop command unbalanced (#12891) 
(cherry picked from commit 3b2b86661be76b7c1404a910c865d78b7936313d)
 2022-11-16 10:20:22 +08:00
XCL
00b18ed422
[Improve]Remove the code logger.isDebugEnabled() #5918 #10542 (#12738) 
Co-authored-by: chunlai.xu <chunlai.xu@zhaopin.com.cn>
 2022-11-07 09:58:19 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds (#12197) 
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.

Co-authored-by: gabrywu <gabrywu@apache.com>
 2022-10-31 16:52:53 +08:00
Yann Ann
ed209bdf82
[Refactor] Migrate all workergroup-related interface functions from ProcessServiceImpl (#12493) 
* Migrate all workergroup-related interface functions from ProcessServiceImpl
 2022-10-26 19:18:32 +08:00
HanayoZz
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor (#12076) 
* resource center refactor - S3 services connection

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
 2022-10-26 13:53:44 +08:00
Yann Ann
04aa125ba2
[Refactor]migrate some task dao functions from ProcessServiceImpl to dao modules. (#12505) 
* migrate some task DAO to dao modules.

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/repository/impl/ProcessInstanceMapDaoImpl.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
 2022-10-26 11:51:58 +08:00
rickchengx
233bbdb09e
[Fix-12451][k8s] Read the kubeconfig from cluster conf (#12452) 2022-10-25 11:25:30 +08:00
Wenjun Ruan
a0d37fc7ae
Set tenantDir permission (#12486) 2022-10-23 21:54:01 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils (#12502) 
* improve JSONUtil

* merge spi.utils into common.utils

* split constants && remove some copy utils method

* remove StringUtils
 2022-10-23 21:23:54 +08:00
Yann Ann
b936b882bb
[Refactor] Migrate all command-related interface functions from ProcessServiceImpl (#12474) 
* migrate all command-related interface functions to CommonService
 2022-10-22 12:39:25 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE (#12449) 2022-10-20 14:26:56 +08:00
Yann Ann
0eef2e3e10
[Improvement][Style] Fix code style of ProcessScheduleTask and ProcessServiceImpl (#12447) 2022-10-20 10:31:16 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412) 
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
 2022-10-19 11:08:04 +08:00
rickchengx
97edc7d652
[Fix-12356][k8s] fix the null exception when submitting k8s task plugin (#12358) 2022-10-19 09:49:56 +08:00
Kerwin
a8e23008ac
Fix timing scheduling trigger master service report to get command parameter null pointer exception (#12419) 2022-10-19 09:43:36 +08:00
houshitao
934d6d21eb
[Bug] [Service] Use MapUtils to judge globalParamsMap is empty. (#12384) 2022-10-17 14:52:45 +08:00
Eric Gao
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom (#12311) 
* move mockito dependencies from sub-modules to root pom

* Add check in CI to block the usage of powermock
 2022-10-14 12:06:51 +08:00
Yann Ann
b357781df2
[Migrate][Test] Migrate junit4 -> junit5 test cases in common, service, and spi modules (#12363) 2022-10-14 11:06:29 +08:00
fuchanghai
22103f975e
[improvement] Migrate all UT cases from jUnit4 to jUnit5 in dao module (#12328) 
* [improvement] Migrate all UT cases from jUnit4 to jUnit5 in dao module

Co-authored-by: Jackie <‘2875334588@qq.com’>
 2022-10-13 15:26:39 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes (#12321) 2022-10-13 10:23:27 +08:00
Jiajie Zhong
0e1c8d8153
[Feature][API] New restful API task and task relation (#12116) 2022-10-13 09:41:03 +08:00
rickchengx
05099bd8f7
[Improvement-12296][Test] Close static mock (#12300) 2022-10-11 16:13:05 +08:00
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project (#12244) 
* Fully remove the usage of powermock from the whole project

* Upgrade org.reflections to 0.10.12
 2022-10-10 10:43:38 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones (#12239) 
* fix DS support cross time zone, use timestamp replace date
 2022-10-01 12:50:42 +08:00
caishunfeng
40090d459e
[Improvement] remove log-server and server module (#12206) 
* remove log-server module

* remove dolphinscheduler-server module
 2022-09-30 16:41:41 +08:00
Wenjun Ruan
aacabc6ec7
Remove workerGroup in registry (#12217) 2022-09-30 13:58:53 +08:00
fuchanghai
e27c79974d
[fix-#11753] send alert error alert data id (#11774) 
* [fix-#11753] send alert error alert data id

Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
 2022-09-29 15:34:40 +08:00
Eric Gao
d1e409e9a2
[Improvement][Test] Remove powermock in service and server modules (#12164) 
* Remove powermock and refactor some related code in dolphinscheduler-service and dolphinscheduler-server modules
* Remove redundant comments
* Add null check
 2022-09-29 08:16:23 +08:00
xuhhui
6466cc7c41
[Improvement][Master] Construct processInstance may NPE when master handling command (#12056) 
* [Improvement][Master] Construct processInstance may NPE when master handling command

* use an enpty map

Co-authored-by: xuhaihui <xuhaihui@cmss.chinamobile.com>
 2022-09-27 09:13:48 +08:00
Wenjun Ruan
fba5a8eaa0
Fix insert command error due to the id is not null (#12092) 2022-09-22 09:52:32 +08:00
Wenjun Ruan
26dad5e2a8
Fix update TaskDefinition error (#12060) 2022-09-20 22:23:16 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task (#11670) 
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
 2022-09-17 13:40:56 +08:00
Wenjun Ruan
d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
Wenjun Ruan
d3a77c68e6
Make LogServiceClient Singleton (#11777) 2022-09-14 16:18:08 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model (#11767) 2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
caishunfeng
f8d46a26c1
[Feature-11530] add state history for process instance (#11757) 
* [Feature] add state history for process instance (#97)

* add state history for process instance

* upsertProcessInstance

* remove unuse method

* fix UT

Co-authored-by: caishunfeng <534328519@qq.com>
 2022-09-03 10:14:05 +08:00
Wenjun Ruan
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize (#11702) 
* Refactor heart beat task, use json to serialize/deserialize
 2022-08-31 16:20:23 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701) 
* Fix kill job may cause worker oom
 2022-08-31 15:25:01 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory (#11614) 
* Add spi priority factory

* Add doc

* Add override log

* Use lombok

* Add comment
 2022-08-24 19:51:47 +08:00
Wenjun Ruan
4d13a5104b
Fix kill task failed will cause the taskGroup cannot release and add taskGroup log (#11469) 2022-08-23 10:05:03 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map (#11402) 2022-08-16 20:25:17 +08:00
JiPeng Wang
13d400ceb6
[Fix-11465] Serial first, the last executing instance will be stopped (#11466) 
* Serial first, the last executing instance will be stopped

* ut fix
 2022-08-13 22:30:56 +08:00
Wenjun Ruan
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost (#11368) 
* Add waiting strategy to support master/worker can recover from registry lost

* throw exception when zookeeper registry start failed due to interrupted
 2022-08-13 09:52:03 +08:00
longtb
be66035dee
[Improvement] Avoid using search in for and start using testSaveTaskDefine (#11383) 
* [Improvement] Avoid using search in for and start using testSaveTaskDefine()

* [Improvement] not import *

Co-authored-by: zhangshunmin <zhangshunmin@kezaihui.com>
 2022-08-12 09:37:24 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status (#11390) 2022-08-11 09:25:53 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task (#11350) 
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
 2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
Wenjun Ruan
acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop (#11254) 
* Fix TaskGroup cannot work and will cause master dead loop

* Remove acquireTaskGroupAgain in ProcessServiceImpl
 2022-08-04 14:43:45 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db (#11221) 2022-08-01 13:04:27 +08:00
JinYong Li
e5cca0e79b
[Fix-11007] [Master] fix forced_success bug (#11088) 
* fix forced_success bug

* add comments

* add transactional

* refactor code

Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
 2022-07-30 23:28:31 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
caishunfeng
65d2b8f14e
[Bug] fix task params pass (#11053) 2022-07-20 10:14:51 +08:00
Mr.An
d7df899909
[Fix-11003]Task group queue is not updated to final state (#11004) 
* fix after the task group is forced to start the task. Task group status has not changed
 2022-07-19 17:48:16 +08:00
Wenjun Ruan
083ab2b5c9
Remove dao in worker (#10994) 2022-07-15 20:07:18 +08:00
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979) 
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed

* Move heartbeat error threshold to applicaiton.yml
 2022-07-15 20:06:53 +08:00
WangJPLeo
2397423eb6
Solve transaction problems under multi-threading (#10917) 2022-07-13 11:23:47 +08:00
Wenjun Ruan
2a67866718
Catch exception when check state in StateWheelExecuteThread (#10908) 
* Catch exception when check state
 2022-07-13 10:51:20 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866) 
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
 2022-07-11 09:57:00 +08:00
Wenjun Ruan
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect (#10839) 
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
 2022-07-09 11:54:59 +08:00
Wenjun Ruan
426567348e
Remove quartz in service (#10748) 
* Remove quartz in service
 2022-07-06 15:43:55 +08:00
Wenjun Ruan
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry (#10786) 
* Fix state event handle error will not retry

* Use state event handler to deal with the event
 2022-07-06 14:53:28 +08:00
WangJPLeo
b7fbf6c4ad
[API] Do not specify rollback for ensure in case of any exception or error. (#10793) 2022-07-06 14:22:46 +08:00
zhuxt2015
182b9c9b26
[Feature][Master] Failover workflow instance base on priority (#10754) 
* failover workflow instance base on priority
 2022-07-04 23:35:24 +08:00
Wenjun Ruan
35a10d092f
[Fix-10666] Workflow submit failed will still in memory and never retry (#10667) 
* Workflow submit failed will still in memory and never retry
 2022-07-04 22:08:15 +08:00
houshitao
f8650b7b7c
[Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun。 (#10727) 
* [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun.

* [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun.

* [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun.

Co-authored-by: houshitao <shitaohou@163.com>
 2022-07-04 22:05:23 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. (#10704) 
* Global parameter and local parameter calculation external expansion.

* k8s task ut fix.

* TimePlaceholderUtils import DateUtils fix

* follow the review comments to fix.

* follow the review comments to fix.

* e2e rerun
 2022-06-30 22:45:25 +08:00
Wenjun Ruan
247ca4ae8a
[Improvement-10617] Add comment in slot check (#10618) 2022-06-30 15:58:49 +08:00
wjf
75581b7543
[Improvement]Delete the getSession in the Registry interface (#10633) 2022-06-29 11:20:52 +08:00
Wenjun Ruan
35b25da863
Validate master/worker config (#10649) 2022-06-28 20:17:43 +08:00