Commit Graph

76 Commits

Author SHA1 Message Date
Paul Zhang
8562f6a878
[Feature][Log]Add timezone information in log output (#9913) 2022-05-06 17:31:44 +08:00
WangJPLeo
a1b6b033ad
[Fix-9906] After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance. (#9907)
* After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance.

* clear logic

* Resource overloading
2022-05-06 15:01:21 +08:00
WangJPLeo
fb0f96ed94
[Fix-9868] A task flow definition isolates the runs of different execution strategies by version numbers. (#9869)
* The thread cache task flow definition should get the latest version.

* Coverage on New Code

* Coverage on New Code

* Coverage on New Code

* use an existing method.

* Increase unit test coverage.

* Task flow definitions enforce policy isolation.
2022-05-03 22:52:42 +08:00
WangJPLeo
31cd1b5e61
Serial wait for subsequent execution (#9847) 2022-04-29 15:44:05 +08:00
WangJPLeo
3cea039239
Task queue status update. (#9832) 2022-04-28 16:41:00 +08:00
WangJPLeo
5c0be8a3d7
A task instance that normally queries the serial wait state. (#9777)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-26 15:47:01 +08:00
WangJPLeo
897d7cb555
Add the host address of the execution server to the sub task task instance. (#9758)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 18:17:48 +08:00
Jiajie Zhong
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server (#9726)
* [common] Make dolphinscheduler_env.sh work

* Change dist tarball `dolphinscheduler_env.sh` location
  from `bin/` to `conf/`, which users could finish their
  change configuration operation in one single directory.
  and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
  when we start our sever instead of adding both
  `$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
WangJPLeo
7bcec7115a
[Fix-9717] The failure policy of the task flow takes effect (#9718)
* Failure policy takes effect.

* Coverage on New Code

* correct description logic

* Compatible with all scenarios

* clearer logic

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 15:29:18 +08:00
caishunfeng
5657cb9aec
[Bug-9719][Master] fix failover fail because task plugins has not been loaded (#9720) 2022-04-24 20:34:21 +08:00
gaojun2048
ebc4253d50
[fix][Service] BusinessTime should format with schedule timezone (#9714)
* BusinessTime should format with schedule timezone

* fix test error

* fix test error

* fix test error
2022-04-24 19:21:21 +08:00
caishunfeng
88d2803fe1
fix task dispatch error overload resource pool of task group (#9667) 2022-04-22 18:39:40 +08:00
caishunfeng
63638601b0
fix process pause and rerun (#9568) 2022-04-19 10:23:56 +08:00
sparklezzz
508ed9769a
[Fix][Master Server] handle warn+failed timeout strategy in workflow execute thread of master server (#8077) (#9485)
Co-authored-by: xudong.zhang <xudong.zhang@nio.com>
2022-04-18 20:34:22 +08:00
Paul Zhang
3815a86a3b
[Improvement][Master] Fix typo for MasterTaskExecThreadTest (#9513) 2022-04-15 11:32:22 +08:00
caishunfeng
66d148872d
[Bug-9501][Worker] fix kill task error before running (#9509) 2022-04-14 21:34:02 +08:00
czeming
706cdb6a8e
[Fix-9174] [Alert] Fix deduplication of alarm information (#9371)
* feat(issue #9174):

Fix-9174
2022-04-14 10:30:25 +08:00
worry
69923546a1
[DS-9263][Improvement][master]optimize failover (#9281)
- add FailoverService.java
- move failover method  from MasterRegistryClient to FailoverService
- move failover code from FailoverExecuteThread to FailoverService

This closes #9263
2022-04-12 11:53:18 +08:00
caishunfeng
b285ccf930
[Future-9396]Support output parameters transfer from parent workflow to child work flow (#9410)
* [Future-9396]Support output parameters transfer from parent workflow to child work flow

* fix note
2022-04-11 20:03:16 +08:00
worry
dce3c132ca
[DS-9387][refactor]Remove the lock in the start method of the MasterRegistryClient class (#9389) 2022-04-08 21:06:28 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
caishunfeng
40e44f6bc4
fix repeated submit task (#9323)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-04-02 14:06:35 +08:00
guoshupei
3e5781b6e1
[Fix-9065] [master] when task submit failed, remove from queue and add state event to thread (#9186)
* [Fix-9065] [master] when task submit failed, remove from queue and add state event to thread

This closes #9065

* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java

yes, your idea is better

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

* [Bug-9065] [master] update logger info

Co-authored-by: guoshupei <guoshupei@lixiang.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-04-01 22:23:35 +08:00
caishunfeng
1073fcae44
[Bug-9295][Master] fix repeated submit task (#9304)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-31 21:10:09 +08:00
caishunfeng
52ba2c6475
[Improvement-9288][Master] add task event thread pool (#9293)
* add task event thread

* license heander

* ci test

* delete unuse file

* fix CI test

Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-31 14:13:33 +08:00
caishunfeng
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719)
* rebase dev

* change task state to dispatch if dispatch success

* update task host when dispatch

* add dispatch task event

* test

* check tenant after enable auto create

* handle dispatch state

Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 17:35:05 +08:00
worry
b4b5241797
[DS-9266][feat][server]Implement show dolphinscheduler banner at launch (#9273)
- add banner.txt
- modify application.yaml

This closes #9266
2022-03-30 16:45:24 +08:00
wangxj3
fd76436d87
[Feature-#8318] [master-server] Sub process transfer parameters to father process (#8522)
* load param from subprocess

* unit test

* modify code style

Co-authored-by: wangxj <wangxj31>
2022-03-30 14:55:59 +08:00
caishunfeng
df519f8044
remove common task log (#9274)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 11:42:47 +08:00
guoshupei
e2c1cc0579
[Fix-9222] [master] Support for deploying multiple masters on one node (#9240)
This closes #9222

Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-29 22:46:02 +08:00
JinYong Li
85e56b5e2a
fix 9243 (#9244) 2022-03-28 20:45:01 +08:00
worry
d3251c9bcc
[Improvement-9227][master]implement use the slot to scan the database (#9228)
when the master assigns tasks by slot,implement use the slot to scan the database.

This closes #9227
2022-03-28 09:52:12 +08:00
caishunfeng
263791a63e
[Bug-9137][Server] fix timezone init (#9219)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
guoshupei
258285e6bb
[Fix-8980] [master] fixed repeat processing command (#9220)
This closes #8980

Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-27 23:33:30 +08:00
Kerwin
d7d756e7b0
Fix DataX and Sqoop Task execute NullPointerException (#9192) 2022-03-26 20:15:41 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
caishunfeng
852597c9bf
[Bug-9070][Server]fix taskExecutionContext json convert error (#9072)
* [Bug-9070][Server]fix taskExecutionContext json convert error

* test

* add license header

Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-22 14:02:07 +08:00
guoshupei
3114c91748
[Bug-8810] [master] add try catch to onSuccess method in the WorkflowExecuteThreadPool (#8896)
This closes #8810

Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-15 16:34:19 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
Kerwin
c4fc59c9bb
[Fix][Master] Fix master-server execution logic thread unsafe problem (#8707)
* Fix thread safety issue in master service

* optimize code.
2022-03-14 16:22:48 +08:00
Martin Huang
b0fc6e7a69
[Feature-5752][server]Add blocking task (#6272)
* fix python task execution error

* modify code style

* delete AbstractTask#setCommand

* migrate blocking task alert feature and common attribute

* blocking task migrate(complete base features)

* migrate blocking task test

* resolve TaskNode.java conflict

* fix package problem in BlockingParameters

* resolve duplicate code

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* optimise code based on pr#6283 in ds master repo

* add log

* modify log

* resolve conflicts

* change hardcode to variable

* Add blocking task UI

* refactor old code to compatable with the newest branch

* add override annotation

* remove old code

* remove never used packages

* add missing packages declaration

* Refine the i18n description

* add UT

* resolve some code smell problems

* add fetch-depth

* enable unit test in sonar analysis

* disable unit test

* remove unused packages

* resolve duplicate codes

* add logs when task status changed and rename some variables

* add BLOCK & READY_BLOCK status and corresponding processing code

* revoke UI

* revert unit-test

Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2022-03-07 10:50:40 +08:00
Yao WANG
67a6813e9c
remove no useful code (#8552)
Co-authored-by: ywang46 <ywang46@paypal.com>
2022-03-03 19:09:35 +08:00
xiangzihao
74afdcf8f3
[Fix-8506][Master] process instance stop result wrong (#8529)
* fix bug_8506

* recover success state
2022-03-01 15:20:02 +08:00
yimaixinchen
a72742b9de
remove unuse import jsonFormat (#8548) 2022-02-25 22:05:05 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477)
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
kezhenxu94
fa19006a18
Add a property to enable/disable metrics module (#8484) 2022-02-24 10:21:47 +08:00
xiangzihao
234f399488
remove worker group lowercase (#8448) 2022-02-19 21:38:58 +08:00
xiangzihao
8200a3f15a
add try catch (#8433) 2022-02-18 21:46:52 +08:00
xiangzihao
864ceb164f
[Fix-Typo] Fix typo in dev branch. (#8395)
* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* test

* fix dev_bug_8337

* fix dev_bug_8337

* fix dev_bug_8337

* fix bug_8337

* fix typo
2022-02-16 17:38:43 +08:00
xiangzihao
08f642c756
[Fix-8337][Master] Process instance can not be kill when task is failure and can be retry (#8387)
* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* fix bug_8337

* test

* fix dev_bug_8337

* fix dev_bug_8337

* fix dev_bug_8337

* fix bug_8337
2022-02-15 19:04:19 +08:00