Commit Graph

676 Commits

Author SHA1 Message Date
SongTao Zhuang
d7582c9864 fix: typo task definition. (#10958) 2022-07-18 17:36:54 +08:00
WangJPLeo
7056e1f9fe [Improvement] Resource authority module database table unique index increased. (#10807)
* Resource authority module database table unique index increased.

* add unique in sql file

* rerun

* ut fix

* e2e rerun

* dao un fix
2022-07-18 16:59:19 +08:00
devosend
7ddaa2f47d [maven-release-plugin] prepare for next development iteration 2022-06-28 16:41:31 +08:00
devosend
0a1b9bdd52 [maven-release-plugin] prepare release 3.0.0-beta-2 2022-06-28 16:41:30 +08:00
devosend
d68dcda2bb [chore] pre-release change pom.xml 2022-06-28 08:24:21 +08:00
Wenjun Ruan
742944c60d
[Fix] [Worker] Fix worker will hang if fails to start (#10501)
* [Fix] [Worker] Fix worker will hang if fails to start (#10342)

* Fix worker will hang if fails to start

* Add .run to ignore

Signed-off-by: ruanwenjun <wenjun@apache.org>

* Add import Epoll

* Add cpu_quota in h2 to fix UT

* Remove cpu_quota,memory_max in TaskDefiniitionMapper
2022-06-22 10:13:55 +08:00
陈家名
476f2395dc [fix][Python] Support same task name in project defferent process definition (#10428)
close: #10431
(cherry picked from commit b86dc53ad1)
2022-06-18 21:47:11 +08:00
Wenjun Ruan
4007382193 Fix database initialize late in standalone. (#10296)
(cherry picked from commit 3a9e67325e)
2022-06-16 18:18:53 +08:00
Jiajie Zhong
57ade38939 [maven-release-plugin] prepare release 3.0.0-beta-1 2022-05-26 20:26:32 +08:00
Jiajie Zhong
e663043c91 [fix] Task group queue query all list error (#10237)
Add support query all task queue by not given specific groupId

close: #10102
(cherry picked from commit d8db2b57d2)
2022-05-25 23:42:35 +08:00
longtb
bb30b88306 [Fix-10076] queryUserByToken use server timezone (#10177)
* [Fix-10076] queryUserByToken use server timezone

* [Fix-10076] queryUserByToken use server timezone

(cherry picked from commit bd34e66b8a)
2022-05-23 14:59:50 +08:00
BaoLiang
b016037a6f [BUG][TaskGroup] Task group does not take effect (#10093)
* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

(cherry picked from commit ee2b855ced)
2022-05-23 14:46:14 +08:00
hstdream
63ab98e67f [Fix-10082]Resource userName is null and udf function manager add a username (#10064)
* Resource userName is null

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

Co-authored-by: houshitao <shitaohou@163.com>
(cherry picked from commit 701b535ae6)
2022-05-23 14:43:31 +08:00
hstdream
cb91fc72ee Create process definition with task group and task group priority cannot be save into db (#10078)
Co-authored-by: houshitao <shitaohou@163.com>
(cherry picked from commit cd70ad584a)
2022-05-23 14:43:19 +08:00
Jiajie Zhong
a9fa6b33a4
[chore] Change release version to 3.0.0-beta-1 (#9957) 2022-05-12 21:49:21 +08:00
Jiajie Zhong
9ce89f112e [chore] Combine version 206 and 210 sql upgrade to 300 (#9959)
(cherry picked from commit 316b611826)
2022-05-11 12:02:22 +08:00
WangJPLeo
fb0f96ed94
[Fix-9868] A task flow definition isolates the runs of different execution strategies by version numbers. (#9869)
* The thread cache task flow definition should get the latest version.

* Coverage on New Code

* Coverage on New Code

* Coverage on New Code

* use an existing method.

* Increase unit test coverage.

* Task flow definitions enforce policy isolation.
2022-05-03 22:52:42 +08:00
Town
fcb32e4c3c
[fix] fix query access token for specified user failed (#9855) 2022-04-30 11:07:00 +08:00
WangJPLeo
ab2d535e54
The node ID under serial is greater than the current ID. (#9820)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-28 11:00:33 +08:00
caishunfeng
70dae6945f
revert delete dumplicate task_definition_log ddl (#8567)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-04-27 15:19:40 +08:00
leiwingqueen
c837580c68
[bug][db] Table relation_project_user have duplicate record (#9536)
* update sql ddl
* fix syntax err
2022-04-27 12:04:08 +08:00
WangJPLeo
f839b527bf
Task group queue supports fuzzy search. (#9784) 2022-04-26 18:03:55 +08:00
WangJPLeo
5c0be8a3d7
A task instance that normally queries the serial wait state. (#9777)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-26 15:47:01 +08:00
WangJPLeo
7bcec7115a
[Fix-9717] The failure policy of the task flow takes effect (#9718)
* Failure policy takes effect.

* Coverage on New Code

* correct description logic

* Compatible with all scenarios

* clearer logic

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 15:29:18 +08:00
yimaixinchen
fb11525e49
[chore] Correct the java doc in funtion DagHelperTest generateDag2 (#9602)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-04-23 10:31:48 +08:00
naziD
10f8c9d983
[Bug-9608] Serialize the task definition failed (#9622)
* BugFix: serialize the task definition failed

* Remove a comment

Co-authored-by: lipandong <pandong.lpd@alibaba-inc.com>
2022-04-21 11:38:41 +08:00
Eric Gao
a863c6f8f1
[Bug][DAO] Update db init script and soft_version (#9628) (#9637) 2022-04-21 11:01:19 +08:00
WangJPLeo
ae84900329
[Fix-9617]New task group project name drop-down data is displayed according to user type. (#9625)
* New task group project name drop-down data by user type

* remove unused import

* avoid import *

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-20 18:24:45 +08:00
qianli2022
165d7aa51f
[Feature][Task] K8s namespace auth manager (#9303)
* k8s auth

* remove log

* fix test

* use constants

* use constants K8S_LOCAL_TEST_CLUSTER

* simple auth get

* change test

* add namespace authorize in user page

* prettier code

* change test data

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
2022-04-20 18:23:23 +08:00
xiangzihao
a3bf10c88d
[Feature][API] Refactor get dependent node information api (#9591)
* feature_8922

* fix comment error

* remove unused import

* fix unused import
2022-04-20 14:53:40 +08:00
Tq
c5b7e5adff
[Bug] [API-9558]fix homepage task instance count method to use submit time to recount (#9559)
* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method JUNIT

* fix homepage task instance count method JUNIT

* fix homepage task instance count method JUNIT
2022-04-19 15:23:57 +08:00
WangJPLeo
b1d57dbce4
Check the status of the child process when the parent process is running (#9567)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-18 20:27:11 +08:00
Tq
1192fb8e12
fix Fault tolerance warning mapper add alert type to insert (#9533) 2022-04-17 10:17:00 +08:00
WangJPLeo
95ccc0c855
[Fix-9516] [Statistics] Statistics Manager shows data should belongs to current user (#9515) 2022-04-15 18:25:25 +08:00
czeming
706cdb6a8e
[Fix-9174] [Alert] Fix deduplication of alarm information (#9371)
* feat(issue #9174):

Fix-9174
2022-04-14 10:30:25 +08:00
Tq
fb0139e959
[Feature] [ALERT-9406]enrich alert info when insert new row into DB (#9445)
* enrich alert info when insert new row into DB

* fix comments
2022-04-12 14:31:38 +08:00
nobolity
037692517a
[Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)

* fix(server): fix the  udf serialize

* fix(master  task): update the udfTest to update the json string

* fix(test): update the udfFuncTest

* fix(common): syn the common.properties

* fix(udfTest): upgrade the udfTest

* fix(common): revent the common.properties
2022-04-11 10:49:46 +08:00
Tq
e2759a8f42
[Feature] [ALERT-9406]add new properties to alert class (#9408)
* add new properties to Alert.java and do minor changes to comments

* fix Integer to int

* fix Integer to int

* fix sql files

* fix not null properties to default null
2022-04-08 23:54:34 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
worry
2bab12f2c8
[Feature-9204][alert] Implement alert send status (#9208)
* [DS-9204][feat][alert,dao] Implement alert send status
- implement alert send status
- add alert send status entity、mapper
- modify alert dao
- modify alert sender
- add test
- add sql

This closes #9204

* [DS-9204][feat][alert,dao] Implement alert send status
- add license header

This closes #9204
2022-04-06 18:08:00 +08:00
JinYong Li
85e56b5e2a
fix 9243 (#9244) 2022-03-28 20:45:01 +08:00
worry
d3251c9bcc
[Improvement-9227][master]implement use the slot to scan the database (#9228)
when the master assigns tasks by slot,implement use the slot to scan the database.

This closes #9227
2022-03-28 09:52:12 +08:00
shuai hou
5bd8ba4268
[fix-8929][DAO]Description Failed to upload the file because the full name was too long (#9020)
* [fix-8929][DAO]Description Failed to upload the file because the full name was too long

* [DS-#8929][fix]Description Failed to upload the file because the full name was too long

* [DS-#8929][fix]Description Failed to upload the file because the full name was too long

* [DS-#8929][fix]Description Failed to upload the file because the full name was too long

Co-authored-by: houshuai <houshuai@jiguang.cn>
2022-03-25 13:40:20 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
hjli
242b5dd5b6
[Bug-8760]Resource file deletion conditions (#8765) 2022-03-15 17:26:41 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
wangyang
0d8079a816
[DS-7992][feat] Alert module judging strategy (#8636) 2022-03-14 14:25:27 +08:00
ouyangyewei
b0ffec5a4c
to issue/#8826 (#8827)
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
2022-03-13 08:39:05 +08:00
Martin Huang
b0fc6e7a69
[Feature-5752][server]Add blocking task (#6272)
* fix python task execution error

* modify code style

* delete AbstractTask#setCommand

* migrate blocking task alert feature and common attribute

* blocking task migrate(complete base features)

* migrate blocking task test

* resolve TaskNode.java conflict

* fix package problem in BlockingParameters

* resolve duplicate code

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* optimise code based on pr#6283 in ds master repo

* add log

* modify log

* resolve conflicts

* change hardcode to variable

* Add blocking task UI

* refactor old code to compatable with the newest branch

* add override annotation

* remove old code

* remove never used packages

* add missing packages declaration

* Refine the i18n description

* add UT

* resolve some code smell problems

* add fetch-depth

* enable unit test in sonar analysis

* disable unit test

* remove unused packages

* resolve duplicate codes

* add logs when task status changed and rename some variables

* add BLOCK & READY_BLOCK status and corresponding processing code

* revoke UI

* revert unit-test

Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2022-03-07 10:50:40 +08:00
xiangzihao
2ab8c1ca7d
[Fix-8616][WorkFlowLineage] work flow lineage search result missing data (#8684)
* fix bug_8616

* remove meaningless query column

* fix code smell

* fix remaining problems
2022-03-04 14:05:59 +08:00