Commit Graph

687 Commits

Author SHA1 Message Date
Jiajie Zhong
ae6aa53f96
[python] Fix submit and run error (#10792)
because we add permission, so it failed out
createQueue method. this patch fix it and
do some refactor of our tenant and queue
validator code
2022-07-08 10:54:41 +08:00
qianli2022
1e48bbc5c6
[Feature-10629][Improvement] Support multi cluster environments - namespace use cluster (#10650)
* server code

* namespace ui

* fix dao test

* fix test

* fix

* docs and image update

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-07-07 16:05:52 +08:00
WangJPLeo
b8a90d801e
[Improvement] Resource authority module database table unique index increased. (#10807)
* Resource authority module database table unique index increased.

* add unique in sql file

* rerun

* ut fix

* e2e rerun

* dao un fix
2022-07-07 14:35:50 +08:00
WangJPLeo
d2fe16d252
[Optimization] Config module resource relation fix. (#10718)
* Config module resource relation fix.

* Boolean judge fix

* e2e rerun

* e2e rerun

* e2e rerun

* ArrayList fix

* change variable name.

* Add transaction processing on create data resource method
2022-07-05 15:51:23 +08:00
Wenjun Ruan
1b86394d3c
Optimize sql script use insert ignore (#10700) 2022-06-30 19:04:57 +08:00
WangJPLeo
bc4ceda4cf
TaskGroup remove query func permission check. (#10557) 2022-06-23 10:00:31 +08:00
WangJPLeo
20348578a2
[Feature] [Permission]Data permission module added and Query class api remove verification. (#10504)
* Data permission module added

* ut fix.

* queueController ut fix.

* queue service ut fix.

* e2e rerun
2022-06-22 15:53:36 +08:00
chuxing
0dd6f4008e
[fix][master]: fix condition npe (#10478)
* [fix][master]: fix condition npe

* [fix][master]: fix condition npe

Co-authored-by: huaangcg <huangcg@getui.com>
2022-06-17 13:40:11 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479)
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
SmallProgrammer
7aa9fa4965
[imporve] Add index to table t_ds_relation_process_instance (#10461) 2022-06-16 15:24:05 +08:00
qianli2022
ff065d8e5b
[Feature][Improvement] Support multi cluster environments - k8s type (#10096)
* service code

* [Feature][UI] Add front-end for cluster manage

* fix e2e

* remove comment on cluster controller

* doc

* img

* setting e2e.yaml

* test

* rerun e2e

* fix bug from comment

* Update index.tsx

use Nspace instead of css.

* Update index.tsx

Remove the style.

* Delete index.module.scss

Remove the useless file.

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>
2022-06-15 13:39:20 +08:00
陈家名
b86dc53ad1
[fix][Python] Support same task name in project defferent process definition (#10428)
close: #10431
2022-06-14 13:53:18 +08:00
WangJPLeo
9782fe4e7c
[Fix-10367] Permission check related key adjustment. (#10368)
* Permission check related key adjustment.

* resource create or upload fix.

* Analysis Permission fix.

* Ut coverage add.

* rerun

* source permission fix.

* ut coverage is achieved

* Reference permission package path modification

* k8s path bug fix.

* k8s path bug fix.

* Corrected controller method name and rerun.
2022-06-09 19:10:34 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup (#10373) 2022-06-08 19:42:05 +08:00
WangJPLeo
e3e39cbdea
[Fix] Privilege Reconstruction Data Query Fix & Resource Creation Post Oper… (#10313)
* Privilege Reconstruction Data Query Fix & Resource Creation Post Operations.

* Resource Retry

* e2e retry

* create project log add

* project list query log add

* clean project log

* delete delay

* delete delay

* remove post handle

* project e2e

* browser refresh

* browser refresh

* e2e fix

* e2e browser refresh

* rowk flow e2e fix

* mapper deduplication

* udf e2e

* e2e
2022-06-01 20:29:10 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
Wenjun Ruan
3a9e67325e
Fix database initialize late in standalone. (#10296) 2022-05-31 11:52:00 +08:00
WangJPLeo
a4948f58e6
[Feature][Permission] Reconstruction of permissions of resource center and monitoring center. (#10307)
* Reconstruction of permissions of resource center and monitoring center.

* clear local logs.

* resource query fix
2022-05-31 08:14:45 +08:00
hstdream
53ab6f7b92
[Feature][API] Modify the permissions of project management, security center, data source center and data quality module. (#10289)
* Resource userName is null

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

Co-authored-by: houshitao <shitaohou@163.com>
2022-05-30 23:32:25 +08:00
WangJPLeo
7ac8ea493c
[Feature][Permission] Project management module authority control refactoring, authority abstract interface setting. (#10261)
* Create process definition with task group and task group priority cannot be save into db

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* The result of the assertion cannot be changed. The only difference is the judgment condition.

* test recovery.

* controller test fix.

* ProjectE2ETest fix.

* WebElement import.

* Prioritize problem solving.

* Judging the length of the set fix

Co-authored-by: houshitao <shitaohou@163.com>
Co-authored-by: hstdream <33045461+hstdream@users.noreply.github.com>
2022-05-29 16:14:16 +08:00
Jiajie Zhong
d8db2b57d2
[fix] Task group queue query all list error (#10237)
Add support query all task queue by not given specific groupId

close: #10102
2022-05-25 16:33:23 +08:00
longtb
bd34e66b8a
[Fix-10076] queryUserByToken use server timezone (#10177)
* [Fix-10076] queryUserByToken use server timezone

* [Fix-10076] queryUserByToken use server timezone
2022-05-23 09:54:11 +08:00
BaoLiang
ee2b855ced
[BUG][TaskGroup] Task group does not take effect (#10093)
* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect
2022-05-18 18:40:36 +08:00
hstdream
701b535ae6
[Fix-10082]Resource userName is null and udf function manager add a username (#10064)
* Resource userName is null

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

Co-authored-by: houshitao <shitaohou@163.com>
2022-05-18 11:33:08 +08:00
hstdream
cd70ad584a
Create process definition with task group and task group priority cannot be save into db (#10078)
Co-authored-by: houshitao <shitaohou@163.com>
2022-05-18 09:41:03 +08:00
qianli2022
bc1c15b142
[Feature-8252][doc] K8s and namespace manager docs and web page update (#9881)
* test

* test

* doc

* fix image error

* fix

* Update docs/docs/en/guide/security.md

* Update docs/docs/en/guide/security.md

* Update docs/docs/en/guide/security.md

* Update docs/docs/zh/guide/security.md

* fix doc

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-11 16:40:30 +08:00
Jiajie Zhong
316b611826
[chore] Combine version 206 and 210 sql upgrade to 300 (#9959) 2022-05-10 11:28:56 +08:00
WangJPLeo
fb0f96ed94
[Fix-9868] A task flow definition isolates the runs of different execution strategies by version numbers. (#9869)
* The thread cache task flow definition should get the latest version.

* Coverage on New Code

* Coverage on New Code

* Coverage on New Code

* use an existing method.

* Increase unit test coverage.

* Task flow definitions enforce policy isolation.
2022-05-03 22:52:42 +08:00
Town
fcb32e4c3c
[fix] fix query access token for specified user failed (#9855) 2022-04-30 11:07:00 +08:00
WangJPLeo
ab2d535e54
The node ID under serial is greater than the current ID. (#9820)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-28 11:00:33 +08:00
caishunfeng
70dae6945f
revert delete dumplicate task_definition_log ddl (#8567)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-04-27 15:19:40 +08:00
leiwingqueen
c837580c68
[bug][db] Table relation_project_user have duplicate record (#9536)
* update sql ddl
* fix syntax err
2022-04-27 12:04:08 +08:00
WangJPLeo
f839b527bf
Task group queue supports fuzzy search. (#9784) 2022-04-26 18:03:55 +08:00
WangJPLeo
5c0be8a3d7
A task instance that normally queries the serial wait state. (#9777)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-26 15:47:01 +08:00
WangJPLeo
7bcec7115a
[Fix-9717] The failure policy of the task flow takes effect (#9718)
* Failure policy takes effect.

* Coverage on New Code

* correct description logic

* Compatible with all scenarios

* clearer logic

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 15:29:18 +08:00
yimaixinchen
fb11525e49
[chore] Correct the java doc in funtion DagHelperTest generateDag2 (#9602)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-04-23 10:31:48 +08:00
naziD
10f8c9d983
[Bug-9608] Serialize the task definition failed (#9622)
* BugFix: serialize the task definition failed

* Remove a comment

Co-authored-by: lipandong <pandong.lpd@alibaba-inc.com>
2022-04-21 11:38:41 +08:00
Eric Gao
a863c6f8f1
[Bug][DAO] Update db init script and soft_version (#9628) (#9637) 2022-04-21 11:01:19 +08:00
WangJPLeo
ae84900329
[Fix-9617]New task group project name drop-down data is displayed according to user type. (#9625)
* New task group project name drop-down data by user type

* remove unused import

* avoid import *

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-20 18:24:45 +08:00
qianli2022
165d7aa51f
[Feature][Task] K8s namespace auth manager (#9303)
* k8s auth

* remove log

* fix test

* use constants

* use constants K8S_LOCAL_TEST_CLUSTER

* simple auth get

* change test

* add namespace authorize in user page

* prettier code

* change test data

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
2022-04-20 18:23:23 +08:00
xiangzihao
a3bf10c88d
[Feature][API] Refactor get dependent node information api (#9591)
* feature_8922

* fix comment error

* remove unused import

* fix unused import
2022-04-20 14:53:40 +08:00
Tq
c5b7e5adff
[Bug] [API-9558]fix homepage task instance count method to use submit time to recount (#9559)
* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method JUNIT

* fix homepage task instance count method JUNIT

* fix homepage task instance count method JUNIT
2022-04-19 15:23:57 +08:00
WangJPLeo
b1d57dbce4
Check the status of the child process when the parent process is running (#9567)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-18 20:27:11 +08:00
Tq
1192fb8e12
fix Fault tolerance warning mapper add alert type to insert (#9533) 2022-04-17 10:17:00 +08:00
WangJPLeo
95ccc0c855
[Fix-9516] [Statistics] Statistics Manager shows data should belongs to current user (#9515) 2022-04-15 18:25:25 +08:00
czeming
706cdb6a8e
[Fix-9174] [Alert] Fix deduplication of alarm information (#9371)
* feat(issue #9174):

Fix-9174
2022-04-14 10:30:25 +08:00
Tq
fb0139e959
[Feature] [ALERT-9406]enrich alert info when insert new row into DB (#9445)
* enrich alert info when insert new row into DB

* fix comments
2022-04-12 14:31:38 +08:00
nobolity
037692517a
[Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)

* fix(server): fix the  udf serialize

* fix(master  task): update the udfTest to update the json string

* fix(test): update the udfFuncTest

* fix(common): syn the common.properties

* fix(udfTest): upgrade the udfTest

* fix(common): revent the common.properties
2022-04-11 10:49:46 +08:00
Tq
e2759a8f42
[Feature] [ALERT-9406]add new properties to alert class (#9408)
* add new properties to Alert.java and do minor changes to comments

* fix Integer to int

* fix Integer to int

* fix sql files

* fix not null properties to default null
2022-04-08 23:54:34 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00