Commit Graph

386 Commits

Author SHA1 Message Date
houshitao
f8650b7b7c
[Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun。 (#10727)
* [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun.

* [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun.

* [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun.

Co-authored-by: houshitao <shitaohou@163.com>
2022-07-04 22:05:23 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. (#10704)
* Global parameter and local parameter calculation external expansion.

* k8s task ut fix.

* TimePlaceholderUtils import DateUtils fix

* follow the review comments to fix.

* follow the review comments to fix.

* e2e rerun
2022-06-30 22:45:25 +08:00
Wenjun Ruan
247ca4ae8a
[Improvement-10617] Add comment in slot check (#10618) 2022-06-30 15:58:49 +08:00
wjf
75581b7543
[Improvement]Delete the getSession in the Registry interface (#10633) 2022-06-29 11:20:52 +08:00
Wenjun Ruan
35b25da863
Validate master/worker config (#10649) 2022-06-28 20:17:43 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover (#10631)
* fix worker failover may lose event
2022-06-28 16:08:35 +08:00
WangJPLeo
b5184138fa
[Feature] Time function analysis extension. (#10624)
* Time function analysis extension.

* param add.

* clear useless logs and update method notes

* permission omission fix.

* extending time functions to optimize static methods

* e2e rerun.
2022-06-28 11:24:51 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516)
* Optimize master log, add workflow instance id and task instance id in log

* Use MDC to set the workflow info in log4j

* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
WangJPLeo
f46faa02c3
[Fix-10452] Serial wait for policy recovery (#10453)
* Serial wait for policy recovery

* processInstance state check null

* add sendresult(Host host, Command command) method and replace the original.
2022-06-22 14:28:50 +08:00
Kerwin
cefcb6d556
[Fix-10522] [API] Fix get executable time null pointer exception (#10523) 2022-06-22 08:58:19 +08:00
Eric Gao
cc06eaaf54
[Improvement][Metrics] Apply micrometer naming convention to metrics (#10477)
* Apply micrometer naming convention to worker metrics
* Apply micrometer naming convention all current metrics
* Fix remaining metrics names, update English docs and add Chinese docs
* Fix metrics names in grafana-demo dashboards
2022-06-21 14:27:06 +08:00
Wenjun Ruan
0bdfa0cff9
Fix PeerTaskInstancePriorityQueue cannot contains method use taskInstanceId to check (#10491) 2022-06-17 16:14:56 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479)
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
hstdream
f6fea06f10
[Improve] Enhance complement function transformation (#10376) 2022-06-12 12:35:08 +08:00
Wenjun Ruan
b0d9d3f9ab
Add some warning log in master (#10383)
* Add some warn log in master

* fix may skip sleep
2022-06-10 19:38:20 +08:00
Wenjun Ruan
e21d7b1551
[Feature][metrics] Add master, worker metrics (#10326)
* Add mater metrics

* fix UT

* Add url to mysql profile

* Add worker metrics

* Update grafana config

* Add system metrics doc

* Add process failover counter

* Add metrics image

* Change jpg to png

* Add command insert metrics

* Fix UT

* Revert UT
2022-06-09 10:55:39 +08:00
Wenjun Ruan
3ecbee3885
Optimize MasterServer, add MasterRPCService (#10371)
* Optimize MasterServer, avoid NPE
2022-06-09 10:00:05 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup (#10373) 2022-06-08 19:42:05 +08:00
Wenjun Ruan
2d3be6b36c
Add dolphinscheduler-scheduler module (#10360)
* Add dolphinscheduler-scheduler module
2022-06-04 16:39:33 +08:00
Wenjun Ruan
022e4886be
Remove quartz at WorkerServer (#10358)
* Remove quartz at WorkerServer

* move k8s and permission from dolphinscheduler-service to dolphinscheduler-api
2022-06-04 00:18:01 +08:00
Wenjun Ruan
0bf4043d5f
Move datasource params to datasouce plugin module (#10353)
* Move datasource params to datasouce plugin module
2022-06-03 12:15:53 +08:00
WangJPLeo
e3e39cbdea
[Fix] Privilege Reconstruction Data Query Fix & Resource Creation Post Oper… (#10313)
* Privilege Reconstruction Data Query Fix & Resource Creation Post Operations.

* Resource Retry

* e2e retry

* create project log add

* project list query log add

* clean project log

* delete delay

* delete delay

* remove post handle

* project e2e

* browser refresh

* browser refresh

* e2e fix

* e2e browser refresh

* rowk flow e2e fix

* mapper deduplication

* udf e2e

* e2e
2022-06-01 20:29:10 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
WangJPLeo
a4948f58e6
[Feature][Permission] Reconstruction of permissions of resource center and monitoring center. (#10307)
* Reconstruction of permissions of resource center and monitoring center.

* clear local logs.

* resource query fix
2022-05-31 08:14:45 +08:00
hstdream
53ab6f7b92
[Feature][API] Modify the permissions of project management, security center, data source center and data quality module. (#10289)
* Resource userName is null

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

Co-authored-by: houshitao <shitaohou@163.com>
2022-05-30 23:32:25 +08:00
cadl
13af2adfc1
[fix] Fix the SERIAL_DISCARD execution_type not working (#10258)
* fix always stop process_instance when execution_type is SERIAL_DISCARD

* test: no running_process, submit it
2022-05-30 09:54:15 +08:00
WangJPLeo
7ac8ea493c
[Feature][Permission] Project management module authority control refactoring, authority abstract interface setting. (#10261)
* Create process definition with task group and task group priority cannot be save into db

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* The result of the assertion cannot be changed. The only difference is the judgment condition.

* test recovery.

* controller test fix.

* ProjectE2ETest fix.

* WebElement import.

* Prioritize problem solving.

* Judging the length of the set fix

Co-authored-by: houshitao <shitaohou@163.com>
Co-authored-by: hstdream <33045461+hstdream@users.noreply.github.com>
2022-05-29 16:14:16 +08:00
BaoLiang
ee2b855ced
[BUG][TaskGroup] Task group does not take effect (#10093)
* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect
2022-05-18 18:40:36 +08:00
He Zhao
5bb1eb04fc
[Feature][plugin] Add k8s task in task plugin (#9425)
* [Feature][plugin] Add k8s task in task plugin

* [Feature][plugin] fix dos and code problems

* [Feature][plugin] refactor some code based on sonar

* [Feature][UI] front-end for k8s task plugin

* [Feature][plugin] delete some front files

* [Feature][plugin] update document

* Update docs/docs/zh/guide/task/k8s.md

* Update docs/docs/en/guide/task/k8s.md

* Update docs/docs/en/guide/task/k8s.md

* Update docs/docs/en/guide/task/k8s.md

* [Feature][UI] front-end change from review comments

* [Feature][UI] replace get namespace list api

* [Feature][plugin] change file name

* Add kubernetes to zh task list

Co-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-18 17:23:42 +08:00
Tq
d4aeee16e5
[Bug] [MASTER-9811]fix cmd param to overwrite global param when executing complement (#9952)
* fix cmd param to overwrite global param when executing complement

* fix cmd param to overwrite global param when executing complement
2022-05-11 18:37:03 +08:00
WangJPLeo
fb0f96ed94
[Fix-9868] A task flow definition isolates the runs of different execution strategies by version numbers. (#9869)
* The thread cache task flow definition should get the latest version.

* Coverage on New Code

* Coverage on New Code

* Coverage on New Code

* use an existing method.

* Increase unit test coverage.

* Task flow definitions enforce policy isolation.
2022-05-03 22:52:42 +08:00
caishunfeng
5d5125d507
[Bug-9829][Api] Fix schedule timezone (#9830) 2022-04-29 10:17:34 +08:00
WangJPLeo
5c0be8a3d7
A task instance that normally queries the serial wait state. (#9777)
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-26 15:47:01 +08:00
Tq
e6dade71bb
fix process instance global param not include to task instance when master executing (#9730) 2022-04-25 14:05:40 +08:00
caishunfeng
5657cb9aec
[Bug-9719][Master] fix failover fail because task plugins has not been loaded (#9720) 2022-04-24 20:34:21 +08:00
gaojun2048
ebc4253d50
[fix][Service] BusinessTime should format with schedule timezone (#9714)
* BusinessTime should format with schedule timezone

* fix test error

* fix test error

* fix test error
2022-04-24 19:21:21 +08:00
caishunfeng
88d2803fe1
fix task dispatch error overload resource pool of task group (#9667) 2022-04-22 18:39:40 +08:00
caishunfeng
63638601b0
fix process pause and rerun (#9568) 2022-04-19 10:23:56 +08:00
Tq
fb0139e959
[Feature] [ALERT-9406]enrich alert info when insert new row into DB (#9445)
* enrich alert info when insert new row into DB

* fix comments
2022-04-12 14:31:38 +08:00
caishunfeng
b285ccf930
[Future-9396]Support output parameters transfer from parent workflow to child work flow (#9410)
* [Future-9396]Support output parameters transfer from parent workflow to child work flow

* fix note
2022-04-11 20:03:16 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
gaojun2048
5ef3f9d668
[optimization] [Service] Optimization ProcessService and add ProcessService interface (#9370) 2022-04-07 12:21:34 +08:00
caishunfeng
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719)
* rebase dev

* change task state to dispatch if dispatch success

* update task host when dispatch

* add dispatch task event

* test

* check tenant after enable auto create

* handle dispatch state

Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 17:35:05 +08:00
wangxj3
fd76436d87
[Feature-#8318] [master-server] Sub process transfer parameters to father process (#8522)
* load param from subprocess

* unit test

* modify code style

Co-authored-by: wangxj <wangxj31>
2022-03-30 14:55:59 +08:00
guoshupei
e2c1cc0579
[Fix-9222] [master] Support for deploying multiple masters on one node (#9240)
This closes #9222

Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-29 22:46:02 +08:00
JinYong Li
85e56b5e2a
fix 9243 (#9244) 2022-03-28 20:45:01 +08:00
Tq
03a0c9f544
[Bug] [ALERT-9224] fix wechat alert functions: app and group chat (#9225)
* fix wechat alert with app and group chat

* add apache header and fix imports

* delete wechat alert 'userSendMsg' property code

* delete wechat alert 'userSendMsg' property code in Test

* delete wechat alert 'userSendMsg' property code in factory Test

* change `getSubWorkFlowParam` from private to protected
2022-03-28 20:33:14 +08:00
worry
d3251c9bcc
[Improvement-9227][master]implement use the slot to scan the database (#9228)
when the master assigns tasks by slot,implement use the slot to scan the database.

This closes #9227
2022-03-28 09:52:12 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
gaojun2048
9996f7a4fa
update some method to protected (#9169) 2022-03-24 21:43:13 +08:00