Commit Graph

2344 Commits

Author SHA1 Message Date
qiaozhanwei
7a9e1c4fbf remove ExecuteTaskRequestCommand and ExecuteTaskResponseCommand 2020-06-29 17:38:11 +08:00
dailidong
038bde5efe [Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms (#2318)
* update logback

* update log

* refactor worker registry (#2107)

* Refactor worker (#2115)

* refactor worker registry

* refactor master server

* Modify workgroupid parameter name (#2105)

* Delete worker group management page

* Modify workgroupid parameter name

* Refactor worker (#2121)

* refactor worker registry

* refactor master server

* refactor MasterSchedulerService

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* not exist in openjdk,just delete

* add master and worker properties

* add master and worker properties

* add master and worker properties

* fix cpu 100% bug

* simplify master、 worker、alert、dao properties

* add master and worker properties

* add master and worker properties

* add master and worker properties

* Optimize code , reduce cost time from 60ms to 0.5ms

* merge code form remote

Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:38:04 +08:00
qiaozhanwei
cc3ab0f191 timing online modify (#2296)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

* master fault tolerant bug modify

* UT add pom.xml

* timing online  modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:37:58 +08:00
Tboy
87b994230c Refactor worker (#2248)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory

* refactor TaskCallbackService

* add ZookeeperNodeManagerTest
2020-06-29 17:37:55 +08:00
Tboy
c61c59ae54 Refactor worker (#2186)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test
2020-06-29 17:37:49 +08:00
qiaozhanwei
8b990b5c1c java doc error modify (#2004)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify
2020-06-29 17:37:14 +08:00
qiaozhanwei
bcc5690f58 LogClientService modify 2020-06-29 17:37:00 +08:00
songgg
0ad610a0f0 [optimization]:remove duplicate code (#2377)
* sqlTask failed to run

* remove duplicate code

* remove duplicate code

* remove duplicate code

* remove duplicate code

* remove duplicate code

Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:42 +08:00
itbasketplayer
d03f9daa0e .gitignore remove taskInstance/index.vue(#2300) and no need to findProcessDefineById 2 times(#2373) (#2374)
* some optimization

* revert

* ProcessDefinitionService#updateProcessDefinition findProcessDefineById 1 time
2020-06-29 17:36:42 +08:00
zixi0825
3f044e7771 #2282 fix workflow dependent bug (#2329)
* fix workflow dependent bug

* fix workflow dependent bug 2

* fix workflow dependent bug 2

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:41 +08:00
break60
4709e44a1e Fix the problem of data echo in script edit box 2020-06-29 17:36:41 +08:00
break60
4b265f057a fix 2020-06-29 17:36:41 +08:00
break60
9c2af1442b Fix the problem of data echo in script edit box 2020-06-29 17:36:41 +08:00
break60
e6527c7a92 Corrections to historical jar packages and resource data 2020-06-29 17:36:41 +08:00
break60
b7851639eb Corrections to historical jar packages and resource data 2020-06-29 17:36:41 +08:00
lgcareer
54f04a378e init full_name in dml of version 1.2.2 (#2341)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:41 +08:00
Rubik-W
d8ccab6a49 fix:mail send fail #2345 (#2346)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:41 +08:00
BoYiZhang
7fa0c0f79a Remove invalid code (#2342)
Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:41 +08:00
break60
18d7c66bec Supplemental licenses and repair resources cannot be renamed 2020-06-29 17:36:39 +08:00
break60
3d72643d4a Modify ans-ui version and timing management style 2020-06-29 17:36:38 +08:00
dailidong
db3c6ce861 Delete ReleaseNotes.md (#2338) 2020-06-29 17:36:38 +08:00
Rubik-W
9b6ab81e60 fix #2334 (#2335)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:38 +08:00
songgg
d8c7a44e82 some annotation optimizations (#2333)
Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:38 +08:00
liwenhe1993
c814a30316 Remove .helmignore file (#2328) 2020-06-29 17:36:37 +08:00
liwenhe1993
053dd3263e Support kubernetes deployment (#2153)
* Support kubernetes deployment

* Support kubernetes deployment
2020-06-29 17:36:37 +08:00
t1mon
1d9bc0cd74 Fix SqlTask kerberos load scope unreasonable. #2178 (#2321)
* Optimize PropertyUtils instantiation.

* Fix info error.

* Fix SqlTask kerberos load scope unreasonable. #2178
2020-06-29 17:36:37 +08:00
Rubik-W
668b86f12b fix:Optimize content returned by interface (#2294) 2020-06-29 17:36:37 +08:00
lgcareer
82d6dc9e52 new feature for #404 add resource tree function (#2323)
* add create resource directory

* add create resource directory

* update the resource test

* add upgrade sql in version 1.2.2

* Adding request parameter id to update queryResourceListPaging

* set isDirectory value is false default

* add full name to update updateResource

* remove request parameter isDirectory to update createResource method

* update queryResourceListPaging with change get to post

* update updateResource method with remove fullName

* File management list modification (#1976)

* add resource component

* add resource tree visitor

* return json string

* update queryResourceList

* upload file need fullName

* add method rootNode

* Shell task resources and authorization resources (#1989)

* File management list modification

* Shell task resources and authorization resources

* download resource when execute task

* download resource when execute task

* update authorization type

* download resource when execute task

* Spark task resource changes (#1990)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* download resource when execute task

* update udf function service

* add resource type in ResourceComponent

* UDF resource tree and change DAG style (#2019)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* add deleteIds method in ResourceMapper and ResourceMapperTest

* Add comments on class and method

* add queryResourceByName method in controller

* update verify-name with change name to full name

* update queryResource with add parameter pid

* update queryResource with add parameter pid

* add resource ids in process definition and delete resource need judge whether it is used by any process definition

* Breadcrumb development (#2033)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix (#2040)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Fix github action rerun failed (#2067)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Change crumb position

* Change crumb position (#2068)

* build resource process definition map

* UDF changed to multiple choice

* UDF changed to multiple choice (#2077)

* Change crumb position

* UDF changed to multiple choice

* build resource process definition map (#2076)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* update resource name also need update all the children full name (#2096)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Limit customization file content to no more than 3000 lines(#2128) (#2140)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList (#2192)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* Modify the main jar package

* Modify the main jar package (#2200)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Modify the main jar package

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* choose main jar with resource tree (#2220)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* getResouDelete checksum and modify parameter namerceId

* revert .env

* remove parameter pid

* Delete request interface

* go back to the last page

* jar interface call

* Fix issue #2234 and #2228

* change resource name with full name

* Fix issue #2234 and #2228 (#2246)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* revert .env

* remove parameter pid

* Fix issue #2234 and #2228

* change resource name with full name

* Fix list query value error

* remove unauth-file with authorize-resource-tree

* Repair data cannot be echoed

* Repair data cannot be echoed

* execute mr and spark task need query resource name before

* Authorized resource interface replacement

* Authorized resource interface replacement

* Filter UDF resources

* Change parameters

* need query all authorized directory children when create task

* Change normalize.scss import method and animation.scss license modification

* Delete file list update processing

* It's fixed that resource not deleted in hdfs when delete it.

* add tooltips

* add tooltips (#2310)

* Echo workflow name and modify udf management name

* [new feature]add resource tree function

* revert front code in order to be same as dev branch

* revert front code in order to be same as dev branch

* revert common.properties and application.properties

* add super method

* update flink parameter test

* update flink parameter and unit test

* update resource service test

* If resource list is empty,need init it

* update flink parameter test

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-29 17:36:36 +08:00
gabry.wu
f9c91b8cbe Adapting partial code(file name start with S #2) to the sonar cloud rule (#2270) 2020-06-29 17:36:36 +08:00
break60
4a8e148b42 Resource tree code merge 2020-06-29 17:36:34 +08:00
孙朝和
588c003860 resolve shell task build command nullpointexception (#2267)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-06-29 17:36:34 +08:00
gabry.wu
30b98b89a2 Adapting partial code(file name start with T) to the sonar cloud rule (#2271) 2020-06-29 17:36:33 +08:00
tswstarplanet
0c5ff32797 Decouple DruidDataSource in ConnectionFactory and cache DataSource instance (#2232)
* Decouple DruidDataSource in ConnectionFactory and cache DataSource
instance

* init dataSource first

* fix code smell

* add unit test

* add unit test

* add unit test

* add unit test

* fix unit test

* delete useless logger
2020-06-29 17:36:33 +08:00
gabry.wu
a1f7614476 Adapting partial code(file name start with W) to the sonar cloud rule (#2287) 2020-06-29 17:36:33 +08:00
WeiYe
6cca69369b Update test code logger class name (#2303) 2020-06-29 17:36:33 +08:00
xingchun-chen
e496206a88 e2e add project and workflow case (#2309)
* add LoginTest license

* Delete useless packages

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

Co-authored-by: chenxingchun <438044805@qq.com>
2020-06-29 17:36:32 +08:00
break60
cd529267f5 Change normalize.scss import method and animation.scss license modification 2020-06-29 17:36:31 +08:00
khadgarmage
e37aecdbc7 add ds logs and server status check with e2e test (#2277) 2020-06-29 17:36:31 +08:00
songquanhe-gitstudy
fc18d4a216 fix: No operations allowed after statement closed when running sql task (#2265)
* sqlTask failed to run

* get correct attribute value and logger content

Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:30 +08:00
khadgarmage
5d0f0e9490 Fix checkoutv2 push trigger on dev (#2263)
* push trigger on dev
2020-06-29 17:36:29 +08:00
break60
e31f38a189 datax add custom 2020-06-29 17:36:29 +08:00
khadgarmage
416e0417ad license and checkoutv2 fix (#2261) 2020-06-29 17:36:29 +08:00
gabry.wu
afa9a6467a Adapting partial code(file name start with Q) to the sonar cloud rule (#2242) 2020-06-29 17:36:29 +08:00
Rubik-W
1acbc0dbfe fix:Depend check bug (#2229)
* fix:#1787

* fix:#1787
2020-06-29 17:36:29 +08:00
gabry.wu
e1771de45a Adapting partial code(file name start with R) to the sonar cloud rule (#2243) 2020-06-29 17:36:29 +08:00
jin
24fa31939b feat:optimizing code (#2245)
* feat:optimizing code

* add licence and  format code
2020-06-29 17:36:29 +08:00
tswstarplanet
bc65253fd1 fix blocker problem (#2244)
* fix blocker problem

* use error level to log exception
2020-06-29 17:36:29 +08:00
John Liu
4898447236 Optimizing code (#2240)
* updated

* optimized the code

* Delete ParameterUtils.java

* rollback source code

* rollback source code

* rollback source code

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:29 +08:00
Simon
5c30205574 support custom datax configuration (#2224)
* fix #1441

* support custom datax config

* support datax custom config

* support datax custom config

* support datax custom config

Co-authored-by: 张世鸣 <zhangshiming@cvte.com>
2020-06-29 17:36:29 +08:00
tswstarplanet
5a74b8bbf5 Refactor ConnectionFactory to static inner holder Singleton (#2204)
* refactor ConnectionFactory to static inner holder Singleton

* remove redundant import

* make getSqlSessionFactory() method private

* fix sonar issue

* remove @Ignore of MailUtilsTest

* add MailUtilsTest to pom.xml to test

* fix MailUtilsTest path error in pom.xml

* modify test method name

* add assert and logger to unit test

* add log

* add log

* add AlertDaoTest

* move AlertDaoTest to new module

* modify test in pom.xml

* remove unnecessary log
2020-06-29 17:36:29 +08:00