Commit Graph

273 Commits

Author SHA1 Message Date
bao liang
d99471806e
[bug fix] some bugs from 1.2.0 to 1.3.0 (#2676)
* fix bug: some bugs about worker group from 1.2.0 to 1.3.0

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix ut bug

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-05-12 16:46:16 +08:00
qiaozhanwei
bb52671fee
Worker Group display #2627 (#2630)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

* add comment

* revert comment

* revert comment

* #2499 buf fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* No master don't create command #2571

* No master don't create command #2571

* No master don't create command #2571

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-08 15:43:11 +08:00
ywywn
8c8e128d18
fix code annotation error(#2603) (#2614) 2020-05-08 15:23:31 +08:00
743294668
30d57f304a
Fix notes (#2615)
Co-authored-by: taoyuxin <taoyuxin@inspur.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
2020-05-08 11:27:29 +08:00
xingchun-chen
d3640ebb68
Merge branch 'dev' into dev 2020-05-06 14:16:40 +08:00
hgaol
85f3ac10a0 feat: add plugin management for alert service 2020-04-30 15:35:18 +08:00
张世鸣
798b133ec1 fix #2481 2020-04-29 14:49:15 +08:00
qiaozhanwei
a12103b08c
no valid worker group,master can kill task directly (#2541)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

* add comment

* revert comment

* revert comment

* #2499 buf fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-28 18:39:53 +08:00
xingchun-chen
7a100086b7
Increase dataX environment variable, sslTrust default value (#2555)
* add LoginTest license

* Delete useless packages

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* Update worker_group_id to worker_group in init.sql

* Update worker_group_id to worker_group in init.sql

* Update worker_group_id to worker_group

* Increase dataX environment variable, sslTrust default value

Co-authored-by: chenxingchun <438044805@qq.com>
2020-04-28 17:57:30 +08:00
t1mon
c50ad1240c
Add query result set limit. (#2459)
* Optimize PropertyUtils instantiation.

* Fix info error.

* Add query result set limit.

* fix code smell

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-04-27 10:23:29 +08:00
qiaozhanwei
db7c0cbd8c
#2499 buf fix (#2518)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

* add comment

* revert comment

* revert comment

* #2499 buf fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-24 16:51:16 +08:00
lgcareer
733e8bcb0d
support download the resources existed in process definition which was old version (#2506)
* fix #2442 and remove unavailable code

* revert verifyResourceName method

* Add ServiceException

* add ServiceExceptionTest

* update ServiceExceptionTest

* add ServiceExceptionTest in pom

* support download the resources existed in process definition which was old version
2020-04-24 14:09:36 +08:00
zixi0825
64d335d9f2
copy process feature #2412 (#2471)
* Update pom.xml (#2467)

* add copy process

* add copy process 2

* add copy process 3

* add copy process 4

* add copy process 5

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-04-24 12:19:30 +08:00
qiaozhanwei
f2bfdc14f7
#2499 bug fix (#2505)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-23 17:37:04 +08:00
qiaozhanwei
54fc32e207
#2486 bug fix (#2487)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-21 18:52:07 +08:00
dailidong
eec92ed1fc
simplify and optimize config (#2469)
* [Refactor worker] simplify and optimize config (#2386)

* simplify config

* simplify config

* simplify and optimize config

* Update HadoopUtils.java

optimize HadoopUtils

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HttpUtils.java

* Update pom.xml

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java
2020-04-21 02:25:45 +00:00
bao liang
4946e88872
move dependent and conditions to master server (#2423)
* move condition and dependent to master

* move conditions to master

* move conditions and dependent task to master

* move conditions to master

* update test

* add log

* add test for dependent task

* add test for dependent task

* update

* update

* refactor complexity code

* refactor complexity code

* add conditions task test

* add conditions task test

* update

* update host to host:port

* update logback.xml

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-04-20 11:31:15 +08:00
songgg
f929c6f3b7
[optimization]:prevent repeated database updates (#2396)
* sqlTask failed to run

* prevent repeated database updates

* prevent repeated database updates

* prevent repeated database updates

Co-authored-by: songqh <songquanhe@foxmail.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-04-16 22:50:26 +08:00
qiaozhanwei
d6798c6db7
1,task status statistics and process status statistics bug fix (#2357) 2,worker group bug fix (#2430)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-16 18:16:19 +08:00
zixi0825
f21837076e
Solve the failure to execute non-query sql (#2413)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: lgcareer <18610854716@163.com>
2020-04-15 17:39:12 +08:00
qiaozhanwei
96835ebda2
1,no worker condition , master will while ture wait for worker startup 2,worker response task status sync wait for result (#2420)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-15 16:19:42 +08:00
qiaozhanwei
f65ab56d2d UT modify 2020-04-09 19:07:17 +08:00
qiaozhanwei
b35baa59b8 Merge remote-tracking branch 'remotes/upstream/refactor-worker' into dev 2020-04-07 17:32:54 +08:00
qiaozhanwei
1276e0d8e1 refactor-worker merge to dev bug fix 2020-04-07 17:04:17 +08:00
bao liang
785c34bc66
fix bug:master fault tolerance error (#2350) (#2375)
fix bug: Visit the worker page of the monitoring center, a null pointer occur(#2349)

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-04-07 16:23:31 +08:00
qiaozhanwei
b45e70f8fa refactor-worker merge to dev bug fix 2020-04-07 15:50:59 +08:00
qiaozhanwei
2ec668df9b refactor-worker merge to dev bug fix 2020-04-07 15:23:36 +08:00
qiaozhanwei
223bdec85a refactor-worker merge to dev bug fix 2020-04-07 15:12:57 +08:00
qiaozhanwei
41d8abdafb test 2020-04-07 14:33:38 +08:00
qiaozhanwei
b5546ddb68 refactor-worker merge to dev bug fix 2020-04-07 14:28:05 +08:00
qiaozhanwei
433b41dd7f refactor-worker merge to dev bug fix 2020-04-07 13:29:20 +08:00
qiaozhanwei
94713cef79 refactor-worker merge to dev bug fix 2020-04-03 15:24:18 +08:00
qiaozhanwei
be1494fd7f refactor-worker merge to dev bug fix 2020-04-03 12:35:56 +08:00
zixi0825
69e000b542
#2282 fix workflow dependent bug (#2329)
* fix workflow dependent bug

* fix workflow dependent bug 2

* fix workflow dependent bug 2

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-04-03 00:16:01 +08:00
qiaozhanwei
4bda3cd92c conflict reslove 2020-04-01 18:29:47 +08:00
qiaozhanwei
152c814126 test 2020-03-31 16:51:19 +08:00
Rubik-W
4440b35c4e
fix #2334 (#2335)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-03-30 16:22:44 +08:00
Tboy
3bafa97f96
Refactor worker (#2331)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory

* refactor TaskCallbackService

* add ZookeeperNodeManagerTest

* add NettyExecutorManagerTest

* refactor TaskKillProcessor

* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest

* add RoundRobinHostManagerTest, ExecutorDispatcherTest

* refactor task response service

* add TaskResponseServiceTest

* modify close method for MasterSchedulerService
2020-03-30 10:39:52 +08:00
qiaozhanwei
68cb81fdf5
1,add UT in pom 2,refactor TaskUpdateQueue (#2326)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

* master fault tolerant bug modify

* UT add pom.xml

* timing online  modify

* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem

* TaskExecutionContext set host

* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify

* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify

* add UT in pom.xml

* revert dolphinscheduler-daemon.sh

* ZookeeperRegister use common.properties zookeeperRoot path

* api start exclude org.apache.dolphinscheduler.server.*

* ZookeeperRegister use common.properties zookeeperRoot path

* 1,api start load server filter
2,SHELL task exitStatusCode modify

* java doc error modify

* java doc error modify

* remove todo

* add UT in pom

* 1,add UT in pom
2,refactor TaskUpdateQueue

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-28 22:34:47 +08:00
t1mon
d4735334a1
Fix SqlTask kerberos load scope unreasonable. #2178 (#2321)
* Optimize PropertyUtils instantiation.

* Fix info error.

* Fix SqlTask kerberos load scope unreasonable. #2178
2020-03-28 17:42:33 +08:00
lgcareer
38c965ddc9
new feature for #404 add resource tree function (#2323)
* add create resource directory

* add create resource directory

* update the resource test

* add upgrade sql in version 1.2.2

* Adding request parameter id to update queryResourceListPaging

* set isDirectory value is false default

* add full name to update updateResource

* remove request parameter isDirectory to update createResource method

* update queryResourceListPaging with change get to post

* update updateResource method with remove fullName

* File management list modification (#1976)

* add resource component

* add resource tree visitor

* return json string

* update queryResourceList

* upload file need fullName

* add method rootNode

* Shell task resources and authorization resources (#1989)

* File management list modification

* Shell task resources and authorization resources

* download resource when execute task

* download resource when execute task

* update authorization type

* download resource when execute task

* Spark task resource changes (#1990)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* download resource when execute task

* update udf function service

* add resource type in ResourceComponent

* UDF resource tree and change DAG style (#2019)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* add deleteIds method in ResourceMapper and ResourceMapperTest

* Add comments on class and method

* add queryResourceByName method in controller

* update verify-name with change name to full name

* update queryResource with add parameter pid

* update queryResource with add parameter pid

* add resource ids in process definition and delete resource need judge whether it is used by any process definition

* Breadcrumb development (#2033)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix (#2040)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Fix github action rerun failed (#2067)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Change crumb position

* Change crumb position (#2068)

* build resource process definition map

* UDF changed to multiple choice

* UDF changed to multiple choice (#2077)

* Change crumb position

* UDF changed to multiple choice

* build resource process definition map (#2076)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* update resource name also need update all the children full name (#2096)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Limit customization file content to no more than 3000 lines(#2128) (#2140)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList (#2192)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* Modify the main jar package

* Modify the main jar package (#2200)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Modify the main jar package

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* choose main jar with resource tree (#2220)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* getResouDelete checksum and modify parameter namerceId

* revert .env

* remove parameter pid

* Delete request interface

* go back to the last page

* jar interface call

* Fix issue #2234 and #2228

* change resource name with full name

* Fix issue #2234 and #2228 (#2246)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* revert .env

* remove parameter pid

* Fix issue #2234 and #2228

* change resource name with full name

* Fix list query value error

* remove unauth-file with authorize-resource-tree

* Repair data cannot be echoed

* Repair data cannot be echoed

* execute mr and spark task need query resource name before

* Authorized resource interface replacement

* Authorized resource interface replacement

* Filter UDF resources

* Change parameters

* need query all authorized directory children when create task

* Change normalize.scss import method and animation.scss license modification

* Delete file list update processing

* It's fixed that resource not deleted in hdfs when delete it.

* add tooltips

* add tooltips (#2310)

* Echo workflow name and modify udf management name

* [new feature]add resource tree function

* revert front code in order to be same as dev branch

* revert front code in order to be same as dev branch

* revert common.properties and application.properties

* add super method

* update flink parameter test

* update flink parameter and unit test

* update resource service test

* If resource list is empty,need init it

* update flink parameter test

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-03-28 15:43:07 +08:00
gabry.wu
48d7612cd5
Adapting partial code(file name start with S #2) to the sonar cloud rule (#2270) 2020-03-28 11:50:38 +08:00
liwenhe1993
c706b21c62
taskProps.getScheduleTime() may be null, but there is no check if it … (#2256)
* taskProps.getScheduleTime() may be null, but there is no check if it is null or not

* Add unit test
2020-03-28 10:42:46 +08:00
Tboy
23ba035182
add TaskResponseServiceTest (#2325)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory

* refactor TaskCallbackService

* add ZookeeperNodeManagerTest

* add NettyExecutorManagerTest

* refactor TaskKillProcessor

* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest

* add RoundRobinHostManagerTest, ExecutorDispatcherTest

* refactor task response service

* add TaskResponseServiceTest
2020-03-27 22:03:09 +08:00
Tboy
5730bfe2be
Refactor worker (#2319)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory

* refactor TaskCallbackService

* add ZookeeperNodeManagerTest

* add NettyExecutorManagerTest

* refactor TaskKillProcessor

* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest

* add RoundRobinHostManagerTest, ExecutorDispatcherTest

* refactor task response service
2020-03-27 09:43:45 +08:00
gabry.wu
eef8bf4e6d
Adapting partial code(file name start with P #2) to the sonar cloud rule (#2260) 2020-03-26 23:36:17 +08:00
孙朝和
d8cba5ff1b
resolve shell task build command nullpointexception (#2267)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-03-26 23:25:15 +08:00
t1mon
efde8fd4e7
Refactor BaseDataSource and unified handling of JDBC resource releases. (#2184)
* Refactor BaseDataSource.isConnectable().

* Add BaseDataSource test.

* Unified handling of JDBC resource releases.

* Fix sonar check error.

* Remove useless logger.

* Refactor BaseDataSource.getJdbcUrl().

* Modify ConnectionUtils.releaseResource().

* Add javadoc

* Remove useless method.

* Add BaseDataSourceTest UT
2020-03-26 23:23:19 +08:00
gabry.wu
d83a94a305
Adapting partial code(file name start with T) to the sonar cloud rule (#2271) 2020-03-26 23:17:28 +08:00
gabry.wu
42f9385ee4
Adapting partial code(file name start with W) to the sonar cloud rule (#2287) 2020-03-26 20:04:30 +08:00