Commit Graph

89 Commits

Author SHA1 Message Date
baoliang
c7005b4944 refactor code style 2020-11-12 11:22:40 +08:00
baoliang
a8205c2a9b merge from 1.3.3-release 2020-11-09 10:57:27 +08:00
bao liang
7239979167
Merge branch 'dev' into 133-merge-dev 2020-11-09 10:08:19 +08:00
zhuangchong
4b502e361f
[Improvement-4012][common/remote] Json util code integration,remove the remote module json util (#4013)
* Json util code integration,remove the remote module json util.

* update code checkstyle.

* update code checkstyle.

* add code checkstyle.

* add test class.

* update JSONUtils class.

* update JSONUtils class.
2020-11-07 19:37:48 +08:00
baoliang
403a4a20f2 merge from 1.3.3-release 2020-11-06 11:44:34 +08:00
baoliang
381d23ea4f merge from 1.3.3-release 2020-11-06 00:03:20 +08:00
lgcareer
51d476be69
[FIX_#3789][remote]cherry pick from dev to support netty heart beat (#3913)
* [FIX_#3789][remote]cherry pick from dev to support netty heart beat

* [FIX_#3789][remote]cherry pick from dev to support netty heart beat

Co-authored-by: Kirs <acm_master@163.com>
2020-10-15 10:28:05 +08:00
Kirs
08dc6913dc
[FIX_BUG_1.3#3789][remote]support netty heart beat (#3868)
* [FIX_BUG_1.3#3789][remote]support netty heart beat

* netty heart beat

* code style

* code style

* code style

* code style

* code style

* delete code docs

* client fail-close
2020-10-12 23:29:38 +08:00
guyinyou
3b581455fc
[Feature]Add Python task "task variable / result transfer" implementation (#3659)
* 增加Python Task的“任务变量/结果传递”实现
Signed-off-by: 古崟佑

* add two files license
Signed-off-by: 古崟佑

* fix 'server/worker/task/AbstractCommandExecutor.java' code style
Signed-off-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>

* update DB
Signed-off-by: 古崟佑

* update DB -- 2
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codestyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* add VarPoolUtils Test
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* add test config for VarPoolUtilsTest
Signed-off-by: 古崟佑

* fix unit test
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest.java
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* change the test class path
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* fix "print the error message"
Signed-off-by: 古崟佑

* fix bug
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

Co-authored-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>
2020-09-25 13:20:42 +08:00
CalvinKirs
db663a13a3
[Improvement][remote]load balance warm up (#3770)
* [Improvement][remote]load balance  warm up

* reformat code

* reformat code

* code smell

* code smell

* add test

* add test

* add test

* add test

* fix bug

* fix bug

* add docs

* add host test

* add host test

* add host test

* add docs

* code reformat

* code reformat
2020-09-22 17:39:39 +08:00
qiaozhanwei
29f1123d1d
[Fix-3616][Server] when worker akc/response master exception , async retry (#3748) 2020-09-17 10:06:59 +08:00
CalvinKirs
7d49fbbaec
[Improvement][remote] set connect time out (#3670) 2020-09-06 16:31:35 +08:00
CalvinKirs
86bcc058a5
[Test][remote]update netty util test and fix init channel error (#3671) 2020-09-04 17:18:09 +08:00
CalvinKirs
d280820f86
[Improvement][remote] linux platform default use epoll (#3667) 2020-09-04 13:27:14 +08:00
CalvinKirs
417de132a7
[improvement][remote]use EpollServerSocketChannel when system is linux (#3657) 2020-09-04 11:10:03 +08:00
CalvinKirs
34a04acafd
[Improvement][remote] improvement netty eventLoopGroup (#3580)
* [Improvement][remote] improvement netty eventLoopGroup
If you are running on linux you can use EpollEventLoopGroup and so get better performance, less GC and have more advanced features that are only available on linux.

* reformat code

* utility classes should not have public constructors

* reformat code

* reformat code

* reformat code

* add test

* code style

* add test

* add test

* fix test error

* add test

* test

* test

* add test config

* add test config
2020-08-25 10:47:40 +08:00
vanilla111
a4ee351a3a
delay execution of tasks and improve some designs (#3427) 2020-08-19 11:36:30 +08:00
CalvinKirs
d6a32ac652
[Feature#3234][cluster]enhanced load balancing (#3235)
* [Future#3234][cluster]enhanced load balancing
weight-based load balancing algorithm
this close # 3234

* remove useless parameter

* code smell

* load balancing according to work group

* add smooth weight round robin

* remove unused constants

* perfect unit test

* code smell

* code smell

* add work weight config

* fix config error

* add weight docs to readme.md
2020-08-10 19:05:21 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
qiaozhanwei
1eb8fb6db3
NettyDecoder class decode method switch case remove break (#3035) 2020-07-10 11:10:13 +08:00
tswstarplanet
bf3cc0d00e
[Bug][Refactor][issue-3157]use cas to avoid thread safe problem (#3158)
Co-authored-by: lgcareer <18610854716@163.com>
2020-07-07 18:45:20 +08:00
dailidong
4a4e940379
fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)
* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method
2020-07-04 15:49:46 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
gaojun2048
58897aa2c5 merge 1.3.0-release branch code 2020-06-29 18:42:26 +08:00
gaojun2048
3cf9afc195 merge 1.3.0-release branch code 2020-06-29 18:42:18 +08:00
qiaozhanwei
7a9e1c4fbf remove ExecuteTaskRequestCommand and ExecuteTaskResponseCommand 2020-06-29 17:38:11 +08:00
dailidong
038bde5efe [Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms (#2318)
* update logback

* update log

* refactor worker registry (#2107)

* Refactor worker (#2115)

* refactor worker registry

* refactor master server

* Modify workgroupid parameter name (#2105)

* Delete worker group management page

* Modify workgroupid parameter name

* Refactor worker (#2121)

* refactor worker registry

* refactor master server

* refactor MasterSchedulerService

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* not exist in openjdk,just delete

* add master and worker properties

* add master and worker properties

* add master and worker properties

* fix cpu 100% bug

* simplify master、 worker、alert、dao properties

* add master and worker properties

* add master and worker properties

* add master and worker properties

* Optimize code , reduce cost time from 60ms to 0.5ms

* merge code form remote

Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:38:04 +08:00
gabry.wu
e1771de45a Adapting partial code(file name start with R) to the sonar cloud rule (#2243) 2020-06-29 17:36:29 +08:00
tswstarplanet
bc65253fd1 fix blocker problem (#2244)
* fix blocker problem

* use error level to log exception
2020-06-29 17:36:29 +08:00
gabry.wu
7ac6ad5491 Adapting partial code(file name start with N) to the sonar cloud rule (#2215) 2020-06-29 17:36:28 +08:00
t1mon
55cc1c7c5b Replace Charset.forName("UTF-8") with StandardCharsets.UTF_8. (#2193) 2020-06-29 17:36:28 +08:00
Rubik-W
bd43bed64d fix: remove redundant import class (#2110) 2020-06-29 17:36:13 +08:00
zixi0825
9bf67d80d0
Revise annotation spelling errors & Enhanced code robustness (#3042)
* revise annotation spelling errors & enhanced code robustness

* revise annotation spelling errors & enhanced code robustness

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-28 10:48:14 +08:00
lenboo
b81872cacd merge from dev-1.3.0 2020-06-19 18:52:30 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
superScala
8031c1624a
add break & default logic for NettyDecoder.decode (#2969)
* add break & default logic for NettyDecoder.decode

* fix the bad code smells

Co-authored-by: gabry <wu_shao_jie@qq.com>
Co-authored-by: gabry.wu <8545796+gabrywu@users.noreply.github.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-16 14:25:40 +08:00
superScala
3a7fd725c7
first show: remove unnecessary code (#2968)
Co-authored-by: gabry <wu_shao_jie@qq.com>
2020-06-14 13:49:20 +08:00
simon824
24120a4641 fix the bad code smell 2020-06-12 11:27:42 +08:00
simon824
65841a5709 1. change windows new line 'CR' to linux new line 'LF'
2. solve conflict
2020-06-12 09:31:04 +08:00
simon824
f643981f48 1. change windows new line 'CR' to linux new line 'LF'
2. Format code
2020-06-11 10:14:15 +08:00
simon824
6e907f150c 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 13:02:50 +08:00
dailidong
186ba3baab
fix oom when no master is active in dev-1.3.0 (#2918)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect
2020-06-08 15:55:39 +08:00
simon
33bd6586dd Using Jackson instead of Fastjson 2020-06-03 09:19:59 +08:00
qiaozhanwei
f4583d78e9
add UT (#2717)
* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-14 17:50:47 +08:00
qiaozhanwei
443a6d1935
delete log files while deleting process instances #2463 (#2693)
* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

* delete log files while deleting process instances #2463

* delete log files while deleting process instances #2463

* delete log files while deleting process instances #2463

* delete log files while deleting process instances #2463

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-12 16:54:29 +08:00
qiaozhanwei
e837a73fcd
EnterpriseWeChatUtils modify (#2502)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-23 16:54:54 +08:00
qiaozhanwei
1f43925c12 remove ExecuteTaskRequestCommand and ExecuteTaskResponseCommand 2020-04-01 18:34:25 +08:00
qiaozhanwei
70d6360149 startup conflict reslove 2020-03-31 17:51:29 +08:00
qiaozhanwei
152c814126 test 2020-03-31 16:51:19 +08:00
qiaozhanwei
c2cf8fb7b1 test 2020-03-30 11:52:27 +08:00