lgcareer
51d476be69
[FIX_#3789][remote]cherry pick from dev to support netty heart beat ( #3913 )
...
* [FIX_#3789][remote]cherry pick from dev to support netty heart beat
* [FIX_#3789][remote]cherry pick from dev to support netty heart beat
Co-authored-by: Kirs <acm_master@163.com>
2020-10-15 10:28:05 +08:00
Kirs
08dc6913dc
[FIX_BUG_1.3#3789][remote]support netty heart beat ( #3868 )
...
* [FIX_BUG_1.3#3789][remote]support netty heart beat
* netty heart beat
* code style
* code style
* code style
* code style
* code style
* delete code docs
* client fail-close
2020-10-12 23:29:38 +08:00
guyinyou
3b581455fc
[Feature]Add Python task "task variable / result transfer" implementation ( #3659 )
...
* 增加Python Task的“任务变量/结果传递”实现
Signed-off-by: 古崟佑
* add two files license
Signed-off-by: 古崟佑
* fix 'server/worker/task/AbstractCommandExecutor.java' code style
Signed-off-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>
* update DB
Signed-off-by: 古崟佑
* update DB -- 2
Signed-off-by: 古崟佑
* fix codeStyle
Signed-off-by: 古崟佑
* fix codestyle
Signed-off-by: 古崟佑
* fix codeStyle
Signed-off-by: 古崟佑
* fix codeStyle
Signed-off-by: 古崟佑
* fix codeStyle
Signed-off-by: 古崟佑
* add VarPoolUtils Test
Signed-off-by: 古崟佑
* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑
* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑
* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑
* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑
* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑
* add test config for VarPoolUtilsTest
Signed-off-by: 古崟佑
* fix unit test
Signed-off-by: 古崟佑
* fix codeStyle
Signed-off-by: 古崟佑
* fix VarPoolUtilsTest.java
Signed-off-by: 古崟佑
* fix
Signed-off-by: 古崟佑
* change the test class path
Signed-off-by: 古崟佑
* fix
Signed-off-by: 古崟佑
* fix "print the error message"
Signed-off-by: 古崟佑
* fix bug
Signed-off-by: 古崟佑
* fix
Signed-off-by: 古崟佑
Co-authored-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>
2020-09-25 13:20:42 +08:00
CalvinKirs
db663a13a3
[Improvement][remote]load balance warm up ( #3770 )
...
* [Improvement][remote]load balance warm up
* reformat code
* reformat code
* code smell
* code smell
* add test
* add test
* add test
* add test
* fix bug
* fix bug
* add docs
* add host test
* add host test
* add host test
* add docs
* code reformat
* code reformat
2020-09-22 17:39:39 +08:00
qiaozhanwei
29f1123d1d
[Fix-3616][Server] when worker akc/response master exception , async retry ( #3748 )
2020-09-17 10:06:59 +08:00
CalvinKirs
7d49fbbaec
[Improvement][remote] set connect time out ( #3670 )
2020-09-06 16:31:35 +08:00
CalvinKirs
86bcc058a5
[Test][remote]update netty util test and fix init channel error ( #3671 )
2020-09-04 17:18:09 +08:00
CalvinKirs
d280820f86
[Improvement][remote] linux platform default use epoll ( #3667 )
2020-09-04 13:27:14 +08:00
CalvinKirs
417de132a7
[improvement][remote]use EpollServerSocketChannel when system is linux ( #3657 )
2020-09-04 11:10:03 +08:00
CalvinKirs
34a04acafd
[Improvement][remote] improvement netty eventLoopGroup ( #3580 )
...
* [Improvement][remote] improvement netty eventLoopGroup
If you are running on linux you can use EpollEventLoopGroup and so get better performance, less GC and have more advanced features that are only available on linux.
* reformat code
* utility classes should not have public constructors
* reformat code
* reformat code
* reformat code
* add test
* code style
* add test
* add test
* fix test error
* add test
* test
* test
* add test config
* add test config
2020-08-25 10:47:40 +08:00
vanilla111
a4ee351a3a
delay execution of tasks and improve some designs ( #3427 )
2020-08-19 11:36:30 +08:00
qiaozhanwei
917a4b20b5
release 1.3.2 version rollback ( #3527 )
...
* [Feature] JVM parameter optimization , related issue #3370
* [Feature] JVM parameter optimization , related issue #3370
* test release 1.3.2 version rollback
* test release 1.3.2 version rollback
* test
* test release 1.3.2 version rollback
* modify tag 1.3.0 to HEAD
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-17 14:21:13 +08:00
qiaozhanwei
32edd19ac5
[maven-release-plugin] prepare for next development iteration
2020-08-17 14:03:19 +08:00
qiaozhanwei
4d2f0b0578
[maven-release-plugin] prepare release 1.3.2
2020-08-17 14:03:08 +08:00
qiaozhanwei
11d3c07e46
modify tag 1.3.0 to HEAD ( #3525 )
...
* [Feature] JVM parameter optimization , related issue #3370
* [Feature] JVM parameter optimization , related issue #3370
* test release 1.3.2 version rollback
* test release 1.3.2 version rollback
* test
* test release 1.3.2 version rollback
* modify tag 1.3.0 to HEAD
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-17 11:46:35 +08:00
qiaozhanwei
f3f895751b
[maven-release-plugin] prepare for next development iteration
2020-08-14 19:20:27 +08:00
qiaozhanwei
f19b125374
[maven-release-plugin] prepare release 1.3.2
2020-08-14 19:20:17 +08:00
qiaozhanwei
4ba6b35b0f
test release 1.3.2 version rollback ( #3504 )
...
* [Feature] JVM parameter optimization , related issue #3370
* [Feature] JVM parameter optimization , related issue #3370
* test release 1.3.2 version rollback
* test release 1.3.2 version rollback
* test
* test release 1.3.2 version rollback
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 18:55:03 +08:00
qiaozhanwei
f50353f8a6
[maven-release-plugin] prepare for next development iteration
2020-08-14 18:26:09 +08:00
qiaozhanwei
afd227a179
[maven-release-plugin] prepare release 1.3.2
2020-08-14 18:25:59 +08:00
qiaozhanwei
660eb5a1e6
test release 1.3.2 version rollback ( #3503 )
...
* [Feature] JVM parameter optimization , related issue #3370
* [Feature] JVM parameter optimization , related issue #3370
* test release 1.3.2 version rollback
* test release 1.3.2 version rollback
* test
* test release 1.3.2 version rollback
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 18:19:19 +08:00
qiaozhanwei
b5f0f23d85
[maven-release-plugin] prepare for next development iteration
2020-08-14 17:37:33 +08:00
qiaozhanwei
88b3ef5d30
[maven-release-plugin] prepare release 1.3.2
2020-08-14 17:37:23 +08:00
qiaozhanwei
45424ec265
test release 1.3.2 version rollback ( #3499 )
...
* [Feature] JVM parameter optimization , related issue #3370
* [Feature] JVM parameter optimization , related issue #3370
* test release 1.3.2 version rollback
* test release 1.3.2 version rollback
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 17:31:09 +08:00
qiaozhanwei
c6f0c8901c
[maven-release-plugin] prepare for next development iteration
2020-08-14 16:04:49 +08:00
qiaozhanwei
8a6732ae4c
[maven-release-plugin] prepare release 1.3.2
2020-08-14 16:04:38 +08:00
CalvinKirs
d6a32ac652
[Feature#3234][cluster]enhanced load balancing ( #3235 )
...
* [Future#3234][cluster]enhanced load balancing
weight-based load balancing algorithm
this close # 3234
* remove useless parameter
* code smell
* load balancing according to work group
* add smooth weight round robin
* remove unused constants
* perfect unit test
* code smell
* code smell
* add work weight config
* fix config error
* add weight docs to readme.md
2020-08-10 19:05:21 +08:00
bao liang
657ea4752f
merge from 1.3.1-release ( #3161 )
...
* merge from 1.3.1-release
* [bugfix] remove Conflicting configuration sudo and user (#3038 )
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
* remove some standard notices in order to cut down the NOTICE file. (#3101 )
* #3084 remove some standard notices in order to cut down the NOTICE file.
* update exclude element sql/soft_version of rat plugin
* update soft_version
* merge from 1.3.1-release
* fix 3112:download udf resource need find the tenant code of resource owner (#3114 )
* fix 3112:download udf resource need find the tenant code of resource owner
* fix 2975:download resource need find the tenant code of resource owner
* remove useless code
* merge from 1.3.1-release
* remove some notes (#3127 )
Co-authored-by: lenboo <baoliang@analysys.com.cn>
* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128 )
* fix Permission denied if not found dolphinscheder-env.sh
* update some comments on method
* add PropertyPlaceholderHelper on license (#3129 )
* update license header location
* add PropertyPlaceholderHelper on license
* merge from 1.3.1-release
* change from 1.3.0 to 1.3.2
* add Set import
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
qiaozhanwei
1eb8fb6db3
NettyDecoder class decode method switch case remove break ( #3035 )
2020-07-10 11:10:13 +08:00
tswstarplanet
bf3cc0d00e
[Bug][Refactor][issue-3157]use cas to avoid thread safe problem ( #3158 )
...
Co-authored-by: lgcareer <18610854716@163.com>
2020-07-07 18:45:20 +08:00
lenboo
572b142a53
[maven-release-plugin] prepare for next development iteration
2020-07-04 16:10:37 +08:00
lenboo
a4272efa55
[maven-release-plugin] prepare release 1.3.1
2020-07-04 16:04:36 +08:00
lenboo
fd3485ad0a
revert version to 1.3.1
2020-07-04 15:54:37 +08:00
dailidong
4a4e940379
fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 ( #3128 )
...
* fix Permission denied if not found dolphinscheder-env.sh
* update some comments on method
2020-07-04 15:49:46 +08:00
lenboo
3d4d816349
[maven-release-plugin] prepare for next development iteration
2020-07-03 15:56:45 +08:00
lenboo
2ff94a4362
[maven-release-plugin] prepare release 1.3.1
2020-07-03 15:55:56 +08:00
lenboo
fd76f172c1
Merge remote-tracking branch 'upstream/dev' into dev-merge
...
# Conflicts:
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
gaojun2048
58897aa2c5
merge 1.3.0-release branch code
2020-06-29 18:42:26 +08:00
gaojun2048
3cf9afc195
merge 1.3.0-release branch code
2020-06-29 18:42:18 +08:00
qiaozhanwei
7a9e1c4fbf
remove ExecuteTaskRequestCommand and ExecuteTaskResponseCommand
2020-06-29 17:38:11 +08:00
dailidong
038bde5efe
[Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms ( #2318 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
* fix cpu 100% bug
* simplify master、 worker、alert、dao properties
* add master and worker properties
* add master and worker properties
* add master and worker properties
* Optimize code , reduce cost time from 60ms to 0.5ms
* merge code form remote
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:38:04 +08:00
khadgarmage
416e0417ad
license and checkoutv2 fix ( #2261 )
2020-06-29 17:36:29 +08:00
gabry.wu
e1771de45a
Adapting partial code(file name start with R) to the sonar cloud rule ( #2243 )
2020-06-29 17:36:29 +08:00
tswstarplanet
bc65253fd1
fix blocker problem ( #2244 )
...
* fix blocker problem
* use error level to log exception
2020-06-29 17:36:29 +08:00
gabry.wu
7ac6ad5491
Adapting partial code(file name start with N) to the sonar cloud rule ( #2215 )
2020-06-29 17:36:28 +08:00
t1mon
55cc1c7c5b
Replace Charset.forName("UTF-8") with StandardCharsets.UTF_8. ( #2193 )
2020-06-29 17:36:28 +08:00
Rubik-W
bd43bed64d
fix: remove redundant import class ( #2110 )
2020-06-29 17:36:13 +08:00
gabry.wu
fdc9e0090b
remove incorrect or unnecessary URL tag ( #2099 )
2020-06-29 17:36:13 +08:00
zixi0825
9bf67d80d0
Revise annotation spelling errors & Enhanced code robustness ( #3042 )
...
* revise annotation spelling errors & enhanced code robustness
* revise annotation spelling errors & enhanced code robustness
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-28 10:48:14 +08:00
lenboo
b81872cacd
merge from dev-1.3.0
2020-06-19 18:52:30 +08:00
lenboo
714ae33e40
merge from dev
2020-06-19 14:20:24 +08:00
superScala
8031c1624a
add break & default logic for NettyDecoder.decode ( #2969 )
...
* add break & default logic for NettyDecoder.decode
* fix the bad code smells
Co-authored-by: gabry <wu_shao_jie@qq.com>
Co-authored-by: gabry.wu <8545796+gabrywu@users.noreply.github.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-16 14:25:40 +08:00
superScala
3a7fd725c7
first show: remove unnecessary code ( #2968 )
...
Co-authored-by: gabry <wu_shao_jie@qq.com>
2020-06-14 13:49:20 +08:00
simon824
24120a4641
fix the bad code smell
2020-06-12 11:27:42 +08:00
simon824
65841a5709
1. change windows new line 'CR' to linux new line 'LF'
...
2. solve conflict
2020-06-12 09:31:04 +08:00
simon824
f643981f48
1. change windows new line 'CR' to linux new line 'LF'
...
2. Format code
2020-06-11 10:14:15 +08:00
simon824
fad09075f9
remove fastjson maven dependence
2020-06-10 09:16:28 +08:00
simon824
6e907f150c
1. specified jackson Date format.
...
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer
rewrite createArrayNode,createObjectNode method will Affect too many lines ，i will fix it in the next pr.
2020-06-09 13:02:50 +08:00
dailidong
186ba3baab
fix oom when no master is active in dev-1.3.0 ( #2918 )
...
* fix worker group config no effect
* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2
* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0
# Conflicts:
# dolphinscheduler-server/src/main/resources/config/install_config.conf
* datasource config
* Update datasource.properties
* fix RunConfig bug
* remove param monitor server state
* fix table T_DS_ALERT
* update h2 database
* fix #2910 master server will show exception for some time when it restart
* fix oom when no master is active
* fix worker oom when master server restart
* fix oom
* fix
* add UT
* fix worker group config no effect
2020-06-08 15:55:39 +08:00
bao liang
422fe5136d
[license] add license and notice ( #2916 )
...
* fix bug: send task result to master until success.
* fix bug: send task result to master until success.
* add sleep
* set null if send result failed
* set null if send result failed
* change max length of username to 39
* test for gc
* change version to 1.3.0
* check and add some license files
* check and add some license files
* revert
Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-06 22:21:37 +08:00
simon
33bd6586dd
Using Jackson instead of Fastjson
2020-06-03 09:19:59 +08:00
qiaozhanwei
f4583d78e9
add UT ( #2717 )
...
* add LoggerServerTest UT
* add LoggerServerTest UT
* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-14 17:50:47 +08:00
qiaozhanwei
443a6d1935
delete log files while deleting process instances #2463 ( #2693 )
...
* script variable has "processDefinitionId" is error #2664
* blank in eamil and left font align #2648
* delete log files while deleting process instances #2463
* delete log files while deleting process instances #2463
* delete log files while deleting process instances #2463
* delete log files while deleting process instances #2463
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-12 16:54:29 +08:00
qiaozhanwei
e837a73fcd
EnterpriseWeChatUtils modify ( #2502 )
...
* dispatch task fail will set task status failed
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* #2486 bug fix
* host and workergroup compatible
* EnterpriseWeChatUtils modify
* EnterpriseWeChatUtils modify
* EnterpriseWeChatUtils modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-23 16:54:54 +08:00
qiaozhanwei
1f43925c12
remove ExecuteTaskRequestCommand and ExecuteTaskResponseCommand
2020-04-01 18:34:25 +08:00
qiaozhanwei
70d6360149
startup conflict reslove
2020-03-31 17:51:29 +08:00
qiaozhanwei
152c814126
test
2020-03-31 16:51:19 +08:00
qiaozhanwei
c2cf8fb7b1
test
2020-03-30 11:52:27 +08:00
dailidong
7e9f45e514
[Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms ( #2318 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
* fix cpu 100% bug
* simplify master、 worker、alert、dao properties
* add master and worker properties
* add master and worker properties
* add master and worker properties
* Optimize code , reduce cost time from 60ms to 0.5ms
* merge code form remote
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-27 09:53:46 +08:00
qiaozhanwei
03e29b9311
1，ZookeeperRegister use common.properties zookeeperRoot path 2，api start exclude org.apache.dolphinscheduler.server.* ( #2307 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency，MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1，get workergroup from zk modify
2，SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db，task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify
* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
* ZookeeperRegister use common.properties zookeeperRoot path
* api start exclude org.apache.dolphinscheduler.server.*
* ZookeeperRegister use common.properties zookeeperRoot path
* 1，api start load server filter
2，SHELL task exitStatusCode modify
* java doc error modify
* java doc error modify
* remove todo
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-25 18:11:49 +08:00
Tboy
92fd6479a9
Refactor worker ( #2279 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
2020-03-23 14:45:07 +08:00
khadgarmage
cddf803ba0
license and checkoutv2 fix ( #2261 )
2020-03-21 20:23:13 +08:00
gabry.wu
4318e860d5
Adapting partial code(file name start with R) to the sonar cloud rule ( #2243 )
2020-03-20 18:46:25 +08:00
tswstarplanet
f4897b563d
fix blocker problem ( #2244 )
...
* fix blocker problem
* use error level to log exception
2020-03-19 23:56:26 +08:00
dailidong
ce2853ad4c
add master and worker properties and remove data quality config ( #2211 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-19 00:07:36 +08:00
gabry.wu
0e5fb86bef
Adapting partial code(file name start with N) to the sonar cloud rule ( #2215 )
2020-03-18 17:02:57 +08:00
t1mon
f6f29f4abb
Replace Charset.forName("UTF-8") with StandardCharsets.UTF_8. ( #2193 )
2020-03-16 17:10:55 +08:00
gabry.wu
1658176ef5
Adapting partial code(file name start with P #1 ) to the sonar cloud rule ( #2171 ) ( #2187 )
2020-03-15 21:25:52 +08:00
t1mon
16eeaf50b4
Fix javadoc and DataxTaskTest UT error ( #2147 )
...
* Fix javadoc and DataxTaskTest UT error
* Fix UT error
* LoggerServiceTest UT error
* WorkerGroupServiceTest UT error
Modify Host.of()
* Add address NPE check
* Trigger actions rerun
* Try to solve maven VM crash
* Try to solve maven VM crash 1
* Try to solve maven VM crash 2
* Try to solve maven VM crash 3
2020-03-12 17:41:32 +08:00
Rubik-W
275df85250
fix: remove redundant import class ( #2110 )
2020-03-08 16:00:47 +08:00
gabry.wu
0e1dd8d7d9
remove incorrect or unnecessary URL tag ( #2099 )
2020-03-07 18:07:53 +08:00
qiaozhanwei
d5c795bd45
fix ugly logger usage ( #2063 )
...
* fix ugly logger usage
* remove space
2020-03-05 11:10:48 +08:00
qiaozhanwei
3e5a939913
delete unused import ( #2066 )
2020-03-05 11:07:44 +08:00
Tboy
71c11bd594
refactor kill logic ( #2060 )
2020-03-02 22:15:08 +08:00
qiaozhanwei
bb3885cfe2
master add kill task logic ( #2058 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
2020-03-02 21:30:04 +08:00
Tboy
3bad56ca15
Refactor worker ( #2042 )
...
* Refactor worker (#10 )
* Refactor worker (#2000 )
* Refactor worker (#2 )
* Refactor worker (#1993 )
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* TaskExecutionContext create modify (#1994 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
* add- register processor
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* buildAckCommand taskInstanceId not set modify (#2002 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify (#2004 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment (#2006 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type (#2012 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* Refactor worker (#2018 )
* Refactor worker (#7 )
* Refactor worker (#2000 )
* Refactor worker (#2 )
* Refactor worker (#1993 )
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* TaskExecutionContext create modify (#1994 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
* add- register processor
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* buildAckCommand taskInstanceId not set modify (#2002 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify (#2004 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment (#2006 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type (#2012 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* Refactor worker (#8 )
* Refactor worker (#2000 )
* Refactor worker (#2 )
* Refactor worker (#1993 )
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* TaskExecutionContext create modify (#1994 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
* add- register processor
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* buildAckCommand taskInstanceId not set modify (#2002 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify (#2004 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment (#2006 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type (#2012 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* add kill command
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* add TaskInstanceCacheManager receive Worker report result，modify master polling db transfrom to cache (#2021 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* refactor heartbeat logic
* update registry and add worker group
* add worker group
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-03-01 11:38:37 +08:00
qiaozhanwei
7f94122f6e
1，worker TaskPros use TaskExecutionContext replase TaskPros 2，Master kill Task , KillTaskProcessor modify ( #2039 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
2020-02-28 17:03:36 +08:00
qiaozhanwei
2bb5ebaf3a
add TaskInstanceCacheManager receive Worker report result，modify master polling db transfrom to cache ( #2021 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
2020-02-26 18:46:05 +08:00
Tboy
9dc9745a80
Refactor worker ( #2018 )
...
* Refactor worker (#7 )
* Refactor worker (#2000 )
* Refactor worker (#2 )
* Refactor worker (#1993 )
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* TaskExecutionContext create modify (#1994 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
* add- register processor
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* buildAckCommand taskInstanceId not set modify (#2002 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify (#2004 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment (#2006 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type (#2012 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* Refactor worker (#8 )
* Refactor worker (#2000 )
* Refactor worker (#2 )
* Refactor worker (#1993 )
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* TaskExecutionContext create modify (#1994 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
* add- register processor
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* buildAckCommand taskInstanceId not set modify (#2002 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify (#2004 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment (#2006 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type (#2012 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* add kill command
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-02-26 09:57:21 +08:00
qiaozhanwei
6053f9b2df
add comment ( #2006 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
2020-02-24 11:40:24 +08:00
Tboy
e4679109af
Refactor worker ( #1997 )
...
* Refactor worker (#2 )
* Refactor worker (#1993 )
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* TaskExecutionContext create modify (#1994 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-02-22 21:39:48 +08:00
Tboy
3800a2a99b
Refactor worker ( #1993 )
...
* Refactor worker (#1 )
* add TaskResponseProcessor (#1983 )
* 1, master persistent task 2. extract master and worker communication model (#1992 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* updates
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-02-21 22:02:18 +08:00
qiaozhanwei
ff86dc7d57
1, master persistent task 2. extract master and worker communication model ( #1992 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
2020-02-21 20:18:02 +08:00
qiaozhanwei
2bb2d0ce7c
master/worker basic communication
2020-02-20 14:55:53 +08:00
qiaozhanwei
c39e511262
master/worker basic communication
2020-02-20 14:04:14 +08:00
qiaozhanwei
910401fb2d
master/worker basic communication
2020-02-20 11:51:51 +08:00
qiaozhanwei
e7c32affc6
master/worker basic communication
2020-02-20 09:29:27 +08:00
Tboy
ac93d78379
Refactor worker ( #1979 )
...
* updates
* move FetchTaskThread logic to WorkerNettyRequestProcessor
* add NettyRemotingClient to scheduler thread
* refactor TaskScheduleThread, add TaskInstanceCallbackService
2020-02-19 21:57:16 +08:00
Tboy
f9500c58b1
Refactor worker ( #1975 )
...
* updates
* move FetchTaskThread logic to WorkerNettyRequestProcessor
* add NettyRemotingClient to scheduler thread
2020-02-19 10:11:41 +08:00
Tboy
5cc2ef5ce6
add scanFutureTable and testcase ( #1971 )
2020-02-17 22:07:19 +08:00
qiaozhanwei
9b75bebf1b
1，add sendAsync method 2，refactor LoggerClient ( #1965 )
...
* 1，remove dolphinscheduler-rpc module 2，add dolphinscheduler-remote module 3，add dolphinscheduler-service module 4，refactor LoggerServer module (#1925 )
* 1，remove dolphinscheduler-rpc module
2，add dolphinscheduler-remote module
3，add dolphinscheduler-service module
4，refactor LoggerServer module
* ProcessUtils modify
* Refactor architecture (#1926 )
* move version to parent pom
* move version properties to parent pom for easy management
* remove freemarker dependency
* delete CombinedApplicationServer
* #1871 correct spelling
* #1873 some updates for TaskQueueZkImpl
* #1875 remove unused properties in pom
* #1878
1. remove tomcat dependency
2. remove combined_logback.xml in api module
3. format pom.xml for not aligning
* #1885 fix api server startup failure
1. add jsp-2.1 dependency
2. remove jasper-runtime dependency
* add stringutils ut (#1921 )
* add stringutils ut
* Newfeature for #1675 . (#1908 )
Continue to finish the rest works, add the cache feature for dependence,mr,python,sub_process,procedure and shell.
* Add modify user name for process definition (#1919 )
* class overrides equals() and should therefore also override hashCode()
* #1862 add modify user in process difinition list
* #1862 add pg-1.2.2 ddl.sql
* modify ScriptRunnerTest
* add updateProessDifinition UT
* modify updateProcessDifinition UT
* modify updateProcessDifinition UT
* modify mysql 1.2.2 ddl.sql&dml.sql
* add scope test to mysql in pom
* modify pg-1.2.2 ddl.sql
* refactor module
* updates
Co-authored-by: khadgarmage <khadgar.mage@outlook.com>
Co-authored-by: zhukai <boness@qq.com>
Co-authored-by: Yelli <amarantine@my.com>
* dolphinscheduler-common remove spring (#1931 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* SpringApplicationContext class title add license (#1932 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* add license (#1934 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* move datasource classes to dao module
* fix send4LetterWord bug
* Refactor architecture (#1936 )
* move datasource classes to dao module
* fix send4LetterWord bug
* exclude jasper-compiler in case of runtime conflict
* LoggerServiceTest remove ProcessDao (#1944 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* LoggerServiceTest remove ProcessDao
* exclude jasper-compiler in case of runtime conflict (#1938 )
* move datasource classes to dao module
* fix send4LetterWord bug
* exclude jasper-compiler in case of runtime conflict
* DataAnaylysisServiceTest and ProcessDefinitionService modify
* remote module add comment
* OSUtilsTest modify
* add finally block to close channel
* add finally block to close channel (#1951 )
* move datasource classes to dao module
* fix send4LetterWord bug
* exclude jasper-compiler in case of runtime conflict
* add finally block to close channel
* refactor log client service
* add sendSync method
* 1，quartz.properties add conf category
2，dolphinscheduler-daemon.sh modify
* dolphinscheduler-binary.xml modify
* add sendAsync method (#1962 )
* add sendAsync method
* add sendAsync method
* 1，add sendAsync method
2，refactor LoggerClient
* LogClientService modify
Co-authored-by: Tboy <technoboy@yeah.net>
Co-authored-by: khadgarmage <khadgar.mage@outlook.com>
Co-authored-by: zhukai <boness@qq.com>
Co-authored-by: Yelli <amarantine@my.com>
2020-02-16 19:12:01 +08:00
qiaozhanwei
99b9c10d78
1，add sendAsync method
...
2，refactor LoggerClient
2020-02-16 18:12:44 +08:00
qiaozhanwei
cfa882c437
Merge remote-tracking branch 'remotes/upstream/refactor-architecture' into dev
2020-02-16 16:53:14 +08:00
Tboy
ac932ba2cf
add sendAsync method ( #1962 )
...
* add sendAsync method
* add sendAsync method
2020-02-16 11:53:29 +08:00
qiaozhanwei
b2e0b8598f
# Conflicts resolve :
...
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/NettyRemotingClient.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/ExecuteTaskRequestCommand.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/ExecuteTaskResponseCommand.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/Ping.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/log/GetLogBytesRequestCommand.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/log/RollViewLogRequestCommand.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/log/ViewLogRequestCommand.java
# dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/handler/NettyClientHandler.java
2020-02-16 11:24:41 +08:00
qiaozhanwei
c1ee133392
Refactor Architecture Basic modification #1658 ( #1946 )
...
* 1，remove dolphinscheduler-rpc module 2，add dolphinscheduler-remote module 3，add dolphinscheduler-service module 4，refactor LoggerServer module (#1925 )
* 1，remove dolphinscheduler-rpc module
2，add dolphinscheduler-remote module
3，add dolphinscheduler-service module
4，refactor LoggerServer module
* ProcessUtils modify
* Refactor architecture (#1926 )
* move version to parent pom
* move version properties to parent pom for easy management
* remove freemarker dependency
* delete CombinedApplicationServer
* #1871 correct spelling
* #1873 some updates for TaskQueueZkImpl
* #1875 remove unused properties in pom
* #1878
1. remove tomcat dependency
2. remove combined_logback.xml in api module
3. format pom.xml for not aligning
* #1885 fix api server startup failure
1. add jsp-2.1 dependency
2. remove jasper-runtime dependency
* add stringutils ut (#1921 )
* add stringutils ut
* Newfeature for #1675 . (#1908 )
Continue to finish the rest works, add the cache feature for dependence,mr,python,sub_process,procedure and shell.
* Add modify user name for process definition (#1919 )
* class overrides equals() and should therefore also override hashCode()
* #1862 add modify user in process difinition list
* #1862 add pg-1.2.2 ddl.sql
* modify ScriptRunnerTest
* add updateProessDifinition UT
* modify updateProcessDifinition UT
* modify updateProcessDifinition UT
* modify mysql 1.2.2 ddl.sql&dml.sql
* add scope test to mysql in pom
* modify pg-1.2.2 ddl.sql
* refactor module
* updates
Co-authored-by: khadgarmage <khadgar.mage@outlook.com>
Co-authored-by: zhukai <boness@qq.com>
Co-authored-by: Yelli <amarantine@my.com>
* dolphinscheduler-common remove spring (#1931 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* SpringApplicationContext class title add license (#1932 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* add license (#1934 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* Refactor architecture (#1936 )
* move datasource classes to dao module
* fix send4LetterWord bug
* LoggerServiceTest remove ProcessDao (#1944 )
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* dolphinscheduler-common remove spring
* LoggerServiceTest remove ProcessDao
* exclude jasper-compiler in case of runtime conflict (#1938 )
* move datasource classes to dao module
* fix send4LetterWord bug
* exclude jasper-compiler in case of runtime conflict
* DataAnaylysisServiceTest and ProcessDefinitionService modify
* remote module add comment
* OSUtilsTest modify
* add finally block to close channel (#1951 )
* move datasource classes to dao module
* fix send4LetterWord bug
* exclude jasper-compiler in case of runtime conflict
* add finally block to close channel
Co-authored-by: Tboy <technoboy@yeah.net>
Co-authored-by: khadgarmage <khadgar.mage@outlook.com>
Co-authored-by: zhukai <boness@qq.com>
Co-authored-by: Yelli <amarantine@my.com>
2020-02-14 22:00:20 +08:00
Technoboy-
820b84bb46
add sendSync method
2020-02-14 21:46:25 +08:00
Technoboy-
8e154ecbe4
refactor log client service
2020-02-13 22:06:34 +08:00
qiaozhanwei
1ac1d582e6
remote module add comment
2020-02-13 15:55:10 +08:00
qiaozhanwei
9a70e77955
1，remove dolphinscheduler-rpc module 2，add dolphinscheduler-remote module 3，add dolphinscheduler-service module 4，refactor LoggerServer module ( #1925 )
...
* 1，remove dolphinscheduler-rpc module
2，add dolphinscheduler-remote module
3，add dolphinscheduler-service module
4，refactor LoggerServer module
* ProcessUtils modify
2020-02-10 18:01:56 +08:00
