Commit Graph

394 Commits

Author SHA1 Message Date
guyinyou
3b581455fc
[Feature]Add Python task "task variable / result transfer" implementation (#3659)
* 增加Python Task的“任务变量/结果传递”实现
Signed-off-by: 古崟佑

* add two files license
Signed-off-by: 古崟佑

* fix 'server/worker/task/AbstractCommandExecutor.java' code style
Signed-off-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>

* update DB
Signed-off-by: 古崟佑

* update DB -- 2
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codestyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* add VarPoolUtils Test
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* add test config for VarPoolUtilsTest
Signed-off-by: 古崟佑

* fix unit test
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest.java
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* change the test class path
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* fix "print the error message"
Signed-off-by: 古崟佑

* fix bug
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

Co-authored-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>
2020-09-25 13:20:42 +08:00
CalvinKirs
db663a13a3
[Improvement][remote]load balance warm up (#3770)
* [Improvement][remote]load balance  warm up

* reformat code

* reformat code

* code smell

* code smell

* add test

* add test

* add test

* add test

* fix bug

* fix bug

* add docs

* add host test

* add host test

* add host test

* add docs

* code reformat

* code reformat
2020-09-22 17:39:39 +08:00
zhuangchong
36d5349610
[fix-3745][server] server get tasktype NPE exception (#3746) 2020-09-17 13:58:43 +08:00
zhuangchong
fad2852768
[Improvement][server] WATERDROP task plug-in optimization in switch case code cleaning. (#3652)
* WATERDROP switch case code checkstyle.

* add TaskManagerTest.

* TaskManagerTest checkstyle.

* TaskManagerTest checkstyle.

* TaskManagerTest add pom maven-surefire

* TaskManagerTest update.
2020-09-14 15:59:56 +08:00
felix.wang
4ed3638750
[Bug][server] fix 3629 pull request (#3706)
* fix 3629 pr

* fix e2e build
2020-09-09 22:09:49 +08:00
Hsu Pu
68bc58892b
[Improvement][server] Refactor code for getTaskLogPath to reduce NPE log during UT (#3629)
* init

* update

* update

* update

* fix

* fix

* enable test

* fix

* fix
2020-09-07 10:39:40 +08:00
Hsu Pu
feec629554
[Improvement][server] unit test for SubProcessTask with PowerMockRunner (#3632) 2020-09-03 21:27:29 +08:00
赤月
69201ac346
[Improvement][ALL] Remove some un-used code and code review #3639 (#3640)
* [Improvement][ALL] Remove some un-used code and code review #3639

* [Improvement][ALL] Remove some un-used code and code review #3639

* [Improvement][ALL] Remove some un-used code and code review #3639

* [Improvement][ALL] Remove some un-used code and code review

* [Improvement][ALL] Remove some un-used code and code review

* [Improvement][ALL] Remove some un-used code and code review
2020-09-03 18:56:32 +08:00
CalvinKirs
632d52cd69
[refactor][server-test] refactor RoundRobinHostManagerTest (#3511)
* [refactor][server-test] refactor RoundRobinHostManagerTest

* add RoundRobinHostManagerTest

* format code

* format code
2020-08-31 20:35:45 +08:00
zhuangchong
efee53bec9
[Improvement][server] server model ProcessUtils.java code cleaning. (#3382)
* server model ProcessUtils.java code cleaning.

* add checkstyle,server model ProcessUtils.java code cleaning.

* ProcessUtils.java checkstyle.

* update.

* update.

Co-authored-by: unknown <zhuangchong-phq@sinosig.com>
2020-08-25 11:13:45 +08:00
felix.wang
2ba529a96c
[Improvement][alert] Optimize alarm information (#3559)
* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* Optimize alarm information

* fix code style

* add license

* fix code smell

* Update dolphinscheduler-alert/src/main/java/org/apache/dolphinscheduler/alert/utils/EnterpriseWeChatUtils.java

Co-authored-by: Yichao Yang <1048262223@qq.com>

* fix code smell

* fix code smell

* 修改ut 和replace问题。

* 修改ut

Co-authored-by: wangjianda <Felix@thinkingdata.com>
Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-08-25 11:10:44 +08:00
Yichao Yang
8fea8f913c
[Test][server] Add the master server test case (#3574) 2020-08-25 10:54:39 +08:00
felix.wang
2f01025802
[Fix-3299][dao&server] Fix 3299,when Json string parsing problem caused by non-standard json format. (#3552)
* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

Co-authored-by: wangjianda <Felix@thinkingdata.com>
2020-08-19 16:27:39 +08:00
JinyLeeChina
d30bc7bcf3
'3549' (#3550) 2020-08-19 16:24:47 +08:00
CalvinKirs
a1ce54b8e6
[Improvement][server] Fix some sonar analyzed bugs (#3415)
* [FIX_BUG][*]fix sonar bugs
fix assert error(assertions always pass)

* fix assert error(assertions always pass)
insert data should judging the success status

* fix assert error(assertions always pass)

* use AtomicReference<Object> to replace unsafe volatile Object

* use AtomicReference<Object> to replace unsafe volatile Object

* lazy initialization holder class Singleton...

* reformat code
judge database operations

* reformat code
judge database operations

* revert

* use ThreadLocalRandom to build random numbers

* reform code

* fix dao test error

* judge database operations
2020-08-19 15:15:28 +08:00
vanilla111
a4ee351a3a
delay execution of tasks and improve some designs (#3427) 2020-08-19 11:36:30 +08:00
Hsu Pu
59610a5661
[Improvement][dao,server] unit test for ConditionsTask (#3385) 2020-08-18 14:46:00 +08:00
XiaotaoYi
0b7b6d4e2a
[bug-3480][server]fix ds muti-level directory in zk, which lead to fa… (#3532)
* [bug-3480][server]fix ds muti-level directory in zk, which lead to fail to assign work

* miss whitespace for if statement
2020-08-18 07:09:39 +08:00
zixi0825
a7a1156ff1
fix zk path error (#3470) 2020-08-12 17:16:43 +08:00
CalvinKirs
d6a32ac652
[Feature#3234][cluster]enhanced load balancing (#3235)
* [Future#3234][cluster]enhanced load balancing
weight-based load balancing algorithm
this close # 3234

* remove useless parameter

* code smell

* load balancing according to work group

* add smooth weight round robin

* remove unused constants

* perfect unit test

* code smell

* code smell

* add work weight config

* fix config error

* add weight docs to readme.md
2020-08-10 19:05:21 +08:00
Yichao Yang
f419fe6239
[Test][server] Add LoggerServerTest (#3405) 2020-08-10 15:01:20 +08:00
zixi0825
40a21816ab
[Fix] [Server] Fix the spell error and add log in SqlTask (#3243)
* fix the spell and modify logger print format
2020-08-07 13:58:14 +08:00
CalvinKirs
6ecc95a338
[FIX_BUG][*] fix some bugs from sonar analysis (#3322)
* [FIX_BUG][*] fix some bugs from sonar analysis

* remove invalid single test

* code smell

* remove exception catches that never happen

* delete invalid exception catch

* remove excess locks

* fix e2e test error

* recovery related exception removal
2020-08-04 02:23:50 +00:00
dailidong
6d903e266f
[Feature][server] add embeded zookeeper server in dolphinscheduler-daemon.sh (#3293)
* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

* simplify common config

* add worker group config

* add worker group config

* add worker group config

* this command just for getting a quick experience,not recommended for production. this operation will start a standalone zookeeper server

* Merge remote-tracking branch 'upstream/dev' into dev

* add logback for zookeeper

* Update logback-zookeeper.xml

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-07-31 17:37:02 +08:00
xsbai
87eb7c8211
adds waterdrop plugin (#3073)
* add plugin waterdrop

* add plugin waterdrop

* add plugin waterdrop

* fix deploy mode error

* Update en_US.js

fix conflict

* Update en_US.js

fix conflict

* Update en_US.js

Add missing commas

* Update zh_CN.js

Add missing commas

* Update waterdrop.vue

Fix the no resource hint

* Update zh_CN.js

Add 'Please select the waterdrop resources'

* Update en_US.js

Add  'Please select the waterdrop resources'

* Update waterdrop.vue

fix bug when click the waterdrop node again, it still show the `client` mode in deploy mode

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 23:20:59 +08:00
zixi0825
5fe31546ba
[Fix][all] fix spell error (#3289)
* fix spell error

* test unit test

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-30 18:33:16 +08:00
chendapao
afd9139c3f
[Feature]use zk own del func (#3310)
* use zk own del func

* Update remove-zk-node.sh

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 18:26:27 +08:00
CalvinKirs
3a0d4da27e
[Test][server] Fix Test ContextConfiguration error. (#3304)
* [FIX_BUG]fix test error

* small change

* add test

* reformat

* fix test no qualifying bean
2020-07-30 13:28:27 +08:00
zixi0825
78856fb08e
remove the duplicate function in MasterExecThread (#3329)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-29 19:15:03 +08:00
lenboo
82295eea52 fix bug#1336 set start/end time before complement the next process 2020-07-28 17:57:49 +08:00
lenboo
39182568b2 fix bug#1336 serial complement data can have multiple process instances 2020-07-27 15:43:08 +08:00
Yichao Yang
0577dc9780
[Test-3288][server] Fix github ci unit test oom (#3305)
* [Improvement] Test github action oom

* Optimize the test case performance

* Remove the unused import
2020-07-25 19:24:16 +08:00
tswstarplanet
5b7efd2d33
[Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client (#3244)
* [Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client

* autowire CuratorZookeeperClient to ZookeeperOperator

* [Improvement] Add the issue specifications reference (#3221)

* log zookeeper address when the connect state change

* resume the operation of add connect state listener in MasterRegistry

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-23 11:02:30 +08:00
LEI SHENG
6f2667bf1a
[Fix-3240][worker] Cannot find a (Map) Key deserializer for type (#3245)
* fix the bug #2923 (Hive JDBC connection parameter ignored) , the hive jdbc connection url ;

* fix the bug index out of bound and add some test case

* add the Licensed

* fix the checkstyle and import StringBuilder

* put HiveDataSourceTest.java to root pom maven-surefire-plugin to avoiding the sonar 0.0% Coverage.

* Update HiveDataSource.java

* fix the bug #3240  [worker] Cannot find a (Map) Key deserializer for type

* [DS-3245][fix]  change toString  use JsonString

- change the method of  UdfFunc.toString use JsonString

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the junit test for full

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the sonarcloud merge error, Swap these 2 arguments so they are in the correct order: expected value, actual value.

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

move the class UdfFuncDeserializer.java to UdfFunc.java

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

add blank line

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the unit test of UdfFunc.java

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

remove the UdfFuncTest.java from maven-surefire-plugin list

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the UdfFuncTest to the root pom maven-surefire-plugin

This closes #3245

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-21 18:54:07 +08:00
amenglu
178d8f829a
Update MasterSchedulerService.java (#3226) 2020-07-17 16:54:52 +08:00
hcz
50fcc492c1
Fix typo (#3230) 2020-07-17 14:26:46 +08:00
Yichao Yang
b4af3fd176
[Feature-2815][server] One worker can belong to different workergroups (#2934)
* [Feature-2815][server] One worker can belong to different workergroups

* Feature: Add test cases

* Update MonitorService.java

* Update MonitorService.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 17:25:13 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
gabry.wu
75259dcc07
simply doExecute using RetryerUtils.retryCall (#2761) 2020-07-13 14:38:40 +08:00
tswstarplanet
88f9bed726
[Bug-3187]close Heartbeat thread pool when MasterRegistry unRegistry (#3188)
* [Bug-3187]close Heartbeat thread pool when MasterRegistry unRegistry

* use shutdownNow; delete redundant code
2020-07-13 14:27:38 +08:00
CalvinKirs
933a67db8d
[Improvement][Server]Code Clean (#3196)
* [*][Server]Code Clean
redundant code removal

* redundant code removal

* redundant code removal
2020-07-12 22:25:25 +08:00
muzhongjiang
c49626b877
Word spelling modification,Comment modification,Log level modification (#3180)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

* Word spelling modification
Comment modification
Word spelling modification,Comment modification,Log level modification

* Update EmailManager.java

* Update FlinkParameters.java

* Update SqlTask.java

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-12 22:20:14 +08:00
iceblue510
1f8851b906
modify sqlTask (#3152) 2020-07-06 17:57:30 +08:00
CalvinKirs
d1282fdd96
In the case of multiple network cards, a valid host is returned (#2924)
* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix lock error

* fix lock error

* end

* end

* end

* end

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* lowering

* messy

* small change

* compile error

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
2020-07-02 19:16:37 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
Yichao Yang
38e485373d
[Feature-2925][server] Init TaskLogger in TaskExecuteProcessor (#2925) (#2965)
* [Feature-2925][common] Add exitVal judge in OSUtils.exeCmd (#2925)

* optimize the logger utils
2020-06-28 17:43:13 +08:00
zixi0825
9bf67d80d0
Revise annotation spelling errors & Enhanced code robustness (#3042)
* revise annotation spelling errors & enhanced code robustness

* revise annotation spelling errors & enhanced code robustness

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-28 10:48:14 +08:00
CalvinKirs
8213da50d3
[WIP] load balance #3054 (#3057)
* Load balancing abstract

* code
smell

* lower weight select
2020-06-28 10:20:59 +08:00
CalvinKirs
ad28c65160
[BUG_FIX]fix multi-threaded tests error. (#3044)
* fix  test mock error

* fix  test mock error
2020-06-23 22:35:12 +08:00
lenboo
ea1c653e08 merge from dev 2020-06-22 10:21:15 +08:00