Commit Graph

398 Commits

Author SHA1 Message Date
zhuangchong
169a4ace77
[Feature-3749][Alert-SPI] SqlTask should send notifications by alert server api (#4080)
* add sqltask send sync alert server.

* update alert-sms license.

* update AlertServer test.

* remote EmailAlertPluginTest.

* update sqltask.

* update test class.
2020-12-10 22:41:11 +08:00
gaojun2048
1d254347df [Feature-3189][alert,spi,dao,plugin-api] base code of dolphinscheduler spi and alert plugin implement (#3601)
* DS SPI

* Add DolphinScheduler SPI , and rebuilt the code of the Alert plug-in based on SPI

* Add DolphinScheduler SPI , and rebuilt the code of the Alert plug-in based on SPI

* add TODO

* delete

* compile

* spi commit

* Plugin Alert

* fix some bug

* add todo

* change web ui from alpacajs to form-create

* remove module

* add plugin schema

* add license header

* update alert and spi module version

* update the alert plugin sub module version

* comment the maven.local.repository param

* move utils from spi to common module

* add license header

* add license header and delete some chinese comment

* update spi packages

* delete no use alert_xx.properties

* update mysql.connector.version back to 5.1.34

* delete no use comment in pom.xml

* update email stmp password

* add license

* add semicolon to sql/upgrade/1.4.0_schema/mysql/dolphinscheduler_ddl.sql file

* format the code style

* format new clase file with checkstyle

* update plugin params to Builder model

* move JSONUtils to SPI because plugin can not dependency common module

* move JSONUtils to SPI because plugin can not dependency common module

* delete collection dependency

* replace PluginParamsTransfer to spi PluginParamsTransfer

* update dolphinscheduler-maven-plugin to 1.0.0

* update license

* update apache-rat-plugin add exclude '.iml' file

* check license

* ArtifactResolver only use in development and configPlugins is not empty

* ArtifactResolver only use in development and configPlugins is not empty

* ArtifactResolver only use in development and configPlugins is not empty

* default datasource should be postgresql

* add license files

* add license files

* postgresql port should be 5432

* postgresql test

* mv show_type to spi

add license header to AlertConstants

* check style fix

* copy check style file from branch dev

* alert show_type set by plugin

* alert show_type set by plugin

* add PluginDefineMapper to dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/registry/DependencyConfig.java

* add Bean to TaskCallbackServiceTestConfig

* add Bean to TaskCallbackServiceTestConfig

* fix check style

* check style fix

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* fix check style

* [Feature-3682][ui]Add form-create plug-in and alarm group management add sample demo (#3683)

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* check style fix

* rollback test change

* rollback test change

* rollback dao pom change

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* add ut to pom.xml

* add upgrade schema to global schema

* fix ut failed

* fix ut failed

* fix ut failed

* fix ut failed

* add test EmailAlertPluginTest to pom.xml

* fix ut failed

* fix ut failed

* fix check style

* update license header to presto license header

* presto license header not check

* fix ut coverage

* fix ut coverage

* fix ut

* fix ut

* fix ut

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

Co-authored-by: break60 <790061044@qq.com>
2020-10-12 15:28:11 +08:00
bao liang
39411ce03b
[FIX-3617][Service]fix 2 tasks instance are generated when fault tolerance (#3873)
* fix 2 tasks instance are generated when fault tolerance

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update ut

* update code style

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-10-10 15:05:56 +08:00
CalvinKirs
11d3c2cdaf
[Improvement][server]lower weight round robin support weight (#3768)
* [Improvement][server]lower weight round robin  support weight

* delete  code

* resolve conflicts
2020-09-28 09:56:46 +08:00
guyinyou
3b581455fc
[Feature]Add Python task "task variable / result transfer" implementation (#3659)
* 增加Python Task的“任务变量/结果传递”实现
Signed-off-by: 古崟佑

* add two files license
Signed-off-by: 古崟佑

* fix 'server/worker/task/AbstractCommandExecutor.java' code style
Signed-off-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>

* update DB
Signed-off-by: 古崟佑

* update DB -- 2
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codestyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* add VarPoolUtils Test
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* add test config for VarPoolUtilsTest
Signed-off-by: 古崟佑

* fix unit test
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest.java
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* change the test class path
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* fix "print the error message"
Signed-off-by: 古崟佑

* fix bug
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

Co-authored-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>
2020-09-25 13:20:42 +08:00
CalvinKirs
db663a13a3
[Improvement][remote]load balance warm up (#3770)
* [Improvement][remote]load balance  warm up

* reformat code

* reformat code

* code smell

* code smell

* add test

* add test

* add test

* add test

* fix bug

* fix bug

* add docs

* add host test

* add host test

* add host test

* add docs

* code reformat

* code reformat
2020-09-22 17:39:39 +08:00
zhuangchong
36d5349610
[fix-3745][server] server get tasktype NPE exception (#3746) 2020-09-17 13:58:43 +08:00
zhuangchong
fad2852768
[Improvement][server] WATERDROP task plug-in optimization in switch case code cleaning. (#3652)
* WATERDROP switch case code checkstyle.

* add TaskManagerTest.

* TaskManagerTest checkstyle.

* TaskManagerTest checkstyle.

* TaskManagerTest add pom maven-surefire

* TaskManagerTest update.
2020-09-14 15:59:56 +08:00
felix.wang
4ed3638750
[Bug][server] fix 3629 pull request (#3706)
* fix 3629 pr

* fix e2e build
2020-09-09 22:09:49 +08:00
Hsu Pu
68bc58892b
[Improvement][server] Refactor code for getTaskLogPath to reduce NPE log during UT (#3629)
* init

* update

* update

* update

* fix

* fix

* enable test

* fix

* fix
2020-09-07 10:39:40 +08:00
Hsu Pu
feec629554
[Improvement][server] unit test for SubProcessTask with PowerMockRunner (#3632) 2020-09-03 21:27:29 +08:00
赤月
69201ac346
[Improvement][ALL] Remove some un-used code and code review #3639 (#3640)
* [Improvement][ALL] Remove some un-used code and code review #3639

* [Improvement][ALL] Remove some un-used code and code review #3639

* [Improvement][ALL] Remove some un-used code and code review #3639

* [Improvement][ALL] Remove some un-used code and code review

* [Improvement][ALL] Remove some un-used code and code review

* [Improvement][ALL] Remove some un-used code and code review
2020-09-03 18:56:32 +08:00
CalvinKirs
632d52cd69
[refactor][server-test] refactor RoundRobinHostManagerTest (#3511)
* [refactor][server-test] refactor RoundRobinHostManagerTest

* add RoundRobinHostManagerTest

* format code

* format code
2020-08-31 20:35:45 +08:00
zhuangchong
efee53bec9
[Improvement][server] server model ProcessUtils.java code cleaning. (#3382)
* server model ProcessUtils.java code cleaning.

* add checkstyle,server model ProcessUtils.java code cleaning.

* ProcessUtils.java checkstyle.

* update.

* update.

Co-authored-by: unknown <zhuangchong-phq@sinosig.com>
2020-08-25 11:13:45 +08:00
felix.wang
2ba529a96c
[Improvement][alert] Optimize alarm information (#3559)
* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* Optimize alarm information

* fix code style

* add license

* fix code smell

* Update dolphinscheduler-alert/src/main/java/org/apache/dolphinscheduler/alert/utils/EnterpriseWeChatUtils.java

Co-authored-by: Yichao Yang <1048262223@qq.com>

* fix code smell

* fix code smell

* 修改ut 和replace问题。

* 修改ut

Co-authored-by: wangjianda <Felix@thinkingdata.com>
Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-08-25 11:10:44 +08:00
Yichao Yang
8fea8f913c
[Test][server] Add the master server test case (#3574) 2020-08-25 10:54:39 +08:00
felix.wang
2f01025802
[Fix-3299][dao&server] Fix 3299,when Json string parsing problem caused by non-standard json format. (#3552)
* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

Co-authored-by: wangjianda <Felix@thinkingdata.com>
2020-08-19 16:27:39 +08:00
JinyLeeChina
d30bc7bcf3
'3549' (#3550) 2020-08-19 16:24:47 +08:00
CalvinKirs
a1ce54b8e6
[Improvement][server] Fix some sonar analyzed bugs (#3415)
* [FIX_BUG][*]fix sonar bugs
fix assert error(assertions always pass)

* fix assert error(assertions always pass)
insert data should judging the success status

* fix assert error(assertions always pass)

* use AtomicReference<Object> to replace unsafe volatile Object

* use AtomicReference<Object> to replace unsafe volatile Object

* lazy initialization holder class Singleton...

* reformat code
judge database operations

* reformat code
judge database operations

* revert

* use ThreadLocalRandom to build random numbers

* reform code

* fix dao test error

* judge database operations
2020-08-19 15:15:28 +08:00
vanilla111
a4ee351a3a
delay execution of tasks and improve some designs (#3427) 2020-08-19 11:36:30 +08:00
Hsu Pu
59610a5661
[Improvement][dao,server] unit test for ConditionsTask (#3385) 2020-08-18 14:46:00 +08:00
XiaotaoYi
0b7b6d4e2a
[bug-3480][server]fix ds muti-level directory in zk, which lead to fa… (#3532)
* [bug-3480][server]fix ds muti-level directory in zk, which lead to fail to assign work

* miss whitespace for if statement
2020-08-18 07:09:39 +08:00
zixi0825
a7a1156ff1
fix zk path error (#3470) 2020-08-12 17:16:43 +08:00
CalvinKirs
d6a32ac652
[Feature#3234][cluster]enhanced load balancing (#3235)
* [Future#3234][cluster]enhanced load balancing
weight-based load balancing algorithm
this close # 3234

* remove useless parameter

* code smell

* load balancing according to work group

* add smooth weight round robin

* remove unused constants

* perfect unit test

* code smell

* code smell

* add work weight config

* fix config error

* add weight docs to readme.md
2020-08-10 19:05:21 +08:00
Yichao Yang
f419fe6239
[Test][server] Add LoggerServerTest (#3405) 2020-08-10 15:01:20 +08:00
zixi0825
40a21816ab
[Fix] [Server] Fix the spell error and add log in SqlTask (#3243)
* fix the spell and modify logger print format
2020-08-07 13:58:14 +08:00
CalvinKirs
6ecc95a338
[FIX_BUG][*] fix some bugs from sonar analysis (#3322)
* [FIX_BUG][*] fix some bugs from sonar analysis

* remove invalid single test

* code smell

* remove exception catches that never happen

* delete invalid exception catch

* remove excess locks

* fix e2e test error

* recovery related exception removal
2020-08-04 02:23:50 +00:00
dailidong
6d903e266f
[Feature][server] add embeded zookeeper server in dolphinscheduler-daemon.sh (#3293)
* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

* simplify common config

* add worker group config

* add worker group config

* add worker group config

* this command just for getting a quick experience,not recommended for production. this operation will start a standalone zookeeper server

* Merge remote-tracking branch 'upstream/dev' into dev

* add logback for zookeeper

* Update logback-zookeeper.xml

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-07-31 17:37:02 +08:00
xsbai
87eb7c8211
adds waterdrop plugin (#3073)
* add plugin waterdrop

* add plugin waterdrop

* add plugin waterdrop

* fix deploy mode error

* Update en_US.js

fix conflict

* Update en_US.js

fix conflict

* Update en_US.js

Add missing commas

* Update zh_CN.js

Add missing commas

* Update waterdrop.vue

Fix the no resource hint

* Update zh_CN.js

Add 'Please select the waterdrop resources'

* Update en_US.js

Add  'Please select the waterdrop resources'

* Update waterdrop.vue

fix bug when click the waterdrop node again, it still show the `client` mode in deploy mode

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 23:20:59 +08:00
zixi0825
5fe31546ba
[Fix][all] fix spell error (#3289)
* fix spell error

* test unit test

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-30 18:33:16 +08:00
chendapao
afd9139c3f
[Feature]use zk own del func (#3310)
* use zk own del func

* Update remove-zk-node.sh

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 18:26:27 +08:00
CalvinKirs
3a0d4da27e
[Test][server] Fix Test ContextConfiguration error. (#3304)
* [FIX_BUG]fix test error

* small change

* add test

* reformat

* fix test no qualifying bean
2020-07-30 13:28:27 +08:00
zixi0825
78856fb08e
remove the duplicate function in MasterExecThread (#3329)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-29 19:15:03 +08:00
lenboo
82295eea52 fix bug#1336 set start/end time before complement the next process 2020-07-28 17:57:49 +08:00
lenboo
39182568b2 fix bug#1336 serial complement data can have multiple process instances 2020-07-27 15:43:08 +08:00
Yichao Yang
0577dc9780
[Test-3288][server] Fix github ci unit test oom (#3305)
* [Improvement] Test github action oom

* Optimize the test case performance

* Remove the unused import
2020-07-25 19:24:16 +08:00
tswstarplanet
5b7efd2d33
[Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client (#3244)
* [Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client

* autowire CuratorZookeeperClient to ZookeeperOperator

* [Improvement] Add the issue specifications reference (#3221)

* log zookeeper address when the connect state change

* resume the operation of add connect state listener in MasterRegistry

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-23 11:02:30 +08:00
LEI SHENG
6f2667bf1a
[Fix-3240][worker] Cannot find a (Map) Key deserializer for type (#3245)
* fix the bug #2923 (Hive JDBC connection parameter ignored) , the hive jdbc connection url ;

* fix the bug index out of bound and add some test case

* add the Licensed

* fix the checkstyle and import StringBuilder

* put HiveDataSourceTest.java to root pom maven-surefire-plugin to avoiding the sonar 0.0% Coverage.

* Update HiveDataSource.java

* fix the bug #3240  [worker] Cannot find a (Map) Key deserializer for type

* [DS-3245][fix]  change toString  use JsonString

- change the method of  UdfFunc.toString use JsonString

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the junit test for full

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the sonarcloud merge error, Swap these 2 arguments so they are in the correct order: expected value, actual value.

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

move the class UdfFuncDeserializer.java to UdfFunc.java

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

add blank line

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the unit test of UdfFunc.java

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

remove the UdfFuncTest.java from maven-surefire-plugin list

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the UdfFuncTest to the root pom maven-surefire-plugin

This closes #3245

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-21 18:54:07 +08:00
amenglu
178d8f829a
Update MasterSchedulerService.java (#3226) 2020-07-17 16:54:52 +08:00
hcz
50fcc492c1
Fix typo (#3230) 2020-07-17 14:26:46 +08:00
Yichao Yang
b4af3fd176
[Feature-2815][server] One worker can belong to different workergroups (#2934)
* [Feature-2815][server] One worker can belong to different workergroups

* Feature: Add test cases

* Update MonitorService.java

* Update MonitorService.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 17:25:13 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
gabry.wu
75259dcc07
simply doExecute using RetryerUtils.retryCall (#2761) 2020-07-13 14:38:40 +08:00
tswstarplanet
88f9bed726
[Bug-3187]close Heartbeat thread pool when MasterRegistry unRegistry (#3188)
* [Bug-3187]close Heartbeat thread pool when MasterRegistry unRegistry

* use shutdownNow; delete redundant code
2020-07-13 14:27:38 +08:00
CalvinKirs
933a67db8d
[Improvement][Server]Code Clean (#3196)
* [*][Server]Code Clean
redundant code removal

* redundant code removal

* redundant code removal
2020-07-12 22:25:25 +08:00
muzhongjiang
c49626b877
Word spelling modification,Comment modification,Log level modification (#3180)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

* Word spelling modification
Comment modification
Word spelling modification,Comment modification,Log level modification

* Update EmailManager.java

* Update FlinkParameters.java

* Update SqlTask.java

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-12 22:20:14 +08:00
iceblue510
1f8851b906
modify sqlTask (#3152) 2020-07-06 17:57:30 +08:00
CalvinKirs
d1282fdd96
In the case of multiple network cards, a valid host is returned (#2924)
* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix lock error

* fix lock error

* end

* end

* end

* end

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* lowering

* messy

* small change

* compile error

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
2020-07-02 19:16:37 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
Yichao Yang
38e485373d
[Feature-2925][server] Init TaskLogger in TaskExecuteProcessor (#2925) (#2965)
* [Feature-2925][common] Add exitVal judge in OSUtils.exeCmd (#2925)

* optimize the logger utils
2020-06-28 17:43:13 +08:00