Commit Graph

227 Commits

Author SHA1 Message Date
Kirs
449cbbe21e
[feature#3356] alert-spi support DingTalk&WeChat (#3869)
* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* [feature][ui]Alert plugin design (#3734)

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* [Feature-3682][ui]Add form-create plug-in and alarm group management add sample demo (#3683)

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* [feature][ui] Add alarm instance page

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* [Feature-3189][alert,spi,dao,plugin-api] base code of dolphinscheduler spi and alert plugin implement (#3601)

* DS SPI

* Add DolphinScheduler SPI , and rebuilt the code of the Alert plug-in based on SPI

* Add DolphinScheduler SPI , and rebuilt the code of the Alert plug-in based on SPI

* add TODO

* delete

* compile

* spi commit

* Plugin Alert

* fix some bug

* add todo

* change web ui from alpacajs to form-create

* remove module

* add plugin schema

* add license header

* update alert and spi module version

* update the alert plugin sub module version

* comment the maven.local.repository param

* move utils from spi to common module

* add license header

* add license header and delete some chinese comment

* update spi packages

* delete no use alert_xx.properties

* update mysql.connector.version back to 5.1.34

* delete no use comment in pom.xml

* update email stmp password

* add license

* add semicolon to sql/upgrade/1.4.0_schema/mysql/dolphinscheduler_ddl.sql file

* format the code style

* format new clase file with checkstyle

* update plugin params to Builder model

* move JSONUtils to SPI because plugin can not dependency common module

* move JSONUtils to SPI because plugin can not dependency common module

* delete collection dependency

* replace PluginParamsTransfer to spi PluginParamsTransfer

* update dolphinscheduler-maven-plugin to 1.0.0

* update license

* update apache-rat-plugin add exclude '.iml' file

* check license

* ArtifactResolver only use in development and configPlugins is not empty

* ArtifactResolver only use in development and configPlugins is not empty

* ArtifactResolver only use in development and configPlugins is not empty

* default datasource should be postgresql

* add license files

* add license files

* postgresql port should be 5432

* postgresql test

* mv show_type to spi

add license header to AlertConstants

* check style fix

* copy check style file from branch dev

* alert show_type set by plugin

* alert show_type set by plugin

* add PluginDefineMapper to dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/registry/DependencyConfig.java

* add Bean to TaskCallbackServiceTestConfig

* add Bean to TaskCallbackServiceTestConfig

* fix check style

* check style fix

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* fix check style

* [Feature-3682][ui]Add form-create plug-in and alarm group management add sample demo (#3683)

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* check style fix

* rollback test change

* rollback test change

* rollback dao pom change

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* add ut to pom.xml

* add upgrade schema to global schema

* fix ut failed

* fix ut failed

* fix ut failed

* fix ut failed

* add test EmailAlertPluginTest to pom.xml

* fix ut failed

* fix ut failed

* fix check style

* update license header to presto license header

* presto license header not check

* fix ut coverage

* fix ut coverage

* fix ut

* fix ut

* fix ut

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

Co-authored-by: break60 <790061044@qq.com>

* [feature#3356] alert-spi support DingTalk

this closes # 3356

* add test

* code style

* we chat alert

* support we chat alert

* support we chat alert

* support we chat alert,update ding talk alert

* code style

* add test

* code style

* clean old code

* clean old code

* code smell

* code style

* add test

* simple config

* code style

* code style

* code style

* delete old file

* fix log content error

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: gaojun2048 <32193458+gaojun2048@users.noreply.github.com>
2020-11-02 10:08:28 +08:00
gaojun2048
1d254347df [Feature-3189][alert,spi,dao,plugin-api] base code of dolphinscheduler spi and alert plugin implement (#3601)
* DS SPI

* Add DolphinScheduler SPI , and rebuilt the code of the Alert plug-in based on SPI

* Add DolphinScheduler SPI , and rebuilt the code of the Alert plug-in based on SPI

* add TODO

* delete

* compile

* spi commit

* Plugin Alert

* fix some bug

* add todo

* change web ui from alpacajs to form-create

* remove module

* add plugin schema

* add license header

* update alert and spi module version

* update the alert plugin sub module version

* comment the maven.local.repository param

* move utils from spi to common module

* add license header

* add license header and delete some chinese comment

* update spi packages

* delete no use alert_xx.properties

* update mysql.connector.version back to 5.1.34

* delete no use comment in pom.xml

* update email stmp password

* add license

* add semicolon to sql/upgrade/1.4.0_schema/mysql/dolphinscheduler_ddl.sql file

* format the code style

* format new clase file with checkstyle

* update plugin params to Builder model

* move JSONUtils to SPI because plugin can not dependency common module

* move JSONUtils to SPI because plugin can not dependency common module

* delete collection dependency

* replace PluginParamsTransfer to spi PluginParamsTransfer

* update dolphinscheduler-maven-plugin to 1.0.0

* update license

* update apache-rat-plugin add exclude '.iml' file

* check license

* ArtifactResolver only use in development and configPlugins is not empty

* ArtifactResolver only use in development and configPlugins is not empty

* ArtifactResolver only use in development and configPlugins is not empty

* default datasource should be postgresql

* add license files

* add license files

* postgresql port should be 5432

* postgresql test

* mv show_type to spi

add license header to AlertConstants

* check style fix

* copy check style file from branch dev

* alert show_type set by plugin

* alert show_type set by plugin

* add PluginDefineMapper to dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/registry/DependencyConfig.java

* add Bean to TaskCallbackServiceTestConfig

* add Bean to TaskCallbackServiceTestConfig

* fix check style

* check style fix

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* fix check style

* [Feature-3682][ui]Add form-create plug-in and alarm group management add sample demo (#3683)

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* check style fix

* rollback test change

* rollback test change

* rollback dao pom change

* [feature-3665][ui]Add element-ui (#3666)

* [feature-3665][ui]Add element-ui

* add license

* Add form-create plug-in and alarm group management add sample demo

* Modify node version

* fix

* fix

* add ut to pom.xml

* add upgrade schema to global schema

* fix ut failed

* fix ut failed

* fix ut failed

* fix ut failed

* add test EmailAlertPluginTest to pom.xml

* fix ut failed

* fix ut failed

* fix check style

* update license header to presto license header

* presto license header not check

* fix ut coverage

* fix ut coverage

* fix ut

* fix ut

* fix ut

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

* fix ut coverage

Co-authored-by: break60 <790061044@qq.com>
2020-10-12 15:28:11 +08:00
bao liang
39411ce03b
[FIX-3617][Service]fix 2 tasks instance are generated when fault tolerance (#3873)
* fix 2 tasks instance are generated when fault tolerance

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update code style

* update ut

* update code style

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-10-10 15:05:56 +08:00
guyinyou
3b581455fc
[Feature]Add Python task "task variable / result transfer" implementation (#3659)
* 增加Python Task的“任务变量/结果传递”实现
Signed-off-by: 古崟佑

* add two files license
Signed-off-by: 古崟佑

* fix 'server/worker/task/AbstractCommandExecutor.java' code style
Signed-off-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>

* update DB
Signed-off-by: 古崟佑

* update DB -- 2
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codestyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* add VarPoolUtils Test
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest codeStyle
Signed-off-by: 古崟佑

* add test config for VarPoolUtilsTest
Signed-off-by: 古崟佑

* fix unit test
Signed-off-by: 古崟佑

* fix codeStyle
Signed-off-by: 古崟佑

* fix VarPoolUtilsTest.java
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* change the test class path
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

* fix "print the error message"
Signed-off-by: 古崟佑

* fix bug
Signed-off-by: 古崟佑

* fix
Signed-off-by: 古崟佑

Co-authored-by: 1941815847Cy4 <1941815847cy4@kuaishou.com>
2020-09-25 13:20:42 +08:00
CalvinKirs
db663a13a3
[Improvement][remote]load balance warm up (#3770)
* [Improvement][remote]load balance  warm up

* reformat code

* reformat code

* code smell

* code smell

* add test

* add test

* add test

* add test

* fix bug

* fix bug

* add docs

* add host test

* add host test

* add host test

* add docs

* code reformat

* code reformat
2020-09-22 17:39:39 +08:00
zhuangchong
fad2852768
[Improvement][server] WATERDROP task plug-in optimization in switch case code cleaning. (#3652)
* WATERDROP switch case code checkstyle.

* add TaskManagerTest.

* TaskManagerTest checkstyle.

* TaskManagerTest checkstyle.

* TaskManagerTest add pom maven-surefire

* TaskManagerTest update.
2020-09-14 15:59:56 +08:00
felix.wang
3a2663eeb7
[Improvement][DAO] druid The lower version is not compatible with LocalDateTime(DRUID 低版本不支持LocalDateTime) (#3716)
* fix  mybatis 3.51版本的LocalDateTime映射报SQLFeatureNotSupportedException

* fix  druid-1.1.14.jar->1.1.22
2020-09-13 23:55:25 +08:00
Hsu Pu
68bc58892b
[Improvement][server] Refactor code for getTaskLogPath to reduce NPE log during UT (#3629)
* init

* update

* update

* update

* fix

* fix

* enable test

* fix

* fix
2020-09-07 10:39:40 +08:00
Hsu Pu
feec629554
[Improvement][server] unit test for SubProcessTask with PowerMockRunner (#3632) 2020-09-03 21:27:29 +08:00
CalvinKirs
632d52cd69
[refactor][server-test] refactor RoundRobinHostManagerTest (#3511)
* [refactor][server-test] refactor RoundRobinHostManagerTest

* add RoundRobinHostManagerTest

* format code

* format code
2020-08-31 20:35:45 +08:00
Yichao Yang
8fea8f913c
[Test][server] Add the master server test case (#3574) 2020-08-25 10:54:39 +08:00
CalvinKirs
34a04acafd
[Improvement][remote] improvement netty eventLoopGroup (#3580)
* [Improvement][remote] improvement netty eventLoopGroup
If you are running on linux you can use EpollEventLoopGroup and so get better performance, less GC and have more advanced features that are only available on linux.

* reformat code

* utility classes should not have public constructors

* reformat code

* reformat code

* reformat code

* add test

* code style

* add test

* add test

* fix test error

* add test

* test

* test

* add test config

* add test config
2020-08-25 10:47:40 +08:00
Yichao Yang
93660f4d61
[Feature-2154][api] Workflow version control (#3053)
* [Feature][api] Workflow version control

* Update messages.properties

* Update messages_en_US.properties

* Update messages_zh_CN.properties
2020-08-25 10:46:58 +08:00
easycode
78eb07b361
[Improvement-3558][API] Improve DataSourceServiceTest.java (#3579)
* Fixed this bug "multiple threads creating folders will result in multiple identical folders" #3487

* 1. Improvement `DataSourceServiceTest` #3558
2. Get rid of some useless arguments to functions in the `DataSourceService` class

* The rollback `ResourceService.java`

* The rollback `ResourceService.java`

* The rollback `ResourceService.java`

* 1.Modify code styles.
2.Make improvement according to Yichao Yang's advice. Thank you very much for guidance.@Yichao Yang

* Modify code styles.

* Circumventing the checkConnection problem

* Modify code styles.
2020-08-24 16:36:02 +08:00
vanilla111
a4ee351a3a
delay execution of tasks and improve some designs (#3427) 2020-08-19 11:36:30 +08:00
Hsu Pu
59610a5661
[Improvement][dao,server] unit test for ConditionsTask (#3385) 2020-08-18 14:46:00 +08:00
rockxsj
2b46db0bf7
[Feature]add presto datasource support (#3308)
* * add presto datasource support
update .gitigonre to igonre some files

* * use another presto driver

* * add LICENSE files about presto-jdbc

* * just for test sonar

Co-authored-by: rockxsj <junjun@mgtv.com>
2020-08-14 08:52:31 +08:00
felix.wang
5584f0cb4d
[Fix-3262][common] When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled (#3264)
* fix bug #3165   get resource.storage.type value   toUpperCase

* fix bug #3176  optimize Gets the value of this property “resource.storage.type”

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix miss

* add KerberosHttpClient fix  kerberos bug

* fix map init

* add juint test

* Extraction of public methods

* Extraction of public methods

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* rebuild test

* rebuild test

* check style

* check style

* check style

* revert UT POM

* Kerberos judgment goes up to hadoopUtils

* fix merge

* Remove connection pool shutdown

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-10 15:18:01 +08:00
Yichao Yang
f419fe6239
[Test][server] Add LoggerServerTest (#3405) 2020-08-10 15:01:20 +08:00
Yichao Yang
4fb06a736d
[Test][api] Introduce api controller unit test example (#3447)
* [Test][api] Introduce api controller test example

* Add controller test
2020-08-10 14:36:31 +08:00
LiemLin
f27c62a5db
workflow lineage (#2421)
* workflow lineage

* workflow lineage unit test

* stash

* add apache license

* Update .env

* code optimize

* add unit test

* add apache license

* add apache license

* optimized code

* add postgresql support and optimized code

* Update pom.xml

* Update en_US.js

* Update zh_CN.js

* remove mock code

* inport i18n

* Update pom.xml

* Delete .env

* add env file

Co-authored-by: linhaiqiang <haiqiang.lin@five-star.cn>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: becarchal <yangbin_57@sina.com>
2020-07-30 13:32:29 +08:00
CalvinKirs
3a0d4da27e
[Test][server] Fix Test ContextConfiguration error. (#3304)
* [FIX_BUG]fix test error

* small change

* add test

* reformat

* fix test no qualifying bean
2020-07-30 13:28:27 +08:00
tswstarplanet
5b7efd2d33
[Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client (#3244)
* [Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client

* autowire CuratorZookeeperClient to ZookeeperOperator

* [Improvement] Add the issue specifications reference (#3221)

* log zookeeper address when the connect state change

* resume the operation of add connect state listener in MasterRegistry

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-23 11:02:30 +08:00
LEI SHENG
6f2667bf1a
[Fix-3240][worker] Cannot find a (Map) Key deserializer for type (#3245)
* fix the bug #2923 (Hive JDBC connection parameter ignored) , the hive jdbc connection url ;

* fix the bug index out of bound and add some test case

* add the Licensed

* fix the checkstyle and import StringBuilder

* put HiveDataSourceTest.java to root pom maven-surefire-plugin to avoiding the sonar 0.0% Coverage.

* Update HiveDataSource.java

* fix the bug #3240  [worker] Cannot find a (Map) Key deserializer for type

* [DS-3245][fix]  change toString  use JsonString

- change the method of  UdfFunc.toString use JsonString

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the junit test for full

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the sonarcloud merge error, Swap these 2 arguments so they are in the correct order: expected value, actual value.

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

move the class UdfFuncDeserializer.java to UdfFunc.java

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

add blank line

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the unit test of UdfFunc.java

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

remove the UdfFuncTest.java from maven-surefire-plugin list

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the UdfFuncTest to the root pom maven-surefire-plugin

This closes #3245

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-21 18:54:07 +08:00
Yichao Yang
b4af3fd176
[Feature-2815][server] One worker can belong to different workergroups (#2934)
* [Feature-2815][server] One worker can belong to different workergroups

* Feature: Add test cases

* Update MonitorService.java

* Update MonitorService.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 17:25:13 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
LEI SHENG
98fdba6740
[Bug 2923] Hive JDBC connection parameter ignored (#3194)
* fix the bug #2923 (Hive JDBC connection parameter ignored) , the hive jdbc connection url ;

* fix the bug index out of bound and add some test case

* add the Licensed

* fix the checkstyle and import StringBuilder

* put HiveDataSourceTest.java to root pom maven-surefire-plugin to avoiding the sonar 0.0% Coverage.

* Update HiveDataSource.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 13:58:01 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
Jave-Chen
f1632e22c5
Set up JDK 11 for SonarCloud in github action. (#3052)
* Set up JDK 11 for SonarCloud in github action.

* Fix javadoc error with JDK 11.

* Prevent Javadoc from stopping if it finds any html errors.
2020-06-25 23:41:43 +08:00
lenboo
e343ee0ee0 remove exception ut 2020-06-24 13:02:37 +08:00
lenboo
6a52efae52 remove exception ut 2020-06-24 12:54:17 +08:00
lenboo
7da095d04d remove exception ut 2020-06-24 11:32:04 +08:00
lenboo
3d23a6c6e6 remove exception ut 2020-06-24 11:04:42 +08:00
lenboo
513950c036 remove exception ut 2020-06-24 10:53:00 +08:00
lenboo
73cae7f046 merge from dev-1.3.0 2020-06-19 15:19:32 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
743294668
a58d1b3b45
Modify OracleDB connection string format. (#2970)
Co-authored-by: taoyuxin <taoyuxin@inspur.com>
2020-06-15 11:12:22 +08:00
break60
b939ab8e1d Delete the pom files animation.scss, _normalize.scss and _animation.scss under the common folder 2020-06-14 20:28:34 +08:00
Simon
2e4adbe03f
Update pom.xml 2020-06-13 09:27:27 +08:00
Simon
eebd0c73bb
Merge pull request #3 from apache/dev
merge
2020-06-13 09:26:40 +08:00
Simon
6c5b0706f5
Update pom.xml 2020-06-13 09:25:56 +08:00
Simon
7f725c32b5
Update pom.xml 2020-06-13 09:24:16 +08:00
BoYiZhang
f56ae9e64a
add http task timeout & test unit (#2955)
* add http task timeout & test unit

* add http task timeout & test unit

Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:52:19 +08:00
simon824
65841a5709 1. change windows new line 'CR' to linux new line 'LF'
2. solve conflict
2020-06-12 09:31:04 +08:00
Simon
71619ed29b
Update pom.xml 2020-06-12 09:22:44 +08:00
Simon
b5a296f4cd
Update pom.xml 2020-06-12 09:21:07 +08:00
Simon
094d09ba74
Update pom.xml 2020-06-12 09:20:28 +08:00
lenboo
1904a7022a change 1.3.0 to 1.3.0-snapshot 2020-06-10 11:42:15 +08:00
eights
822addac30 add sqoop task UT to pom 2020-06-10 10:43:22 +08:00
simon824
fad09075f9 remove fastjson maven dependence 2020-06-10 09:16:28 +08:00