Commit Graph

373 Commits

Author SHA1 Message Date
zixi0825
40a21816ab
[Fix] [Server] Fix the spell error and add log in SqlTask (#3243)
* fix the spell and modify logger print format
2020-08-07 13:58:14 +08:00
CalvinKirs
6ecc95a338
[FIX_BUG][*] fix some bugs from sonar analysis (#3322)
* [FIX_BUG][*] fix some bugs from sonar analysis

* remove invalid single test

* code smell

* remove exception catches that never happen

* delete invalid exception catch

* remove excess locks

* fix e2e test error

* recovery related exception removal
2020-08-04 02:23:50 +00:00
dailidong
6d903e266f
[Feature][server] add embeded zookeeper server in dolphinscheduler-daemon.sh (#3293)
* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

* simplify common config

* add worker group config

* add worker group config

* add worker group config

* this command just for getting a quick experience,not recommended for production. this operation will start a standalone zookeeper server

* Merge remote-tracking branch 'upstream/dev' into dev

* add logback for zookeeper

* Update logback-zookeeper.xml

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-07-31 17:37:02 +08:00
xsbai
87eb7c8211
adds waterdrop plugin (#3073)
* add plugin waterdrop

* add plugin waterdrop

* add plugin waterdrop

* fix deploy mode error

* Update en_US.js

fix conflict

* Update en_US.js

fix conflict

* Update en_US.js

Add missing commas

* Update zh_CN.js

Add missing commas

* Update waterdrop.vue

Fix the no resource hint

* Update zh_CN.js

Add 'Please select the waterdrop resources'

* Update en_US.js

Add  'Please select the waterdrop resources'

* Update waterdrop.vue

fix bug when click the waterdrop node again, it still show the `client` mode in deploy mode

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 23:20:59 +08:00
zixi0825
5fe31546ba
[Fix][all] fix spell error (#3289)
* fix spell error

* test unit test

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-30 18:33:16 +08:00
chendapao
afd9139c3f
[Feature]use zk own del func (#3310)
* use zk own del func

* Update remove-zk-node.sh

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 18:26:27 +08:00
CalvinKirs
3a0d4da27e
[Test][server] Fix Test ContextConfiguration error. (#3304)
* [FIX_BUG]fix test error

* small change

* add test

* reformat

* fix test no qualifying bean
2020-07-30 13:28:27 +08:00
zixi0825
78856fb08e
remove the duplicate function in MasterExecThread (#3329)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-29 19:15:03 +08:00
lenboo
82295eea52 fix bug#1336 set start/end time before complement the next process 2020-07-28 17:57:49 +08:00
lenboo
39182568b2 fix bug#1336 serial complement data can have multiple process instances 2020-07-27 15:43:08 +08:00
Yichao Yang
0577dc9780
[Test-3288][server] Fix github ci unit test oom (#3305)
* [Improvement] Test github action oom

* Optimize the test case performance

* Remove the unused import
2020-07-25 19:24:16 +08:00
tswstarplanet
5b7efd2d33
[Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client (#3244)
* [Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client

* autowire CuratorZookeeperClient to ZookeeperOperator

* [Improvement] Add the issue specifications reference (#3221)

* log zookeeper address when the connect state change

* resume the operation of add connect state listener in MasterRegistry

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-23 11:02:30 +08:00
LEI SHENG
6f2667bf1a
[Fix-3240][worker] Cannot find a (Map) Key deserializer for type (#3245)
* fix the bug #2923 (Hive JDBC connection parameter ignored) , the hive jdbc connection url ;

* fix the bug index out of bound and add some test case

* add the Licensed

* fix the checkstyle and import StringBuilder

* put HiveDataSourceTest.java to root pom maven-surefire-plugin to avoiding the sonar 0.0% Coverage.

* Update HiveDataSource.java

* fix the bug #3240  [worker] Cannot find a (Map) Key deserializer for type

* [DS-3245][fix]  change toString  use JsonString

- change the method of  UdfFunc.toString use JsonString

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the junit test for full

This closes #3245

* [DS-3245][fix]  change toString  use JsonString

- fix the sonarcloud merge error, Swap these 2 arguments so they are in the correct order: expected value, actual value.

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

move the class UdfFuncDeserializer.java to UdfFunc.java

This closes #3245

* [DS-3245][fix]  move the Class UdfFuncDeserializer to UdfFunc.java

add blank line

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the unit test of UdfFunc.java

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

remove the UdfFuncTest.java from maven-surefire-plugin list

This closes #3245

* [DS-3245][fix]  add the unit test of UdfFunc

add the UdfFuncTest to the root pom maven-surefire-plugin

This closes #3245

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-21 18:54:07 +08:00
amenglu
178d8f829a
Update MasterSchedulerService.java (#3226) 2020-07-17 16:54:52 +08:00
hcz
50fcc492c1
Fix typo (#3230) 2020-07-17 14:26:46 +08:00
Yichao Yang
b4af3fd176
[Feature-2815][server] One worker can belong to different workergroups (#2934)
* [Feature-2815][server] One worker can belong to different workergroups

* Feature: Add test cases

* Update MonitorService.java

* Update MonitorService.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 17:25:13 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
gabry.wu
75259dcc07
simply doExecute using RetryerUtils.retryCall (#2761) 2020-07-13 14:38:40 +08:00
tswstarplanet
88f9bed726
[Bug-3187]close Heartbeat thread pool when MasterRegistry unRegistry (#3188)
* [Bug-3187]close Heartbeat thread pool when MasterRegistry unRegistry

* use shutdownNow; delete redundant code
2020-07-13 14:27:38 +08:00
CalvinKirs
933a67db8d
[Improvement][Server]Code Clean (#3196)
* [*][Server]Code Clean
redundant code removal

* redundant code removal

* redundant code removal
2020-07-12 22:25:25 +08:00
muzhongjiang
c49626b877
Word spelling modification,Comment modification,Log level modification (#3180)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

* Word spelling modification
Comment modification
Word spelling modification,Comment modification,Log level modification

* Update EmailManager.java

* Update FlinkParameters.java

* Update SqlTask.java

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-12 22:20:14 +08:00
iceblue510
1f8851b906
modify sqlTask (#3152) 2020-07-06 17:57:30 +08:00
CalvinKirs
d1282fdd96
In the case of multiple network cards, a valid host is returned (#2924)
* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix lock error

* fix lock error

* end

* end

* end

* end

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* lowering

* messy

* small change

* compile error

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
2020-07-02 19:16:37 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
Yichao Yang
38e485373d
[Feature-2925][server] Init TaskLogger in TaskExecuteProcessor (#2925) (#2965)
* [Feature-2925][common] Add exitVal judge in OSUtils.exeCmd (#2925)

* optimize the logger utils
2020-06-28 17:43:13 +08:00
zixi0825
9bf67d80d0
Revise annotation spelling errors & Enhanced code robustness (#3042)
* revise annotation spelling errors & enhanced code robustness

* revise annotation spelling errors & enhanced code robustness

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-28 10:48:14 +08:00
CalvinKirs
8213da50d3
[WIP] load balance #3054 (#3057)
* Load balancing abstract

* code
smell

* lower weight select
2020-06-28 10:20:59 +08:00
CalvinKirs
ad28c65160
[BUG_FIX]fix multi-threaded tests error. (#3044)
* fix  test mock error

* fix  test mock error
2020-06-23 22:35:12 +08:00
lenboo
ea1c653e08 merge from dev 2020-06-22 10:21:15 +08:00
CalvinKirs
4aa1826718
fix testSqoopTaskTest bug (#3024) 2020-06-19 19:28:11 +08:00
lenboo
b81872cacd merge from dev-1.3.0 2020-06-19 18:52:30 +08:00
lenboo
73cae7f046 merge from dev-1.3.0 2020-06-19 15:19:32 +08:00
gabry.wu
a65cb48191
always send emails no matter there are results (#3015)
* always send emails no matter there are results

* always send emails no matter there are results
2020-06-19 14:50:10 +08:00
dailidong
620ecffa72
fix singe resource manager will NPE (#3013)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java
2020-06-19 14:45:48 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
gabry.wu
f1f9b59b3e
Merge branch 'dev' into removeUnnecessaryLock 2020-06-13 19:49:40 +08:00
Rubik-W
c0ddadef4d
Merge branch 'dev' into removeUnnecessaryLock 2020-06-13 12:09:52 +08:00
simon824
3eb367096a Resolved the conflict 2020-06-13 09:52:58 +08:00
Simon
eebd0c73bb
Merge pull request #3 from apache/dev
merge
2020-06-13 09:26:40 +08:00
BoYiZhang
f56ae9e64a
add http task timeout & test unit (#2955)
* add http task timeout & test unit

* add http task timeout & test unit

Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:52:19 +08:00
lijufeng2016
284c50f66c
Compatible with flink1.10 or newer (#2952)
* 兼容flink1.10以上版本

* fix null point bug

Co-authored-by: 李巨丰 <lijf@2345.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:23:28 +08:00
simon824
1b7d8ee236 fix the bad code smell 2020-06-12 12:35:18 +08:00
simon824
65953fe257 fix testBuildDataxJobContentJson 2020-06-12 12:10:17 +08:00
simon824
24120a4641 fix the bad code smell 2020-06-12 11:27:42 +08:00
simon824
65841a5709 1. change windows new line 'CR' to linux new line 'LF'
2. solve conflict
2020-06-12 09:31:04 +08:00
Simon
0b11275880
Merge pull request #2 from apache/dev
merge
2020-06-12 09:22:58 +08:00
Simon
4e37be68af
Update SqoopTaskTest.java 2020-06-12 09:18:00 +08:00
simon824
f643981f48 1. change windows new line 'CR' to linux new line 'LF'
2. Format code
2020-06-11 10:14:15 +08:00
Eights
a420644aa3 sqoop task source type or target type is null throw exception 2020-06-10 22:20:01 +08:00
lenboo
ee8581b21e change delete status according to delete action result 2020-06-10 15:15:03 +08:00