Commit Graph

279 Commits

Author SHA1 Message Date
felix.wang
5584f0cb4d
[Fix-3262][common] When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled (#3264)
* fix bug #3165   get resource.storage.type value   toUpperCase

* fix bug #3176  optimize Gets the value of this property “resource.storage.type”

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix miss

* add KerberosHttpClient fix  kerberos bug

* fix map init

* add juint test

* Extraction of public methods

* Extraction of public methods

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* rebuild test

* rebuild test

* check style

* check style

* check style

* revert UT POM

* Kerberos judgment goes up to hadoopUtils

* fix merge

* Remove connection pool shutdown

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-10 15:18:01 +08:00
Yichao Yang
3ed0afa6ac
[Improvement-3369][api] Introduce data analysis service interface for clear code (#3439)
* [Improvement][api] Introduce data analysis service interface for clear code

* Remove the bad smell

* Remove the bad smell
2020-08-10 14:47:45 +08:00
CalvinKirs
6ecc95a338
[FIX_BUG][*] fix some bugs from sonar analysis (#3322)
* [FIX_BUG][*] fix some bugs from sonar analysis

* remove invalid single test

* code smell

* remove exception catches that never happen

* delete invalid exception catch

* remove excess locks

* fix e2e test error

* recovery related exception removal
2020-08-04 02:23:50 +00:00
BoYiZhang
d70d8a6054
[Feature-3222][datasource] Store password in ciphertext instead (#3330)
* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* dag  add close button

* reset last version

* reset last version

* dag add close buttion

dag add close buttion

* update  CLICK_SAVE_WORKFLOW_BUTTON  xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* Update CreateWorkflowLocator.java

modify submit workflow button

* Update CreateWorkflowLocator.java

* Update CreateWorkflowLocator.java

modify CLICK_ADD_BUTTON

* Update CreateWorkflowLocator.java

delete print

* Update CreateWorkflowLocator.java

1

* Update CreateWorkflowLocator.java

1

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dag.vue

* Update dag.vue

* Update dag.vue

* Update CreateWorkflowLocator.java

* Revert "Setting '-XX:+DisableExplicitGC ' causes netty memory leaks"

This reverts commit 3a2cba7a

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* update test unit

* add test unit

* add test unit

* update e2e test

* update unit test

* update unit test

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-08-01 14:28:12 +08:00
xsbai
87eb7c8211
adds waterdrop plugin (#3073)
* add plugin waterdrop

* add plugin waterdrop

* add plugin waterdrop

* fix deploy mode error

* Update en_US.js

fix conflict

* Update en_US.js

fix conflict

* Update en_US.js

Add missing commas

* Update zh_CN.js

Add missing commas

* Update waterdrop.vue

Fix the no resource hint

* Update zh_CN.js

Add 'Please select the waterdrop resources'

* Update en_US.js

Add  'Please select the waterdrop resources'

* Update waterdrop.vue

fix bug when click the waterdrop node again, it still show the `client` mode in deploy mode

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 23:20:59 +08:00
RedemptionC
0c5754c3f2
add feature : query top n success process instance order by running duration (#3315)
* add feature : query top n success process instance order by running duration

* add feature : add params check

* add feature : remove code smell

* add feature : remove code smell

* add feature : add param check

* add feature : add param check
improve handleEscape method in ParameterUtils.java

* top n : add  java doc and format code
2020-07-30 18:38:59 +08:00
zixi0825
5fe31546ba
[Fix][all] fix spell error (#3289)
* fix spell error

* test unit test

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-30 18:33:16 +08:00
felix.wang
b555d0a086
[Improvement][common] HttpClient singleton mode #3332 (#3350)
* Optimize the httpclient

* Optimize the httpclient singleton

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java
2020-07-30 13:24:54 +08:00
felix.wang
e026d303ea
Optimize the httpclient (#3337) 2020-07-29 19:03:44 +08:00
t1mon
07c8193bb5
[Feature-2930][api] Displays path and host on the instance log panel. #2930 (#3154)
* Optimize PropertyUtils instantiation.

* Fix info error.

* [Feature-2930][api] Displays path and host on the instance log panel. #2930

* [update] Add log-head in download log.

* [update] clear code smell.
2020-07-23 10:57:56 +08:00
CalvinKirs
38e7494f78
replace CollectionUtils deprecated method (#3027)
* replace deprecated method

* replace deprecated method

* replace deprecated method

* replace deprecated method

* replace deprecated method

* messy

* messy

* messy

* add missing license

Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 19:42:33 +08:00
CalvinKirs
6f9970b189
[Feature-2574][Server]Specify Network Interface (#3186)
* [Feature-2574][Server]Specify Network Interface
This closes #2574

* code smell

* code smell

* move DOLPHIN_SCHEDULER_PREFERRED_NETWORK_INTERFACE to Constants.java
2020-07-13 18:51:38 +08:00
Yichao Yang
b4af3fd176
[Feature-2815][server] One worker can belong to different workergroups (#2934)
* [Feature-2815][server] One worker can belong to different workergroups

* Feature: Add test cases

* Update MonitorService.java

* Update MonitorService.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 17:25:13 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
muzhongjiang
c49626b877
Word spelling modification,Comment modification,Log level modification (#3180)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

* Word spelling modification
Comment modification
Word spelling modification,Comment modification,Log level modification

* Update EmailManager.java

* Update FlinkParameters.java

* Update SqlTask.java

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-12 22:20:14 +08:00
rockxsj
dcdd7dedd0
Modify the AWS S3 request encryption method to V4. (#3182)
Co-authored-by: rockxsj <junjun@mgtv.com>
2020-07-11 08:56:38 +08:00
felix-thinkingdata
1e7582e910
#3176 optimize Gets the value of this property “resource.storage.type”, Comparison with enumerated types (#3178)
* fix bug #3165   get resource.storage.type value   toUpperCase

* fix bug #3176  optimize Gets the value of this property “resource.storage.type”

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-10 15:21:42 +08:00
felix-thinkingdata
a23a3e2d12
fix bug #3165 get resource.storage.type value toUpperCase (#3166)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-09 18:28:04 +08:00
liukunyuan
0692dca4e6
update CommonUtils.java (#3137)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-05 21:50:42 +08:00
t1mon
83589c7b7c
Optimize kerberos tgt expire time in HadoopUtils. (#2700)
* Optimize PropertyUtils instantiation.

* Fix info error.

* Optimize kerberos tgt expire time.

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-04 18:22:30 +08:00
CalvinKirs
d1282fdd96
In the case of multiple network cards, a valid host is returned (#2924)
* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix lock error

* fix lock error

* end

* end

* end

* end

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* lowering

* messy

* small change

* compile error

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
2020-07-02 19:16:37 +08:00
muzhongjiang
a9eb7b18b0
[bugfix] fixed "jackson enum conversion : InvalidFormatException" (#3078)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-02 18:07:57 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
Yichao Yang
38e485373d
[Feature-2925][server] Init TaskLogger in TaskExecuteProcessor (#2925) (#2965)
* [Feature-2925][common] Add exitVal judge in OSUtils.exeCmd (#2925)

* optimize the logger utils
2020-06-28 17:43:13 +08:00
lenboo
ea1c653e08 merge from dev 2020-06-22 10:21:15 +08:00
CalvinKirs
518d03b6a6
Simplify some methods for easy understanding (#2920)
Co-authored-by: gabry.wu <8545796+gabrywu@users.noreply.github.com>
2020-06-21 15:48:35 +08:00
lenboo
73cae7f046 merge from dev-1.3.0 2020-06-19 15:19:32 +08:00
dailidong
620ecffa72
fix singe resource manager will NPE (#3013)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java
2020-06-19 14:45:48 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
Yichao Yang
0744d4070c
[Hotfix-2922][common] Fix Jackson UDde error (#2922) (#2926)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-17 11:33:41 +08:00
Yichao Yang
0e7403e6a8
[Feature-2927][common] Fix NEF when get info from OSUtils (#2927) (#2964)
* [Feature-2927][common] Fix NEF when get info from OSUtils (#2927)

* Update OSUtils.java

Change java doc

Co-authored-by: gabry.wu <8545796+gabrywu@users.noreply.github.com>
2020-06-14 11:17:12 +08:00
simon824
3fa4dd702b Resolved the conflict 2020-06-13 09:47:18 +08:00
Simon
eebd0c73bb
Merge pull request #3 from apache/dev
merge
2020-06-13 09:26:40 +08:00
BoYiZhang
f56ae9e64a
add http task timeout & test unit (#2955)
* add http task timeout & test unit

* add http task timeout & test unit

Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:52:19 +08:00
lijufeng2016
284c50f66c
Compatible with flink1.10 or newer (#2952)
* 兼容flink1.10以上版本

* fix null point bug

Co-authored-by: 李巨丰 <lijf@2345.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:23:28 +08:00
simon824
1b7d8ee236 fix the bad code smell 2020-06-12 12:35:18 +08:00
simon824
24120a4641 fix the bad code smell 2020-06-12 11:27:42 +08:00
simon824
cfecfb799b 1. change windows new line 'CR' to linux new line 'LF'
2. solve conflict
2020-06-12 09:59:40 +08:00
Simon
0b11275880
Merge pull request #2 from apache/dev
merge
2020-06-12 09:22:58 +08:00
lenboo
1904a7022a change 1.3.0 to 1.3.0-snapshot 2020-06-10 11:42:15 +08:00
simon824
fad09075f9 remove fastjson maven dependence 2020-06-10 09:16:28 +08:00
eights
5d83d797d3 modify sqoop task UT 2020-06-10 09:11:17 +08:00
eights
c8f28ab2ba sqoop task optimization 2020-06-09 03:16:35 -07:00
simon824
8cabb7a907 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 14:15:13 +08:00
simon824
6e907f150c 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 13:02:50 +08:00
bao liang
422fe5136d
[license] add license and notice (#2916)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

* test for gc

* change version to 1.3.0

* check and add some license files

* check and add some license files

* revert

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-06 22:21:37 +08:00
Rubik-W
7c9f5960e5
Fixbug datax task (#2909)
* fix:  local param bug

* fix: UT bug

Co-authored-by: Rubik-W <whm_777@163.com>
2020-06-06 21:12:41 +08:00
simon824
572ee0fc64 Using Jackson instead of Fastjson 2020-06-04 16:23:59 +08:00
simon
8bab8abcd2 ut fix 2020-06-03 19:21:19 +08:00
simon
bb089288c4 Using Jackson instead of Fastjson 2020-06-03 12:40:28 +08:00