Commit Graph

447 Commits

Author SHA1 Message Date
felix.wang
2ba529a96c
[Improvement][alert] Optimize alarm information (#3559)
* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format.

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* #3299  Json string parsing problem caused by non-standard json format. fix  code style

* Optimize alarm information

* fix code style

* add license

* fix code smell

* Update dolphinscheduler-alert/src/main/java/org/apache/dolphinscheduler/alert/utils/EnterpriseWeChatUtils.java

Co-authored-by: Yichao Yang <1048262223@qq.com>

* fix code smell

* fix code smell

* 修改ut 和replace问题。

* 修改ut

Co-authored-by: wangjianda <Felix@thinkingdata.com>
Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-08-25 11:10:44 +08:00
Zhou Zheng
31cd1f6cb2 adapt to code style 2020-08-23 15:14:18 +08:00
Zhou Zheng
295796a5c5 fix 2020-08-22 22:47:59 +08:00
Zhou Zheng
6bdd11da51 Merge branch 'dev' into dev-FTS 2020-08-22 22:31:07 +08:00
vanilla111
a4ee351a3a
delay execution of tasks and improve some designs (#3427) 2020-08-19 11:36:30 +08:00
dengc367
ad89f433f1
[Feature]modify some cases from rockxsj:Feature-presto to add presto datasource support (#3468)
* Feature presto (#1)

* * add presto datasource support
update .gitigonre to igonre some files

* * use another presto driver

* * add LICENSE files about presto-jdbc

* * just for test sonar

Co-authored-by: rockxsj <junjun@mgtv.com>

* modify the io.prestosql.jdbc.PrestoDriver to com.facebook.presto.jdbc.PrestoDriver

* add presto connection in sql node

Co-authored-by: rockxsj <junjun@mgtv.com>
2020-08-17 17:28:46 +08:00
qiaozhanwei
917a4b20b5
release 1.3.2 version rollback (#3527)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

* modify tag 1.3.0 to HEAD

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-17 14:21:13 +08:00
qiaozhanwei
32edd19ac5 [maven-release-plugin] prepare for next development iteration 2020-08-17 14:03:19 +08:00
qiaozhanwei
4d2f0b0578 [maven-release-plugin] prepare release 1.3.2 2020-08-17 14:03:08 +08:00
qiaozhanwei
11d3c07e46
modify tag 1.3.0 to HEAD (#3525)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

* modify tag 1.3.0 to HEAD

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-17 11:46:35 +08:00
qiaozhanwei
f3f895751b [maven-release-plugin] prepare for next development iteration 2020-08-14 19:20:27 +08:00
qiaozhanwei
f19b125374 [maven-release-plugin] prepare release 1.3.2 2020-08-14 19:20:17 +08:00
qiaozhanwei
4ba6b35b0f
test release 1.3.2 version rollback (#3504)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 18:55:03 +08:00
qiaozhanwei
f50353f8a6 [maven-release-plugin] prepare for next development iteration 2020-08-14 18:26:09 +08:00
qiaozhanwei
afd227a179 [maven-release-plugin] prepare release 1.3.2 2020-08-14 18:25:59 +08:00
qiaozhanwei
660eb5a1e6
test release 1.3.2 version rollback (#3503)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 18:19:19 +08:00
qiaozhanwei
b5f0f23d85 [maven-release-plugin] prepare for next development iteration 2020-08-14 17:37:33 +08:00
qiaozhanwei
88b3ef5d30 [maven-release-plugin] prepare release 1.3.2 2020-08-14 17:37:23 +08:00
qiaozhanwei
45424ec265
test release 1.3.2 version rollback (#3499)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 17:31:09 +08:00
qiaozhanwei
c6f0c8901c [maven-release-plugin] prepare for next development iteration 2020-08-14 16:04:49 +08:00
qiaozhanwei
8a6732ae4c [maven-release-plugin] prepare release 1.3.2 2020-08-14 16:04:38 +08:00
rockxsj
2b46db0bf7
[Feature]add presto datasource support (#3308)
* * add presto datasource support
update .gitigonre to igonre some files

* * use another presto driver

* * add LICENSE files about presto-jdbc

* * just for test sonar

Co-authored-by: rockxsj <junjun@mgtv.com>
2020-08-14 08:52:31 +08:00
zixi0825
46ab86d6df
batch copy or move process #2753 (#2884)
* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* Roll back the package.json file

* rerun test

* revert blank

* resolve conflicts

* resolve conflicts

* resolve ut error

* resolve sonar error

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-13 15:37:46 +08:00
felix.wang
5584f0cb4d
[Fix-3262][common] When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled (#3264)
* fix bug #3165   get resource.storage.type value   toUpperCase

* fix bug #3176  optimize Gets the value of this property “resource.storage.type”

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix miss

* add KerberosHttpClient fix  kerberos bug

* fix map init

* add juint test

* Extraction of public methods

* Extraction of public methods

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* rebuild test

* rebuild test

* check style

* check style

* check style

* revert UT POM

* Kerberos judgment goes up to hadoopUtils

* fix merge

* Remove connection pool shutdown

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-10 15:18:01 +08:00
Yichao Yang
3ed0afa6ac
[Improvement-3369][api] Introduce data analysis service interface for clear code (#3439)
* [Improvement][api] Introduce data analysis service interface for clear code

* Remove the bad smell

* Remove the bad smell
2020-08-10 14:47:45 +08:00
Zhou Zheng
90a060128f 修改与新增commandType相关的代码
1. dataAnalysis
2. alertManager
3. businessTimeUtils的schedule选择
2020-08-07 15:39:48 +08:00
Yelli
e4097a488c
cancel spark task version check (#3406)
Co-authored-by: Eights-Li <yelli.hl@gmail.com>
2020-08-05 10:20:34 +08:00
dailidong
d64127b90a optimize TheadUtils test 2020-08-04 17:57:23 +08:00
lgcareer
a3793be7da
[Fix-3390][server]Running hive sql task need find the hdfs path correctly (#3396)
* [Fix-3390][api]Running hive sql task need find the hdfs path correctly

* [Fix-3390][api]Running hive sql task need find the hdfs path correctly

* update soft version
2020-08-04 15:04:56 +08:00
CalvinKirs
6ecc95a338
[FIX_BUG][*] fix some bugs from sonar analysis (#3322)
* [FIX_BUG][*] fix some bugs from sonar analysis

* remove invalid single test

* code smell

* remove exception catches that never happen

* delete invalid exception catch

* remove excess locks

* fix e2e test error

* recovery related exception removal
2020-08-04 02:23:50 +00:00
lgcareer
3744167d52
[Fix-3364]Running the spark task fail because the can't find the jar (#3378) 2020-08-03 10:31:27 +08:00
BoYiZhang
d70d8a6054
[Feature-3222][datasource] Store password in ciphertext instead (#3330)
* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* dag  add close button

* reset last version

* reset last version

* dag add close buttion

dag add close buttion

* update  CLICK_SAVE_WORKFLOW_BUTTON  xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* Update CreateWorkflowLocator.java

modify submit workflow button

* Update CreateWorkflowLocator.java

* Update CreateWorkflowLocator.java

modify CLICK_ADD_BUTTON

* Update CreateWorkflowLocator.java

delete print

* Update CreateWorkflowLocator.java

1

* Update CreateWorkflowLocator.java

1

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dag.vue

* Update dag.vue

* Update dag.vue

* Update CreateWorkflowLocator.java

* Revert "Setting '-XX:+DisableExplicitGC ' causes netty memory leaks"

This reverts commit 3a2cba7a

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* update test unit

* add test unit

* add test unit

* update e2e test

* update unit test

* update unit test

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-08-01 14:28:12 +08:00
xsbai
87eb7c8211
adds waterdrop plugin (#3073)
* add plugin waterdrop

* add plugin waterdrop

* add plugin waterdrop

* fix deploy mode error

* Update en_US.js

fix conflict

* Update en_US.js

fix conflict

* Update en_US.js

Add missing commas

* Update zh_CN.js

Add missing commas

* Update waterdrop.vue

Fix the no resource hint

* Update zh_CN.js

Add 'Please select the waterdrop resources'

* Update en_US.js

Add  'Please select the waterdrop resources'

* Update waterdrop.vue

fix bug when click the waterdrop node again, it still show the `client` mode in deploy mode

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 23:20:59 +08:00
RedemptionC
0c5754c3f2
add feature : query top n success process instance order by running duration (#3315)
* add feature : query top n success process instance order by running duration

* add feature : add params check

* add feature : remove code smell

* add feature : remove code smell

* add feature : add param check

* add feature : add param check
improve handleEscape method in ParameterUtils.java

* top n : add  java doc and format code
2020-07-30 18:38:59 +08:00
zixi0825
5fe31546ba
[Fix][all] fix spell error (#3289)
* fix spell error

* test unit test

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-30 18:33:16 +08:00
felix.wang
b555d0a086
[Improvement][common] HttpClient singleton mode #3332 (#3350)
* Optimize the httpclient

* Optimize the httpclient singleton

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java
2020-07-30 13:24:54 +08:00
felix.wang
e026d303ea
Optimize the httpclient (#3337) 2020-07-29 19:03:44 +08:00
Zhou Zheng
3fceee9570 从强制成功的节点处启动-apiserver部分
1. 新添加executeType
2. 新添加对应的commandType
3. 修改controller对应的service部分
4. 补充processService中的verifyIsNeedCreateCommand的判断逻辑
2020-07-23 22:50:32 +08:00
Zhou Zheng
d86ceeecbe [implement] change task state 2020-07-23 21:54:48 +08:00
t1mon
07c8193bb5
[Feature-2930][api] Displays path and host on the instance log panel. #2930 (#3154)
* Optimize PropertyUtils instantiation.

* Fix info error.

* [Feature-2930][api] Displays path and host on the instance log panel. #2930

* [update] Add log-head in download log.

* [update] clear code smell.
2020-07-23 10:57:56 +08:00
lgcareer
f854c5f794
[Feature-2815]cherry pick commit from dev for one worker can belong different worker groups (#3277)
* [Feature-2815]cherry pick commit from dev for one worker can belong to different worker groups

* fix 2975:download resource need find the tenant code of resource owner

* change NetUtils to OSUtils because NetUtils didn't exist in the 1.3.2-release branch

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-22 19:39:03 +08:00
CalvinKirs
38e7494f78
replace CollectionUtils deprecated method (#3027)
* replace deprecated method

* replace deprecated method

* replace deprecated method

* replace deprecated method

* replace deprecated method

* messy

* messy

* messy

* add missing license

Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 19:42:33 +08:00
CalvinKirs
6f9970b189
[Feature-2574][Server]Specify Network Interface (#3186)
* [Feature-2574][Server]Specify Network Interface
This closes #2574

* code smell

* code smell

* move DOLPHIN_SCHEDULER_PREFERRED_NETWORK_INTERFACE to Constants.java
2020-07-13 18:51:38 +08:00
Yichao Yang
b4af3fd176
[Feature-2815][server] One worker can belong to different workergroups (#2934)
* [Feature-2815][server] One worker can belong to different workergroups

* Feature: Add test cases

* Update MonitorService.java

* Update MonitorService.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 17:25:13 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
muzhongjiang
c49626b877
Word spelling modification,Comment modification,Log level modification (#3180)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

* Word spelling modification
Comment modification
Word spelling modification,Comment modification,Log level modification

* Update EmailManager.java

* Update FlinkParameters.java

* Update SqlTask.java

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-12 22:20:14 +08:00
rockxsj
dcdd7dedd0
Modify the AWS S3 request encryption method to V4. (#3182)
Co-authored-by: rockxsj <junjun@mgtv.com>
2020-07-11 08:56:38 +08:00
felix-thinkingdata
1e7582e910
#3176 optimize Gets the value of this property “resource.storage.type”, Comparison with enumerated types (#3178)
* fix bug #3165   get resource.storage.type value   toUpperCase

* fix bug #3176  optimize Gets the value of this property “resource.storage.type”

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-10 15:21:42 +08:00
felix-thinkingdata
a23a3e2d12
fix bug #3165 get resource.storage.type value toUpperCase (#3166)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-09 18:28:04 +08:00
liukunyuan
0692dca4e6
update CommonUtils.java (#3137)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-05 21:50:42 +08:00
t1mon
83589c7b7c
Optimize kerberos tgt expire time in HadoopUtils. (#2700)
* Optimize PropertyUtils instantiation.

* Fix info error.

* Optimize kerberos tgt expire time.

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-04 18:22:30 +08:00
lenboo
572b142a53 [maven-release-plugin] prepare for next development iteration 2020-07-04 16:10:37 +08:00
lenboo
a4272efa55 [maven-release-plugin] prepare release 1.3.1 2020-07-04 16:04:36 +08:00
lenboo
fd3485ad0a revert version to 1.3.1 2020-07-04 15:54:37 +08:00
dailidong
4a4e940379
fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)
* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method
2020-07-04 15:49:46 +08:00
lenboo
3d4d816349 [maven-release-plugin] prepare for next development iteration 2020-07-03 15:56:45 +08:00
lenboo
2ff94a4362 [maven-release-plugin] prepare release 1.3.1 2020-07-03 15:55:56 +08:00
CalvinKirs
d1282fdd96
In the case of multiple network cards, a valid host is returned (#2924)
* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* In the case of multiple network cards, a valid host is returned

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix LOCAL_ADDRESS not assigned  value

* fix lock error

* fix lock error

* end

* end

* end

* end

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* code smell

* lowering

* messy

* small change

* compile error

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
2020-07-02 19:16:37 +08:00
muzhongjiang
a9eb7b18b0
[bugfix] fixed "jackson enum conversion : InvalidFormatException" (#3078)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-02 18:07:57 +08:00
dailidong
7445fa946b fix singe resource manager will NPE (#3013)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java
2020-06-30 14:11:07 +08:00
lenboo
fd76f172c1 Merge remote-tracking branch 'upstream/dev' into dev-merge
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
2020-06-30 11:17:14 +08:00
gaojun2048
58897aa2c5 merge 1.3.0-release branch code 2020-06-29 18:42:26 +08:00
gaojun2048
3cf9afc195 merge 1.3.0-release branch code 2020-06-29 18:42:18 +08:00
bao liang
31b593d023 change max length of user name to 39 (#2856)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-29 17:39:41 +08:00
lgcareer
a9cbca1e55 add source license full text into source license (#2840)
* add source license full text into source license

* exclude ScriptRunner.java on rat check
2020-06-29 17:39:38 +08:00
qiaozhanwei
5c9123f9ae blank in eamil and left font align #2648 (#2673)
* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:39:17 +08:00
dailidong
88bdb20a91 Update HttpUtils.java 2020-06-29 17:38:48 +08:00
qiaozhanwei
a085aa8fa3 modify UT 2020-06-29 17:38:23 +08:00
qiaozhanwei
cc3ab0f191 timing online modify (#2296)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

* master fault tolerant bug modify

* UT add pom.xml

* timing online  modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:37:58 +08:00
songgg
d8c7a44e82 some annotation optimizations (#2333)
Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:38 +08:00
lgcareer
82d6dc9e52 new feature for #404 add resource tree function (#2323)
* add create resource directory

* add create resource directory

* update the resource test

* add upgrade sql in version 1.2.2

* Adding request parameter id to update queryResourceListPaging

* set isDirectory value is false default

* add full name to update updateResource

* remove request parameter isDirectory to update createResource method

* update queryResourceListPaging with change get to post

* update updateResource method with remove fullName

* File management list modification (#1976)

* add resource component

* add resource tree visitor

* return json string

* update queryResourceList

* upload file need fullName

* add method rootNode

* Shell task resources and authorization resources (#1989)

* File management list modification

* Shell task resources and authorization resources

* download resource when execute task

* download resource when execute task

* update authorization type

* download resource when execute task

* Spark task resource changes (#1990)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* download resource when execute task

* update udf function service

* add resource type in ResourceComponent

* UDF resource tree and change DAG style (#2019)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* add deleteIds method in ResourceMapper and ResourceMapperTest

* Add comments on class and method

* add queryResourceByName method in controller

* update verify-name with change name to full name

* update queryResource with add parameter pid

* update queryResource with add parameter pid

* add resource ids in process definition and delete resource need judge whether it is used by any process definition

* Breadcrumb development (#2033)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix (#2040)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Fix github action rerun failed (#2067)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Change crumb position

* Change crumb position (#2068)

* build resource process definition map

* UDF changed to multiple choice

* UDF changed to multiple choice (#2077)

* Change crumb position

* UDF changed to multiple choice

* build resource process definition map (#2076)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* update resource name also need update all the children full name (#2096)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Limit customization file content to no more than 3000 lines(#2128) (#2140)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList (#2192)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* Modify the main jar package

* Modify the main jar package (#2200)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Modify the main jar package

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* choose main jar with resource tree (#2220)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* getResouDelete checksum and modify parameter namerceId

* revert .env

* remove parameter pid

* Delete request interface

* go back to the last page

* jar interface call

* Fix issue #2234 and #2228

* change resource name with full name

* Fix issue #2234 and #2228 (#2246)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* revert .env

* remove parameter pid

* Fix issue #2234 and #2228

* change resource name with full name

* Fix list query value error

* remove unauth-file with authorize-resource-tree

* Repair data cannot be echoed

* Repair data cannot be echoed

* execute mr and spark task need query resource name before

* Authorized resource interface replacement

* Authorized resource interface replacement

* Filter UDF resources

* Change parameters

* need query all authorized directory children when create task

* Change normalize.scss import method and animation.scss license modification

* Delete file list update processing

* It's fixed that resource not deleted in hdfs when delete it.

* add tooltips

* add tooltips (#2310)

* Echo workflow name and modify udf management name

* [new feature]add resource tree function

* revert front code in order to be same as dev branch

* revert front code in order to be same as dev branch

* revert common.properties and application.properties

* add super method

* update flink parameter test

* update flink parameter and unit test

* update resource service test

* If resource list is empty,need init it

* update flink parameter test

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-29 17:36:36 +08:00
gabry.wu
f9c91b8cbe Adapting partial code(file name start with S #2) to the sonar cloud rule (#2270) 2020-06-29 17:36:36 +08:00
gabry.wu
30b98b89a2 Adapting partial code(file name start with T) to the sonar cloud rule (#2271) 2020-06-29 17:36:33 +08:00
songquanhe-gitstudy
fc18d4a216 fix: No operations allowed after statement closed when running sql task (#2265)
* sqlTask failed to run

* get correct attribute value and logger content

Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:30 +08:00
gabry.wu
e1771de45a Adapting partial code(file name start with R) to the sonar cloud rule (#2243) 2020-06-29 17:36:29 +08:00
Simon
5c30205574 support custom datax configuration (#2224)
* fix #1441

* support custom datax config

* support datax custom config

* support datax custom config

* support datax custom config

Co-authored-by: 张世鸣 <zhangshiming@cvte.com>
2020-06-29 17:36:29 +08:00
ruson
2bb55e4993 add repace time when run history job and batch complement job ,not current time (#2196)
* add release notes in 1.2.1 (#1654)

* fix Monitor bug (#1656)

* fix Monitor bug

* fix zk monitor bug

* fix api url (#1673)

* get root path from zookeeper config

* Fixed DAG zoom in and zoom out nodes separated from arrows #1679 (#1685)

* fix api url

* Fixed DAG zoom in and zoom out nodes separated from arrows #1679

* check license and update (#1725)

* Fixed tasks_queue and tasks_kill did not exist in zookeeper #1696 (#1734)

Co-authored-by: elonlo <gamelifedong@gmail.com>

* upgrade jackson from 2.9.8 to 2.10.1 (#1767)

* add out put log when master/worker server start (#1769)

* merge hadoop.properties into common.properties

* merge hadoop,zookeeper.properties into common.properties
remove combined.properties/master.properties/worker.properties

* change db user/pwd to test/test

* rename .dolphinscheduler_en.sh to dolphinscheduler_env.sh
remove some unused in install.sh

* add out put log when master/worker server start...

* add start log when servers start

* add check download resource permission in order to fix issues 1770 (#1788)

* merge dev branch front-end code (#1786)

* fix #1775 bug,delete process definition when process instance is running (#1790)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* fix udfs assignment and task instance page jump #1789 (#1791)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* update method checkDownloadPermission in order to fix issues 1770 (#1794)

* add check download resource permission in order to fix issues 1770

* update method checkDownloadPermission in order to fix issues 1770

* update method listUnauthorizedResource in order to fix issues 1770

* update method listUnauthorizedResource in order to fix issues 1770 (#1797)

* if login user is admin,it will has permission to query all udf function (#1799)

* Fixed space and icon display issues before and after the input box (#1798)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* donot submit task to queue if sub process (#1793)

* dockerfile modify (#1800)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* Fix data echo, style and popup cannot be closed (#1801)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value(#1810) (#1815)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* [Fix issue #1770]check udf and data source in order to fix issue 1770 (#1817)

* if login user is admin,it will has permission to query all udf function

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* revert common.properties

* update the test method name

* remove useless code and import in unit test

* refactor has permission and check permission

* #1813 remove "_001" after the master/server register path in zookeepe (#1824)

* donot submit task to queue if sub process

* [feature] #1813 remove "_001" after the master/server register path in zookeeper (#1820)

* change master/worker register path.

* remove "_" from register path.

* remove install.sh server.servlet.context-path replace (#1823)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface (#1827)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface

* rpm package add UI (#1846)

* [fix #1828] when the executor of process instance is not the owner of udf resouce, the path of the read resource file is incorrect (#1847)

* fix issue 1828:get the udf resource path error when create udf function

* update grantResources

* first verify whether udf resource is bound by udf function

* update grantResources

* update testListAuthorizedUdfFunc

* update getUserInfo in order to run success

* add LoggerServer RPC PORT modified. #1848 (#1849)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* add LoggerServer RPC PORT modified. #1848

* LoggerService Logger RPC PORT get Error modify . #1848 (#1857)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* add LoggerServer RPC PORT modified. #1848

* LoggerService Logger RPC PORT get Error modify . #1848

* Fix UDF function list delete data without refresh and page data display widened #1851 (#1852)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface

* Fix UDF function list delete data without refresh and page data display widened

* [Fix #1828]check whether has permission to download udf file or delete udf function (#1858)

* fix issue 1828:get the udf resource path error when create udf function

* update grantResources

* first verify whether udf resource is bound by udf function

* update grantResources

* update testListAuthorizedUdfFunc

* update getUserInfo in order to run success

* check whether has permission to download udf file or delete udf file

* update listAuthorizedResourceById in ResourceMapper.xml

* add log, don't swallow exception info (#1877)

* Added hints in Resource Center (#1891)

* update jackson version from 2.9.8 to 2.10.1 (#1892)

* update release notes and modify the plugin of rat (#1899)

* modify how to build (#1902)

* update release notes and modify the plugin of rat

* modify how to build

* [maven-release-plugin] prepare release 1.2.1

* [maven-release-plugin] prepare for next development iteration

* get authorized udf resources need exclude all created by self (#1905)

* get authorized udf resources need exclude all created by self

* reset pom version to 1.2.1-SNAPSHOT

* Update ReleaseNotes.md

add detail info

* Update ReleaseNotes.md

* update notice year form 2019 to 2019-2020 (#1907)

* [maven-release-plugin] prepare release 1.2.1

* [maven-release-plugin] prepare for next development iteration

* 当手动重跑历史活着补数的时候对shell和sql脚本中的[YYYYmmddd...]变量赋值与传递的日期,而不是当前日期

Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: bao liang <29528966+lenboo@users.noreply.github.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: elonlo <gamelifedong@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: zhangchunyang <18910529250@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: 谷雨 <lixun@guyudeMacBook-Pro.local>
2020-06-29 17:36:28 +08:00
t1mon
9c754378f3 Optimize PropertyUtils instantiation. (#2199)
* Optimize PropertyUtils instantiation.

* Fix info error.
2020-06-29 17:36:28 +08:00
tswstarplanet
585f833549 change static field to instance field (#2190) 2020-06-29 17:36:28 +08:00
tswstarplanet
77c4bcb96e Cache HadoopUtils instance with specific days expire time (#2181)
* Cache HadoopUtils instance with 7 days expire time

* solve sonar issue

* add kerberos expire time config

* move KERBEROS_EXPIRE_TIME to Constants.java
2020-06-29 17:36:28 +08:00
gabry.wu
2f4fda7cdb Adapting partial code(file name start with M #1) to the sonar cloud rule (#2173)
* Adapting partial code(file name start with M #1) to the sonar cloud rule

* remove unused import
2020-06-29 17:36:28 +08:00
gabry.wu
7f7f50a828 Adapting partial code(file name start with L) to the sonar cloud rule (#2171) 2020-06-29 17:36:28 +08:00
gabry.wu
bdfb9d6a50 Adapting partial code(file name start with J) to the sonar cloud rule (#2165)
* Adapting partial code(file name start with J) to the sonar cloud rule

* fix unit failure
2020-06-29 17:36:27 +08:00
Rubik-W
8b61afc9be The master and worker server exit exception #2163 (#2176)
* fix: #2163

* fix: format
2020-06-29 17:36:27 +08:00
gabry.wu
610c95a470 use static method in static class JSON (#2156) 2020-06-29 17:36:27 +08:00
gabry.wu
33e86a97bf Adapting partial code(file name start with I) to the sonar cloud rule (#2155) 2020-06-29 17:36:27 +08:00
gabry.wu
156533ad5e Adapting partial code(file name start with H) to the sonar cloud rule (#2145) 2020-06-29 17:36:22 +08:00
tswstarplanet
6b6584ac8e use logger to print exception (#2151) 2020-06-29 17:36:22 +08:00
gabry.wu
493499f118 Adapting partial code(file name start with F) to the sonar cloud rule (#2045)
* Adapting partial code(file name start with F) to the sonar cloud rule

* add more unit test

* add License

* add includes configuration to maven-surefire-plugin

* fix getResourceFilesList incorrect logic
2020-06-29 17:36:22 +08:00
gabry.wu
0209cb7431 move UT class to reasonable package (#2116) 2020-06-29 17:36:13 +08:00
gabry.wu
fdc9e0090b remove incorrect or unnecessary URL tag (#2099) 2020-06-29 17:36:13 +08:00
Yichao Yang
38e485373d
[Feature-2925][server] Init TaskLogger in TaskExecuteProcessor (#2925) (#2965)
* [Feature-2925][common] Add exitVal judge in OSUtils.exeCmd (#2925)

* optimize the logger utils
2020-06-28 17:43:13 +08:00
lenboo
ea1c653e08 merge from dev 2020-06-22 10:21:15 +08:00
CalvinKirs
518d03b6a6
Simplify some methods for easy understanding (#2920)
Co-authored-by: gabry.wu <8545796+gabrywu@users.noreply.github.com>
2020-06-21 15:48:35 +08:00
lenboo
73cae7f046 merge from dev-1.3.0 2020-06-19 15:19:32 +08:00
dailidong
620ecffa72
fix singe resource manager will NPE (#3013)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java
2020-06-19 14:45:48 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
Yichao Yang
0744d4070c
[Hotfix-2922][common] Fix Jackson UDde error (#2922) (#2926)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-17 11:33:41 +08:00
Yichao Yang
0e7403e6a8
[Feature-2927][common] Fix NEF when get info from OSUtils (#2927) (#2964)
* [Feature-2927][common] Fix NEF when get info from OSUtils (#2927)

* Update OSUtils.java

Change java doc

Co-authored-by: gabry.wu <8545796+gabrywu@users.noreply.github.com>
2020-06-14 11:17:12 +08:00
simon824
3fa4dd702b Resolved the conflict 2020-06-13 09:47:18 +08:00
Simon
eebd0c73bb
Merge pull request #3 from apache/dev
merge
2020-06-13 09:26:40 +08:00
BoYiZhang
f56ae9e64a
add http task timeout & test unit (#2955)
* add http task timeout & test unit

* add http task timeout & test unit

Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:52:19 +08:00
lijufeng2016
284c50f66c
Compatible with flink1.10 or newer (#2952)
* 兼容flink1.10以上版本

* fix null point bug

Co-authored-by: 李巨丰 <lijf@2345.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-12 18:23:28 +08:00
simon824
1b7d8ee236 fix the bad code smell 2020-06-12 12:35:18 +08:00
simon824
24120a4641 fix the bad code smell 2020-06-12 11:27:42 +08:00
simon824
cfecfb799b 1. change windows new line 'CR' to linux new line 'LF'
2. solve conflict
2020-06-12 09:59:40 +08:00
Simon
0b11275880
Merge pull request #2 from apache/dev
merge
2020-06-12 09:22:58 +08:00
lenboo
1904a7022a change 1.3.0 to 1.3.0-snapshot 2020-06-10 11:42:15 +08:00
simon824
fad09075f9 remove fastjson maven dependence 2020-06-10 09:16:28 +08:00
eights
5d83d797d3 modify sqoop task UT 2020-06-10 09:11:17 +08:00
eights
c8f28ab2ba sqoop task optimization 2020-06-09 03:16:35 -07:00
simon824
8cabb7a907 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 14:15:13 +08:00
simon824
6e907f150c 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 13:02:50 +08:00
bao liang
422fe5136d
[license] add license and notice (#2916)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

* test for gc

* change version to 1.3.0

* check and add some license files

* check and add some license files

* revert

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-06 22:21:37 +08:00
Rubik-W
7c9f5960e5
Fixbug datax task (#2909)
* fix:  local param bug

* fix: UT bug

Co-authored-by: Rubik-W <whm_777@163.com>
2020-06-06 21:12:41 +08:00
simon824
572ee0fc64 Using Jackson instead of Fastjson 2020-06-04 16:23:59 +08:00
simon
8bab8abcd2 ut fix 2020-06-03 19:21:19 +08:00
simon
bb089288c4 Using Jackson instead of Fastjson 2020-06-03 12:40:28 +08:00
simon
dea2d4634d Using Jackson instead of Fastjson 2020-06-02 20:13:23 +08:00
simon
f26e246706 Using Jackson instead of Fastjson 2020-06-02 20:00:49 +08:00
张世鸣
e386af700d Using Jackson instead of Fastjson 2020-06-02 17:22:39 +08:00
张世鸣
a9f72c2081 Using Jackson instead of Fastjson 2020-06-02 15:54:34 +08:00
bao liang
41e4986eb2
change max length of user name to 39 (#2856)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-01 17:25:25 +08:00
Han Gao
8f8ddae030
Using Jackson instead of Fastjson (#2850)
* Using Jackson instead of Fastjson

* Fix some json bugs

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-31 18:11:10 +08:00
CalvinKirs
c8ab27acb0
Optimization of enumerated values method (#2677)
* The value method of enum is a deep copy. If there are many parameters, it is not recommended to use the value method. Specifically see jmh results

* beautiful misunderstanding

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-31 10:29:46 +08:00
itbasketplayer
1513aae3ce
add job history to judge application status/2625 (#2848)
* job history status url when application number threshold is reached(default 10000,maybe it was set to 1000)

* job history status url when application number threshold is reached(default 10000,maybe it was set to 1000)

* job history status url when application number threshold is reached(default 10000,maybe it was set to 1000)

Co-authored-by: yuhaibin@lizhi.fm <35716fc5847f6d154cf556296453ca91>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-30 23:07:17 +08:00
lgcareer
ef4c1e7930
add source license full text into source license (#2840)
* add source license full text into source license

* exclude ScriptRunner.java on rat check
2020-05-29 17:23:34 +08:00
lgcareer
fadb69b181
When update the data source,the host and port show error (#2821)
* When update the data source,the host and port show error

* update DataSourceServiceTest

* update DataSourceServiceTest

* update DataSourceServiceTest
2020-05-27 15:51:58 +08:00
qiaozhanwei
0130da2bdf
Fix [Bug] process definition json worker group convert #2794 (#2795)
* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-22 18:46:05 +08:00
tlhhup
396b1716e4
fix bug:#2735 (#2770)
1. store process id in zk.
2. resolve host and port through zk path.
3. modify ui key.
2020-05-22 11:20:54 +08:00
zixi0825
6d0375e46d
fix sql task bug #2739 (#2741)
* Solve the failure to execute non-query sql

* fix sql task bug

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: lgcareer <18610854716@163.com>
2020-05-21 16:09:36 +08:00
xuchangqun
cfd83235a2
Hourly dependency configuration, 'currentHour' and 'last24Hours' (#2696)
* [feature] Add hourly dependency scenarios

* [Feature] #2680

add DependentUtilsTest UT

Co-authored-by: changqun.xu <changqun.xu@17zuoye.com>
Co-authored-by: xuchangqun <changqun.x@gmail.com>
2020-05-21 14:28:10 +08:00
bao liang
184d64e852
[BUG FIX] fix bug: dependent task failed when conditions task exists (#2768)
* fix bug 2464: change dependent task for process.

* remove unused code

* add ut

* add ut

* update comments

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-05-21 10:34:28 +08:00
qiaozhanwei
2330cc8872
master select worker filter high load worker #2704 (#2733)
* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-19 11:38:41 +08:00
gabry.wu
772a8e2b37
add retryCallSilent to RetryerUtils making it easy to use (#2729)
* add retryCallSilent to RetryerUtils making it easy to use

* add more detail to java doc of retryCallSilent
2020-05-19 10:18:21 +08:00
bao liang
a9da86abb4
[bug fix] fix bug:2634:Process shows success,when the task of the process is killed in the terminal (#2731)
* fix bug: some bugs about worker group from 1.2.0 to 1.3.0

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix ut bug

* fix bug: Process shows success,when the task of the process is killed in the termina

* fix bug #2634: Process shows success,when the task of the process is killed in the terminal

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-05-18 18:27:16 +08:00
dailidong
b51c999552
update worker group config (#2746)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* Update datasource.properties
2020-05-18 18:04:24 +08:00
break60
d588d62620 Modify ip configuration 2020-05-18 16:16:04 +08:00
xingchun-chen
7010bb2ef2
Modify HDFS/S3 path (#2736)
* Modify HDFS/S3 path
 

* Modify HDFS/S3 path
 

* Update common.properties

Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-18 15:35:11 +08:00
gabry.wu
1771aa50d5
change variable name ,make it easy to understand (#2647)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-14 19:24:49 +08:00
gabry.wu
c28a697ee8
add RetryerUtils to retry async & sync function gracefully (#2697)
* add RetryerUtils tool class for async & sync retry invoking

* add RetryerUtils tool class for async & sync retry invoking

* add guava-retrying-2.0.0.jar to known-dependencies.txt

* fix javadoc

* fix code smell

Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-05-14 17:51:50 +08:00
qiaozhanwei
83ede1037e
blank in eamil and left font align #2648 (#2673)
* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-11 14:49:29 +08:00
JinyLeeChina
daea87a515
'ExecutionStatus' (#2635)
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-05-11 14:43:49 +08:00
s751167520
cb5285384e
Fix some code specification issues (#2482)
* Fix some code specification issues

* Pre-compile regular expressions to improve efficiency and solve concurrency problems

* Solve the nullpoint problem that Object.equals may bring

* Modify if else structure, add {} to improve code readability

Co-Authored-By: daili <daili@users.noreply.github.com>

* Update ResourceTreeVisitor.java

re-run test process

Co-authored-by: daili <daili@users.noreply.github.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-10 23:05:12 +08:00
dailidong
8493aa8209
simplify common config (#2613)
* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

* simplify common config

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
2020-05-08 16:08:08 +08:00
dailidong
6a92bef5e3
add development state config (#2609)
* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
2020-05-06 18:38:19 +08:00
hgaol
906ffa09a9 resolve code checks 2020-04-30 15:35:18 +08:00
hgaol
3c6d813617 add license to plugin api pom file 2020-04-30 15:35:18 +08:00
hgaol
85f3ac10a0 feat: add plugin management for alert service 2020-04-30 15:35:18 +08:00
lgcareer
fd79be39a9
verify whether the filename start with '/' (#2544)
* fix #2442 when the resource name contains '$',need translate it to '\$'

* fix #2442 when the resource name contains '$',need translate it to '/$'

* verify whether the filename start with '/'

Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-04-28 11:23:28 +08:00