Commit Graph

74 Commits

Author SHA1 Message Date
rockxsj
b6a0d55949
fix and update k8s deploy (#3215)
* Modify the AWS S3 request encryption method to V4.

* * update Dockerfile to add mysql-client for mysql database service test
* update wechat alert config replace placeholder from  to {xx} because of the  will be treat as env variable
* fix unit test of EnterpriseWeChatUtilsTest
* update values.yaml default config for helm deploy DS cluster
* update values.yaml default config for custom install
* update templates of helm for custom install
* update some name of env varibale, for example change from POSTGRESQL_HOST to DATABASE_HOST
* delele the redundancy requirements.yaml for heml
* update README* in the docker dir and subdir
* update startup scripts for support mysql as backend database

* * fix the ENV name wrong in docker-compose.yml and docker-stack.yml
* remove the didn't need port mapping in postgresql & zookeeper & dolphinscheduler-api

* * keep expose the port of postgresql & zk & api because of unit test needed

* * remove mysql-client package from dockerfile

Co-authored-by: rockxsj <junjun@mgtv.com>
2020-07-16 16:36:29 +08:00
johnnygo920
1dfde72452
[BUG#2359][Alert] Mail Chinese name is too long garbled (#3199)
* 解决中文附件名过长时邮件附件中文乱码问题

* Solve the problem of messy Chinese name in excel attachment

* SonarCloud Code Analysis is not running . retry commit

* Update MailUtils.java

Co-authored-by: johnnygo920 <W@ngchen2020>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-14 23:22:42 +08:00
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
muzhongjiang
c49626b877
Word spelling modification,Comment modification,Log level modification (#3180)
* add state

* fixed bug "jackson enum conversion  : InvalidFormatException"

* Word spelling modification
Comment modification
Word spelling modification,Comment modification,Log level modification

* Update EmailManager.java

* Update FlinkParameters.java

* Update SqlTask.java

Co-authored-by: mzjnumber1@163.com <mzjnumber1@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-12 22:20:14 +08:00
lenboo
8e7bdfa276 update ut 2020-06-22 10:17:48 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
simon824
24120a4641 fix the bad code smell 2020-06-12 11:27:42 +08:00
lenboo
1904a7022a change 1.3.0 to 1.3.0-snapshot 2020-06-10 11:42:15 +08:00
simon824
fad09075f9 remove fastjson maven dependence 2020-06-10 09:16:28 +08:00
simon824
6e907f150c 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 13:02:50 +08:00
bao liang
422fe5136d
[license] add license and notice (#2916)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

* test for gc

* change version to 1.3.0

* check and add some license files

* check and add some license files

* revert

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-06 22:21:37 +08:00
simon
bb089288c4 Using Jackson instead of Fastjson 2020-06-03 12:40:28 +08:00
simon
f26e246706 Using Jackson instead of Fastjson 2020-06-02 20:00:49 +08:00
simon
39d2713d0a Using Jackson instead of Fastjson 2020-06-02 18:19:21 +08:00
张世鸣
e386af700d Using Jackson instead of Fastjson 2020-06-02 17:22:39 +08:00
张世鸣
a9f72c2081 Using Jackson instead of Fastjson 2020-06-02 15:54:34 +08:00
Han Gao
8f8ddae030
Using Jackson instead of Fastjson (#2850)
* Using Jackson instead of Fastjson

* Fix some json bugs

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-31 18:11:10 +08:00
gabry.wu
1771aa50d5
change variable name ,make it easy to understand (#2647)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-14 19:24:49 +08:00
xiaozhaoying
de3a81cab5
support ali nail(ding talk) , issue:2458 (#2594)
* Create TextMessage.java

* Create DingTalkUtils.java

* Update alert.properties

* Update Constants.java

* Update Constants.java

* Create DingTalkMsgFormatter.java

* Update alert.properties

* Create DingTalkMsgFormatterTest.java

* Create DingTalkUtilsTest.java

* Update DingTalkUtils.java

* Create DingTalkManager.java

* Update EmailAlertPlugin.java

* feature: test send msg with proxy

* fix:delete comment and add exception info

* Update DingTalkUtilsTest.java

* Update DingTalkUtils.java

* Update DingTalkManager.java

* Update EmailAlertPlugin.java

* Update DingTalkUtilsTest.java

* Update DingTalkMsgFormatter.java

* Update DingTalkUtils.java

* Update DingTalkMsgFormatterTest.java

* Update DingTalkUtilsTest.java

* Update TextMessage.java

* Update DingTalkManager.java

* Delete TextMessage.java

* fix: add test case

* fix:add http client test case

* fix:delete ignore test case

* Update pom.xml

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-14 10:18:22 +08:00
bao liang
d99471806e
[bug fix] some bugs from 1.2.0 to 1.3.0 (#2676)
* fix bug: some bugs about worker group from 1.2.0 to 1.3.0

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix ut bug

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-05-12 16:46:16 +08:00
qiaozhanwei
83ede1037e
blank in eamil and left font align #2648 (#2673)
* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-11 14:49:29 +08:00
chenxingchun
d37e843cd0 Modify the alert log file 2020-05-11 14:13:28 +08:00
hgaol
906ffa09a9 resolve code checks 2020-04-30 15:35:18 +08:00
hgaol
85f3ac10a0 feat: add plugin management for alert service 2020-04-30 15:35:18 +08:00
qiaozhanwei
f2bfdc14f7
#2499 bug fix (#2505)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-23 17:37:04 +08:00
qiaozhanwei
e837a73fcd
EnterpriseWeChatUtils modify (#2502)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-23 16:54:54 +08:00
qiaozhanwei
28766df53f
send mail error, #2466 bug fix (#2470)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-21 14:40:19 +08:00
dailidong
eec92ed1fc
simplify and optimize config (#2469)
* [Refactor worker] simplify and optimize config (#2386)

* simplify config

* simplify config

* simplify and optimize config

* Update HadoopUtils.java

optimize HadoopUtils

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HttpUtils.java

* Update pom.xml

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java
2020-04-21 02:25:45 +00:00
qiaozhanwei
94e8bb15c9 Merge remote-tracking branch 'remotes/upstream/dev' into dev 2020-04-01 16:20:30 +08:00
Rubik-W
f60edc8e61
fix:mail send fail #2345 (#2346)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-03-31 21:58:15 +08:00
qiaozhanwei
152c814126 test 2020-03-31 16:51:19 +08:00
dailidong
ace907e7dd
[refactor-worker] simplify master、 worker、alert、dao properties (#2304)
* update logback

* update log

* refactor worker registry (#2107)

* Refactor worker (#2115)

* refactor worker registry

* refactor master server

* Modify workgroupid parameter name (#2105)

* Delete worker group management page

* Modify workgroupid parameter name

* Refactor worker (#2121)

* refactor worker registry

* refactor master server

* refactor MasterSchedulerService

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* not exist in openjdk,just delete

* add master and worker properties

* add master and worker properties

* add master and worker properties

* fix cpu 100% bug

* simplify master、 worker、alert、dao properties

* add master and worker properties

* add master and worker properties

* add master and worker properties

Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-25 15:16:42 +08:00
tswstarplanet
3c5227ac0f
Refactor ConnectionFactory to static inner holder Singleton (#2204)
* refactor ConnectionFactory to static inner holder Singleton

* remove redundant import

* make getSqlSessionFactory() method private

* fix sonar issue

* remove @Ignore of MailUtilsTest

* add MailUtilsTest to pom.xml to test

* fix MailUtilsTest path error in pom.xml

* modify test method name

* add assert and logger to unit test

* add log

* add log

* add AlertDaoTest

* move AlertDaoTest to new module

* modify test in pom.xml

* remove unnecessary log
2020-03-18 23:48:19 +08:00
gabry.wu
fcc57320d6
Adapting partial code(file name start with M #1) to the sonar cloud rule (#2173)
* Adapting partial code(file name start with M #1) to the sonar cloud rule

* remove unused import
2020-03-15 19:05:33 +08:00
gabry.wu
a076ae4c36
Adapting partial code(file name start with J) to the sonar cloud rule (#2165)
* Adapting partial code(file name start with J) to the sonar cloud rule

* fix unit failure
2020-03-14 18:12:14 +08:00
gabry.wu
daad5efbab
use static method in static class JSON (#2156) 2020-03-13 23:53:22 +08:00
t1mon
46e30b1e50
Add mail server configuration description information (#2143)
* Add mail server configuration description information

* Remove mail server configuration note in alert.properties
2020-03-12 17:50:37 +08:00
tswstarplanet
afb907b4b0
fix unequal number of arguments and placeholder (#2149) 2020-03-12 17:41:10 +08:00
gabry.wu
ebf7a980d2
Adapting partial code(file name start with F) to the sonar cloud rule (#2045)
* Adapting partial code(file name start with F) to the sonar cloud rule

* add more unit test

* add License

* add includes configuration to maven-surefire-plugin

* fix getResourceFilesList incorrect logic
2020-03-10 18:59:45 +08:00
dailidong
f551913249
split logback config into seperate child module logback (#2120)
* update logback

* update log
2020-03-09 19:03:51 +08:00
gabry.wu
450a1f56fc
Adapting partial code(file name start with E) to the sonar cloud rule (#1999)
* Adapting partial code(file name start with E) to the sonar cloud rule

* remove isEmpty invoke

* resolve conflicts
2020-03-08 21:28:49 +08:00
qiaozhanwei
3e5a939913
delete unused import (#2066) 2020-03-05 11:07:44 +08:00
gabry.wu
44b76468a8
Adapting partial code(file name start with C) to the sonar cloud rules (#1918)
* Adapting partial code(file name start with C) to the sonar cloud rules

* add some UT

* add some UT(rollback JSONUtils.checkJsonVaild)

* add some UT(fix OSUtils.exeCmd UT)
2020-02-13 12:42:08 +08:00
khadgarmage
c447bb489f add stringutils ut (#1921)
* add stringutils ut
2020-02-10 13:52:32 +08:00
dailidong
c63cd3e237
combine logback config of master/worker/alert/api server to one (#1898)
* combine logback config of master/worker/alert/api server to one logback.xml
2020-02-06 11:08:06 +08:00
Jave-Chen
c522ea7ebb Fix bug: NullPointerException etc. (#1721)
* #1720 Fix bug: NullPointerException etc.

* fix code smell

* Update DateInterval.java

refactor equals method

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-01-17 16:55:01 +08:00
ZHAOLIPAN
68f736953b fix typos (#1821) 2020-01-14 16:15:34 +08:00
Yelli
8399ab4718 add null check in markdownTable & modify EnterpriseWeChatManager logger class (#1809) 2020-01-14 14:36:38 +08:00
Yelli
1a86e7bae7 add retMap null check for AlertSender (#1796)
* fix bug: zk hasTask method NPE

* add retMap null check for AlertSender
2020-01-10 16:33:20 +08:00
DK.Pino
9eb823c0aa Refactor alert template (#1713)
* refactor Alert Template

* refactor alert template(remove freemarker)

* add license

* update abstract class to interface

* make test into root pom

* fix powermock static

* fix powermock static

* remove custom exception

* remove custom exception
2020-01-07 10:20:58 +08:00