Commit Graph

198 Commits

Author SHA1 Message Date
bao liang
657ea4752f
merge from 1.3.1-release (#3161)
* merge from 1.3.1-release

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* remove some standard notices in order to cut down the NOTICE file. (#3101)

* #3084 remove some standard notices in order to cut down the NOTICE file.

* update exclude element sql/soft_version of rat plugin

* update soft_version

* merge from 1.3.1-release

* fix 3112:download udf resource need find the tenant code of resource owner (#3114)

* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code

* merge from 1.3.1-release

* remove some notes (#3127)

Co-authored-by: lenboo <baoliang@analysys.com.cn>

* fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)

* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method

* add PropertyPlaceholderHelper on license (#3129)

* update license header location

* add PropertyPlaceholderHelper on license

* merge from 1.3.1-release

* change from 1.3.0  to 1.3.2

* add Set import

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-13 16:35:42 +08:00
LEI SHENG
98fdba6740
[Bug 2923] Hive JDBC connection parameter ignored (#3194)
* fix the bug #2923 (Hive JDBC connection parameter ignored) , the hive jdbc connection url ;

* fix the bug index out of bound and add some test case

* add the Licensed

* fix the checkstyle and import StringBuilder

* put HiveDataSourceTest.java to root pom maven-surefire-plugin to avoiding the sonar 0.0% Coverage.

* Update HiveDataSource.java

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-13 13:58:01 +08:00
lenboo
b81872cacd merge from dev-1.3.0 2020-06-19 18:52:30 +08:00
lenboo
714ae33e40 merge from dev 2020-06-19 14:20:24 +08:00
743294668
a58d1b3b45
Modify OracleDB connection string format. (#2970)
Co-authored-by: taoyuxin <taoyuxin@inspur.com>
2020-06-15 11:12:22 +08:00
simon824
24120a4641 fix the bad code smell 2020-06-12 11:27:42 +08:00
lenboo
1904a7022a change 1.3.0 to 1.3.0-snapshot 2020-06-10 11:42:15 +08:00
simon824
fad09075f9 remove fastjson maven dependence 2020-06-10 09:16:28 +08:00
simon824
6e907f150c 1. specified jackson Date format.
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer

rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
2020-06-09 13:02:50 +08:00
bao liang
422fe5136d
[license] add license and notice (#2916)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

* test for gc

* change version to 1.3.0

* check and add some license files

* check and add some license files

* revert

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-06 22:21:37 +08:00
simon824
9a92ab7982 Using Jackson instead of Fastjson 2020-06-04 17:56:59 +08:00
simon824
9ada3c7aac Using Jackson instead of Fastjson 2020-06-04 17:20:24 +08:00
simon824
32863755ec Using Jackson instead of Fastjson 2020-06-04 16:56:31 +08:00
simon824
4197ee8a8f Using Jackson instead of Fastjson 2020-06-04 16:46:11 +08:00
simon824
572ee0fc64 Using Jackson instead of Fastjson 2020-06-04 16:23:59 +08:00
lgcareer
4441d91dcf
Fix bug:If user didn't set the value of tenant,release process definition fail (#2885)
* copy resources need get top directory first

* copy resources need get top directory first

* fix #2860:copy files and create empty directory

* Fix bug:If user didn't set the value of tenant,release process definition online fail
2020-06-03 13:50:06 +08:00
simon
f26e246706 Using Jackson instead of Fastjson 2020-06-02 20:00:49 +08:00
张世鸣
e386af700d Using Jackson instead of Fastjson 2020-06-02 17:22:39 +08:00
张世鸣
a9f72c2081 Using Jackson instead of Fastjson 2020-06-02 15:54:34 +08:00
Yelli
28e28caebd
fix table not exist when init db metadata (#2838)
* fix table not exist when init db metadata

* fix db init error

* fix ds-1.3.0 init db

* init db modify process instance json

Co-authored-by: eights <yelli.hl@foxmail.com>
2020-05-29 21:51:19 +08:00
lgcareer
fadb69b181
When update the data source,the host and port show error (#2821)
* When update the data source,the host and port show error

* update DataSourceServiceTest

* update DataSourceServiceTest

* update DataSourceServiceTest
2020-05-27 15:51:58 +08:00
lgcareer
5b633cf412
fix oracle connect failure (#2812)
* use method getJdbcUrl of parent

* Remove the incorrect print message
2020-05-25 19:46:39 +08:00
qiaozhanwei
d0fbfeaf17
process definition json worker group convert #2794 (#2809)
* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-25 17:18:15 +08:00
qiaozhanwei
0130da2bdf
Fix [Bug] process definition json worker group convert #2794 (#2795)
* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-22 18:46:05 +08:00
bao liang
3d857bba92
[BUG FIX] fix bug:cannot pause work flow when task state is "submit success" (#2783)
* feature: add number configuration for master dispatch tasks

* fix bug(#2762) the master would be blocked when worker group not exists

* fix bug(#2762) the master would be blocked when worker group not exists

* fix ut

* fix ut

* fix bug(2781): cannot pause work flow when task state is "submit success"

* fix code smell

* add mysql other param blank judge

* test

* update comments

* update comments

* add ut

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-05-22 10:42:24 +08:00
Tboy
86f4276f0f
Fix MySQL autoDeserialize bug (#2728)
* Fix MySQL autoDeserialize bug

* add MySQLDataSourceTest
2020-05-19 18:48:58 +08:00
dailidong
b51c999552
update worker group config (#2746)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* Update datasource.properties
2020-05-18 18:04:24 +08:00
lgcareer
f52bad5e8c
fix #2449:It's should successful to cancel authorize a resource of a user if all the online process definitions created by this user didn't use this resource. (#2725) 2020-05-18 11:05:12 +08:00
liwenhe1993
29a353e689
Using docker/docker-swarm/docker-compose.yml to run pg and zk in ci_ut.yml (#2692)
* Using docker/docker-swarm/docker-compose.yml to run pg and zk in ci_ut.yml

1. add initdb volume for pg service in docker-compose.yml
2. modify ci_ut.yml

* add sudo in ci_ut.yml

* Query the metadata of indexes is not correct in postgresql/dolphinscheduler_ddl.sql

1. replace information_schema.STATISTICS to pg_stat_all_indexes
2. modify ProcessDefinitionMapperTest.java

* Remove inner zk and pg

* add inner pg
2020-05-12 20:24:17 +08:00
Rubik-W
68e3487e34
feature: Add user state #1923 (#2424)
* feature: #1923

* fix:docker sql script add column

* fix: update sql script

* fix: UT bug

* fix: UT bug

* fix: UT bug

* fix: e2e UT bug

* fix: e2e button location

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-11 11:45:29 +08:00
s751167520
cb5285384e
Fix some code specification issues (#2482)
* Fix some code specification issues

* Pre-compile regular expressions to improve efficiency and solve concurrency problems

* Solve the nullpoint problem that Object.equals may bring

* Modify if else structure, add {} to improve code readability

Co-Authored-By: daili <daili@users.noreply.github.com>

* Update ResourceTreeVisitor.java

re-run test process

Co-authored-by: daili <daili@users.noreply.github.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-10 23:05:12 +08:00
qiaozhanwei
bb52671fee
Worker Group display #2627 (#2630)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

* add comment

* revert comment

* revert comment

* #2499 buf fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* No master don't create command #2571

* No master don't create command #2571

* No master don't create command #2571

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

* Worker Group display #2627

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-05-08 15:43:11 +08:00
zixi0825
b08ba3461e
process batch export and import #2501 (#2560)
* Increase dataX environment variable, sslTrust default value (#2555)

* add LoginTest license

* Delete useless packages

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* Update worker_group_id to worker_group in init.sql

* Update worker_group_id to worker_group in init.sql

* Update worker_group_id to worker_group

* Increase dataX environment variable, sslTrust default value

Co-authored-by: chenxingchun <438044805@qq.com>

* no valid worker group,master can kill task directly (#2541)

* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* send mail error, #2466 bug fix

* #2486 bug fix

* host and workergroup compatible

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* EnterpriseWeChatUtils modify

* #2499 bug fix

* add comment

* revert comment

* revert comment

* #2499 buf fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* #2499 bug fix

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

* no valid worker group,master can kill task directly

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* add batch import and export process definition

* fix unit test bug

* add unit test

* add unit test 2

* fix duplicated code problem

* fix duplicated code problem 2

* fix duplicated code problem 3

* fix duplicated code problem 4

* fix the vulnerability bug

* fix the vulnerability bug

* fix the code smells

* fix the code smells 1

* fix the code smells 2

* fix the code smells 3

Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-05-06 16:33:45 +08:00
tlhhup
c3353c2b5e
Zookeeper management interface in monitoring center displays abnormal #2547 (#2595)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-05-06 13:56:05 +08:00
dailidong
eec92ed1fc
simplify and optimize config (#2469)
* [Refactor worker] simplify and optimize config (#2386)

* simplify config

* simplify config

* simplify and optimize config

* Update HadoopUtils.java

optimize HadoopUtils

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HttpUtils.java

* Update pom.xml

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java

* Update HadoopUtilsTest.java
2020-04-21 02:25:45 +00:00
bao liang
4946e88872
move dependent and conditions to master server (#2423)
* move condition and dependent to master

* move conditions to master

* move conditions and dependent task to master

* move conditions to master

* update test

* add log

* add test for dependent task

* add test for dependent task

* update

* update

* refactor complexity code

* refactor complexity code

* add conditions task test

* add conditions task test

* update

* update host to host:port

* update logback.xml

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-04-20 11:31:15 +08:00
qiaozhanwei
d6798c6db7
1,task status statistics and process status statistics bug fix (#2357) 2,worker group bug fix (#2430)
* dispatch task fail will set task status failed

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

* 1,task status statistics and process status statistics bug fix (#2357)
2,worker group bug fix

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-04-16 18:16:19 +08:00
xingchun-chen
347f05f08c
Update worker_group_id to worker_group (#2433)
* add LoginTest license

* Delete useless packages

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* e2e add project and workflow case

* Update worker_group_id to worker_group in init.sql

* Update worker_group_id to worker_group in init.sql

* Update worker_group_id to worker_group

Co-authored-by: chenxingchun <438044805@qq.com>
2020-04-15 18:30:31 +08:00
lgcareer
c881c9517e
It's to remove static of method dataSource,If not remove static the transaction will not work. (#2422)
* It's to remove `static` of method dataSource,If not remove `static` the transaction will not work.

* update testQueryDetailsById because it didn't run success
2020-04-15 10:22:02 +08:00
lgcareer
733acdfd76
It is necessary to check whether the resource is valid (#2399)
* init full_name in dml of version 1.2.2

* redefine request parameter resourceIds

* redefine request parameter resourceIds

* Actually grant resource file if choose the directory

* To cancel authorized resource need check whether it is used by the process definition which is online

* If release the process definition online,It is necessary to check whether resource is valid

* update ResourceServiceTest and ResourceMapperTest

* add batchUpdateResourceTest

* add getHdfsFileNameTest and getHdfsResourceFileNameTest

* update ResourceServiceTest and ResourceMapperTest

* extract getResourceProcessMap to ResourceProcessDefinitonUtils

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-04-11 17:27:10 +08:00
qiaozhanwei
b1a773f341 update UT in pom.xml 2020-04-10 09:27:18 +08:00
qiaozhanwei
9de9f18c08 update UT in pom.xml 2020-04-10 08:52:39 +08:00
qiaozhanwei
f65ab56d2d UT modify 2020-04-09 19:07:17 +08:00
qiaozhanwei
43b990656a modify UT 2020-04-09 16:40:01 +08:00
qiaozhanwei
f3c85835b1 mapper add Transactional and rollback 2020-04-09 13:40:47 +08:00
qiaozhanwei
0afe9a3fc5 modify datasource.properties postgresql password 2020-04-09 13:27:56 +08:00
qiaozhanwei
18548ad1c1 modify datasource.properties postgresql password 2020-04-09 13:27:03 +08:00
qiaozhanwei
692b7acdde datasource.properties modify 2020-04-08 18:13:27 +08:00
qiaozhanwei
3a066f2272 javadoc compile error modify 2020-04-07 17:17:39 +08:00
qiaozhanwei
b5546ddb68 refactor-worker merge to dev bug fix 2020-04-07 14:28:05 +08:00