Commit Graph

165 Commits

Author SHA1 Message Date
xingchun-chen
57eee5695a Merge pull request #3331 from lenboo/dev-complement
[FIX-1336] [master] serial complement data can have multiple process instances
2020-07-29 16:08:36 +08:00
bao liang
1b98db9240 fix bug#1336 serial complement data can have multiple process instances (#3317)
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-28 13:59:22 +08:00
lgcareer
6b964be60a
[Feature][server]support one worker can belongs different worker groups when execute install script (#3307) 2020-07-28 10:55:35 +08:00
lgcareer
f854c5f794
[Feature-2815]cherry pick commit from dev for one worker can belong different worker groups (#3277)
* [Feature-2815]cherry pick commit from dev for one worker can belong to different worker groups

* fix 2975:download resource need find the tenant code of resource owner

* change NetUtils to OSUtils because NetUtils didn't exist in the 1.3.2-release branch

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-22 19:39:03 +08:00
lenboo
572b142a53 [maven-release-plugin] prepare for next development iteration 2020-07-04 16:10:37 +08:00
lenboo
a4272efa55 [maven-release-plugin] prepare release 1.3.1 2020-07-04 16:04:36 +08:00
lenboo
fd3485ad0a revert version to 1.3.1 2020-07-04 15:54:37 +08:00
bao liang
4817257c04
remove some notes (#3127)
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-04 15:42:53 +08:00
lenboo
3d4d816349 [maven-release-plugin] prepare for next development iteration 2020-07-03 15:56:45 +08:00
lenboo
2ff94a4362 [maven-release-plugin] prepare release 1.3.1 2020-07-03 15:55:56 +08:00
lgcareer
02e22e8a74
fix 3112:download udf resource need find the tenant code of resource owner (#3114)
* fix 3112:download udf resource need find the tenant code of resource owner

* fix 2975:download resource need find the tenant code of resource owner

* remove useless code
2020-07-02 14:42:22 +08:00
lgcareer
07a6aec1fc
fix 2975:download resource need find the tenant code of resource owner (#3109) 2020-07-01 19:04:03 +08:00
dailidong
7445fa946b fix singe resource manager will NPE (#3013)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java
2020-06-30 14:11:07 +08:00
gaojun2048
58897aa2c5 merge 1.3.0-release branch code 2020-06-29 18:42:26 +08:00
gaojun2048
3cf9afc195 merge 1.3.0-release branch code 2020-06-29 18:42:18 +08:00
Yelli
0c91564b4c fix bug: sqoop mysql import params error (#2896)
* fix bug: sqoop mysql import params error

* fix bug: sqoop mysql import params error
2020-06-29 17:39:50 +08:00
dailidong
8e8ffa93c9 fix getWorkers bug on RunConfig (#2747)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug
2020-06-29 17:39:25 +08:00
743294668
3b0e8c9440 Fix notes (#2615)
Co-authored-by: taoyuxin <taoyuxin@inspur.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
2020-06-29 17:39:12 +08:00
songgg
b06c118af6 [optimization]:prevent repeated database updates (#2396)
* sqlTask failed to run

* prevent repeated database updates

* prevent repeated database updates

* prevent repeated database updates

Co-authored-by: songqh <songquanhe@foxmail.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:38:37 +08:00
Tboy
87b994230c Refactor worker (#2248)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory

* refactor TaskCallbackService

* add ZookeeperNodeManagerTest
2020-06-29 17:37:55 +08:00
Tboy
c61c59ae54 Refactor worker (#2186)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test
2020-06-29 17:37:49 +08:00
zixi0825
3f044e7771 #2282 fix workflow dependent bug (#2329)
* fix workflow dependent bug

* fix workflow dependent bug 2

* fix workflow dependent bug 2

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:41 +08:00
Rubik-W
9b6ab81e60 fix #2334 (#2335)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-06-29 17:36:38 +08:00
t1mon
1d9bc0cd74 Fix SqlTask kerberos load scope unreasonable. #2178 (#2321)
* Optimize PropertyUtils instantiation.

* Fix info error.

* Fix SqlTask kerberos load scope unreasonable. #2178
2020-06-29 17:36:37 +08:00
lgcareer
82d6dc9e52 new feature for #404 add resource tree function (#2323)
* add create resource directory

* add create resource directory

* update the resource test

* add upgrade sql in version 1.2.2

* Adding request parameter id to update queryResourceListPaging

* set isDirectory value is false default

* add full name to update updateResource

* remove request parameter isDirectory to update createResource method

* update queryResourceListPaging with change get to post

* update updateResource method with remove fullName

* File management list modification (#1976)

* add resource component

* add resource tree visitor

* return json string

* update queryResourceList

* upload file need fullName

* add method rootNode

* Shell task resources and authorization resources (#1989)

* File management list modification

* Shell task resources and authorization resources

* download resource when execute task

* download resource when execute task

* update authorization type

* download resource when execute task

* Spark task resource changes (#1990)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* download resource when execute task

* update udf function service

* add resource type in ResourceComponent

* UDF resource tree and change DAG style (#2019)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* add deleteIds method in ResourceMapper and ResourceMapperTest

* Add comments on class and method

* add queryResourceByName method in controller

* update verify-name with change name to full name

* update queryResource with add parameter pid

* update queryResource with add parameter pid

* add resource ids in process definition and delete resource need judge whether it is used by any process definition

* Breadcrumb development (#2033)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix (#2040)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Fix github action rerun failed (#2067)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Change crumb position

* Change crumb position (#2068)

* build resource process definition map

* UDF changed to multiple choice

* UDF changed to multiple choice (#2077)

* Change crumb position

* UDF changed to multiple choice

* build resource process definition map (#2076)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* update resource name also need update all the children full name (#2096)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Limit customization file content to no more than 3000 lines(#2128) (#2140)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList (#2192)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* Modify the main jar package

* Modify the main jar package (#2200)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Modify the main jar package

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* choose main jar with resource tree (#2220)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* getResouDelete checksum and modify parameter namerceId

* revert .env

* remove parameter pid

* Delete request interface

* go back to the last page

* jar interface call

* Fix issue #2234 and #2228

* change resource name with full name

* Fix issue #2234 and #2228 (#2246)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* revert .env

* remove parameter pid

* Fix issue #2234 and #2228

* change resource name with full name

* Fix list query value error

* remove unauth-file with authorize-resource-tree

* Repair data cannot be echoed

* Repair data cannot be echoed

* execute mr and spark task need query resource name before

* Authorized resource interface replacement

* Authorized resource interface replacement

* Filter UDF resources

* Change parameters

* need query all authorized directory children when create task

* Change normalize.scss import method and animation.scss license modification

* Delete file list update processing

* It's fixed that resource not deleted in hdfs when delete it.

* add tooltips

* add tooltips (#2310)

* Echo workflow name and modify udf management name

* [new feature]add resource tree function

* revert front code in order to be same as dev branch

* revert front code in order to be same as dev branch

* revert common.properties and application.properties

* add super method

* update flink parameter test

* update flink parameter and unit test

* update resource service test

* If resource list is empty,need init it

* update flink parameter test

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-29 17:36:36 +08:00
gabry.wu
f9c91b8cbe Adapting partial code(file name start with S #2) to the sonar cloud rule (#2270) 2020-06-29 17:36:36 +08:00
孙朝和
588c003860 resolve shell task build command nullpointexception (#2267)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-06-29 17:36:34 +08:00
gabry.wu
30b98b89a2 Adapting partial code(file name start with T) to the sonar cloud rule (#2271) 2020-06-29 17:36:33 +08:00
gabry.wu
a1f7614476 Adapting partial code(file name start with W) to the sonar cloud rule (#2287) 2020-06-29 17:36:33 +08:00
songquanhe-gitstudy
fc18d4a216 fix: No operations allowed after statement closed when running sql task (#2265)
* sqlTask failed to run

* get correct attribute value and logger content

Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:30 +08:00
Rubik-W
1acbc0dbfe fix:Depend check bug (#2229)
* fix:#1787

* fix:#1787
2020-06-29 17:36:29 +08:00
gabry.wu
e1771de45a Adapting partial code(file name start with R) to the sonar cloud rule (#2243) 2020-06-29 17:36:29 +08:00
Simon
5c30205574 support custom datax configuration (#2224)
* fix #1441

* support custom datax config

* support datax custom config

* support datax custom config

* support datax custom config

Co-authored-by: 张世鸣 <zhangshiming@cvte.com>
2020-06-29 17:36:29 +08:00
ruson
2bb55e4993 add repace time when run history job and batch complement job ,not current time (#2196)
* add release notes in 1.2.1 (#1654)

* fix Monitor bug (#1656)

* fix Monitor bug

* fix zk monitor bug

* fix api url (#1673)

* get root path from zookeeper config

* Fixed DAG zoom in and zoom out nodes separated from arrows #1679 (#1685)

* fix api url

* Fixed DAG zoom in and zoom out nodes separated from arrows #1679

* check license and update (#1725)

* Fixed tasks_queue and tasks_kill did not exist in zookeeper #1696 (#1734)

Co-authored-by: elonlo <gamelifedong@gmail.com>

* upgrade jackson from 2.9.8 to 2.10.1 (#1767)

* add out put log when master/worker server start (#1769)

* merge hadoop.properties into common.properties

* merge hadoop,zookeeper.properties into common.properties
remove combined.properties/master.properties/worker.properties

* change db user/pwd to test/test

* rename .dolphinscheduler_en.sh to dolphinscheduler_env.sh
remove some unused in install.sh

* add out put log when master/worker server start...

* add start log when servers start

* add check download resource permission in order to fix issues 1770 (#1788)

* merge dev branch front-end code (#1786)

* fix #1775 bug,delete process definition when process instance is running (#1790)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* fix udfs assignment and task instance page jump #1789 (#1791)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* update method checkDownloadPermission in order to fix issues 1770 (#1794)

* add check download resource permission in order to fix issues 1770

* update method checkDownloadPermission in order to fix issues 1770

* update method listUnauthorizedResource in order to fix issues 1770

* update method listUnauthorizedResource in order to fix issues 1770 (#1797)

* if login user is admin,it will has permission to query all udf function (#1799)

* Fixed space and icon display issues before and after the input box (#1798)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* donot submit task to queue if sub process (#1793)

* dockerfile modify (#1800)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* Fix data echo, style and popup cannot be closed (#1801)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value(#1810) (#1815)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* [Fix issue #1770]check udf and data source in order to fix issue 1770 (#1817)

* if login user is admin,it will has permission to query all udf function

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* revert common.properties

* update the test method name

* remove useless code and import in unit test

* refactor has permission and check permission

* #1813 remove "_001" after the master/server register path in zookeepe (#1824)

* donot submit task to queue if sub process

* [feature] #1813 remove "_001" after the master/server register path in zookeeper (#1820)

* change master/worker register path.

* remove "_" from register path.

* remove install.sh server.servlet.context-path replace (#1823)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface (#1827)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface

* rpm package add UI (#1846)

* [fix #1828] when the executor of process instance is not the owner of udf resouce, the path of the read resource file is incorrect (#1847)

* fix issue 1828:get the udf resource path error when create udf function

* update grantResources

* first verify whether udf resource is bound by udf function

* update grantResources

* update testListAuthorizedUdfFunc

* update getUserInfo in order to run success

* add LoggerServer RPC PORT modified. #1848 (#1849)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* add LoggerServer RPC PORT modified. #1848

* LoggerService Logger RPC PORT get Error modify . #1848 (#1857)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* add LoggerServer RPC PORT modified. #1848

* LoggerService Logger RPC PORT get Error modify . #1848

* Fix UDF function list delete data without refresh and page data display widened #1851 (#1852)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface

* Fix UDF function list delete data without refresh and page data display widened

* [Fix #1828]check whether has permission to download udf file or delete udf function (#1858)

* fix issue 1828:get the udf resource path error when create udf function

* update grantResources

* first verify whether udf resource is bound by udf function

* update grantResources

* update testListAuthorizedUdfFunc

* update getUserInfo in order to run success

* check whether has permission to download udf file or delete udf file

* update listAuthorizedResourceById in ResourceMapper.xml

* add log, don't swallow exception info (#1877)

* Added hints in Resource Center (#1891)

* update jackson version from 2.9.8 to 2.10.1 (#1892)

* update release notes and modify the plugin of rat (#1899)

* modify how to build (#1902)

* update release notes and modify the plugin of rat

* modify how to build

* [maven-release-plugin] prepare release 1.2.1

* [maven-release-plugin] prepare for next development iteration

* get authorized udf resources need exclude all created by self (#1905)

* get authorized udf resources need exclude all created by self

* reset pom version to 1.2.1-SNAPSHOT

* Update ReleaseNotes.md

add detail info

* Update ReleaseNotes.md

* update notice year form 2019 to 2019-2020 (#1907)

* [maven-release-plugin] prepare release 1.2.1

* [maven-release-plugin] prepare for next development iteration

* 当手动重跑历史活着补数的时候对shell和sql脚本中的[YYYYmmddd...]变量赋值与传递的日期,而不是当前日期

Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: bao liang <29528966+lenboo@users.noreply.github.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: elonlo <gamelifedong@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: zhangchunyang <18910529250@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: 谷雨 <lixun@guyudeMacBook-Pro.local>
2020-06-29 17:36:28 +08:00
gabry.wu
015a8ca269 Adapting partial code(file name start with M #2) to the sonar cloud rule (#2179)
* Adapting partial code(file name start with M #2) to the sonar cloud rule

* remove unused import
2020-06-29 17:36:28 +08:00
t1mon
55cc1c7c5b Replace Charset.forName("UTF-8") with StandardCharsets.UTF_8. (#2193) 2020-06-29 17:36:28 +08:00
gabry.wu
2f4fda7cdb Adapting partial code(file name start with M #1) to the sonar cloud rule (#2173)
* Adapting partial code(file name start with M #1) to the sonar cloud rule

* remove unused import
2020-06-29 17:36:28 +08:00
gabry.wu
7f7f50a828 Adapting partial code(file name start with L) to the sonar cloud rule (#2171) 2020-06-29 17:36:28 +08:00
Rubik-W
e7c7cea4af fix: NullPointException in the case of task queuing (#2175) 2020-06-29 17:36:28 +08:00
Rubik-W
8b61afc9be The master and worker server exit exception #2163 (#2176)
* fix: #2163

* fix: format
2020-06-29 17:36:27 +08:00
gabry.wu
610c95a470 use static method in static class JSON (#2156) 2020-06-29 17:36:27 +08:00
gabry.wu
156533ad5e Adapting partial code(file name start with H) to the sonar cloud rule (#2145) 2020-06-29 17:36:22 +08:00
gabry.wu
493499f118 Adapting partial code(file name start with F) to the sonar cloud rule (#2045)
* Adapting partial code(file name start with F) to the sonar cloud rule

* add more unit test

* add License

* add includes configuration to maven-surefire-plugin

* fix getResourceFilesList incorrect logic
2020-06-29 17:36:22 +08:00
gabry.wu
fdc9e0090b remove incorrect or unnecessary URL tag (#2099) 2020-06-29 17:36:13 +08:00
Tboy
7ed28af4fe
Upgrade curator version (#2083)
* upgrade curator version
issue: https://github.com/apache/incubator-dolphinscheduler/issues/2020
curator issue: https://github.com/apache/curator/pull/297

* add DefaultEnsembleProvider override

* add some unit test

* add License
2020-03-05 19:00:06 +08:00
dailidong
4230d5a737
add embedded database h2 and update embedded zookeeper (#2082)
add embedded database h2 and update embedded zookeeper
2020-03-05 17:32:40 +08:00
dailidong
6ec85540bf
fix typo (#2093) 2020-03-05 16:36:20 +08:00
qiaozhanwei
d5c795bd45
fix ugly logger usage (#2063)
* fix ugly logger usage

* remove space
2020-03-05 11:10:48 +08:00
qiaozhanwei
3e5a939913
delete unused import (#2066) 2020-03-05 11:07:44 +08:00
qiaozhanwei
b8014d91cf
add workflow id, name for email content (#2071)
* add tenantService UT

* fix spell

* #464 email content

* modify mysql1.2.2 ddl
2020-03-05 11:04:00 +08:00