Commit Graph

199 Commits

Author SHA1 Message Date
lgcareer
b01898a8eb [release]revert pom version to 1.3.3-release 2020-10-29 17:29:38 +08:00
lgcareer
1076a2349c [maven-release-plugin] prepare for next development iteration 2020-10-28 19:19:46 +08:00
lgcareer
ed016499aa [maven-release-plugin] prepare release 1.3.3 2020-10-28 19:19:35 +08:00
lgcareer
0d21fa9e3b revert pom version to 1.3.3-release 2020-10-28 19:11:35 +08:00
lgcareer
92a793288f [maven-release-plugin] prepare for next development iteration 2020-10-17 18:14:23 +08:00
lgcareer
c144d81579 [maven-release-plugin] prepare release 1.3.3 2020-10-17 18:14:13 +08:00
lgcareer
ae85116fa5
change version from 1.3.2-SNAPSHOT to 1.3.3-SNAPSHOT (#3934) 2020-10-17 17:50:02 +08:00
lgcareer
2e4f9f19ce
[Fix-#3713][common]Fix that catfile method Stream not closed (#3810)
* [Bug-3713][HadoopUtils] catfile method Stream not closed  (#3715)

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* dag  add close button

* reset last version

* reset last version

* dag add close buttion

dag add close buttion

* update  CLICK_SAVE_WORKFLOW_BUTTON  xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* Update CreateWorkflowLocator.java

modify submit workflow button

* Update CreateWorkflowLocator.java

* Update CreateWorkflowLocator.java

modify CLICK_ADD_BUTTON

* Update CreateWorkflowLocator.java

delete print

* Update CreateWorkflowLocator.java

1

* Update CreateWorkflowLocator.java

1

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dag.vue

* Update dag.vue

* Update dag.vue

* Update CreateWorkflowLocator.java

* Revert "Setting '-XX:+DisableExplicitGC ' causes netty memory leaks"

This reverts commit 3a2cba7a

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dolphinscheduler-daemon.sh

* catfile method Stream not closed

* catfile method Stream not closed

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>

* [Fix-#3713][common]Fix that catfile method Stream not closed

Co-authored-by: BoYiZhang <39816903+BoYiZhang@users.noreply.github.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-09-24 15:28:02 +08:00
qiaozhanwei
29f1123d1d
[Fix-3616][Server] when worker akc/response master exception , async retry (#3748) 2020-09-17 10:06:59 +08:00
qiaozhanwei
917a4b20b5
release 1.3.2 version rollback (#3527)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

* modify tag 1.3.0 to HEAD

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-17 14:21:13 +08:00
qiaozhanwei
32edd19ac5 [maven-release-plugin] prepare for next development iteration 2020-08-17 14:03:19 +08:00
qiaozhanwei
4d2f0b0578 [maven-release-plugin] prepare release 1.3.2 2020-08-17 14:03:08 +08:00
qiaozhanwei
11d3c07e46
modify tag 1.3.0 to HEAD (#3525)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

* modify tag 1.3.0 to HEAD

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-17 11:46:35 +08:00
qiaozhanwei
f3f895751b [maven-release-plugin] prepare for next development iteration 2020-08-14 19:20:27 +08:00
qiaozhanwei
f19b125374 [maven-release-plugin] prepare release 1.3.2 2020-08-14 19:20:17 +08:00
qiaozhanwei
4ba6b35b0f
test release 1.3.2 version rollback (#3504)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 18:55:03 +08:00
qiaozhanwei
f50353f8a6 [maven-release-plugin] prepare for next development iteration 2020-08-14 18:26:09 +08:00
qiaozhanwei
afd227a179 [maven-release-plugin] prepare release 1.3.2 2020-08-14 18:25:59 +08:00
qiaozhanwei
660eb5a1e6
test release 1.3.2 version rollback (#3503)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

* test

* test release 1.3.2 version rollback

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 18:19:19 +08:00
qiaozhanwei
b5f0f23d85 [maven-release-plugin] prepare for next development iteration 2020-08-14 17:37:33 +08:00
qiaozhanwei
88b3ef5d30 [maven-release-plugin] prepare release 1.3.2 2020-08-14 17:37:23 +08:00
qiaozhanwei
45424ec265
test release 1.3.2 version rollback (#3499)
* [Feature] JVM parameter optimization , related issue #3370

* [Feature] JVM parameter optimization , related issue #3370

* test release 1.3.2 version rollback

* test release 1.3.2 version rollback

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-08-14 17:31:09 +08:00
qiaozhanwei
c6f0c8901c [maven-release-plugin] prepare for next development iteration 2020-08-14 16:04:49 +08:00
qiaozhanwei
8a6732ae4c [maven-release-plugin] prepare release 1.3.2 2020-08-14 16:04:38 +08:00
Yelli
e4097a488c
cancel spark task version check (#3406)
Co-authored-by: Eights-Li <yelli.hl@gmail.com>
2020-08-05 10:20:34 +08:00
lgcareer
a3793be7da
[Fix-3390][server]Running hive sql task need find the hdfs path correctly (#3396)
* [Fix-3390][api]Running hive sql task need find the hdfs path correctly

* [Fix-3390][api]Running hive sql task need find the hdfs path correctly

* update soft version
2020-08-04 15:04:56 +08:00
lgcareer
3744167d52
[Fix-3364]Running the spark task fail because the can't find the jar (#3378) 2020-08-03 10:31:27 +08:00
lgcareer
f854c5f794
[Feature-2815]cherry pick commit from dev for one worker can belong different worker groups (#3277)
* [Feature-2815]cherry pick commit from dev for one worker can belong to different worker groups

* fix 2975:download resource need find the tenant code of resource owner

* change NetUtils to OSUtils because NetUtils didn't exist in the 1.3.2-release branch

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-22 19:39:03 +08:00
lenboo
572b142a53 [maven-release-plugin] prepare for next development iteration 2020-07-04 16:10:37 +08:00
lenboo
a4272efa55 [maven-release-plugin] prepare release 1.3.1 2020-07-04 16:04:36 +08:00
lenboo
fd3485ad0a revert version to 1.3.1 2020-07-04 15:54:37 +08:00
dailidong
4a4e940379
fix Permission denied if not found dolphinscheder-env.sh on 1.3.1 (#3128)
* fix Permission denied if not found dolphinscheder-env.sh

* update some comments on method
2020-07-04 15:49:46 +08:00
lenboo
3d4d816349 [maven-release-plugin] prepare for next development iteration 2020-07-03 15:56:45 +08:00
lenboo
2ff94a4362 [maven-release-plugin] prepare release 1.3.1 2020-07-03 15:55:56 +08:00
dailidong
7445fa946b fix singe resource manager will NPE (#3013)
* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java
2020-06-30 14:11:07 +08:00
gaojun2048
58897aa2c5 merge 1.3.0-release branch code 2020-06-29 18:42:26 +08:00
gaojun2048
3cf9afc195 merge 1.3.0-release branch code 2020-06-29 18:42:18 +08:00
bao liang
31b593d023 change max length of user name to 39 (#2856)
* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

Co-authored-by: baoliang <baoliang@analysys.com.cn>
2020-06-29 17:39:41 +08:00
lgcareer
a9cbca1e55 add source license full text into source license (#2840)
* add source license full text into source license

* exclude ScriptRunner.java on rat check
2020-06-29 17:39:38 +08:00
qiaozhanwei
5c9123f9ae blank in eamil and left font align #2648 (#2673)
* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:39:17 +08:00
dailidong
88bdb20a91 Update HttpUtils.java 2020-06-29 17:38:48 +08:00
qiaozhanwei
a085aa8fa3 modify UT 2020-06-29 17:38:23 +08:00
qiaozhanwei
cc3ab0f191 timing online modify (#2296)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

* master fault tolerant bug modify

* UT add pom.xml

* timing online  modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-06-29 17:37:58 +08:00
songgg
d8c7a44e82 some annotation optimizations (#2333)
Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:38 +08:00
lgcareer
82d6dc9e52 new feature for #404 add resource tree function (#2323)
* add create resource directory

* add create resource directory

* update the resource test

* add upgrade sql in version 1.2.2

* Adding request parameter id to update queryResourceListPaging

* set isDirectory value is false default

* add full name to update updateResource

* remove request parameter isDirectory to update createResource method

* update queryResourceListPaging with change get to post

* update updateResource method with remove fullName

* File management list modification (#1976)

* add resource component

* add resource tree visitor

* return json string

* update queryResourceList

* upload file need fullName

* add method rootNode

* Shell task resources and authorization resources (#1989)

* File management list modification

* Shell task resources and authorization resources

* download resource when execute task

* download resource when execute task

* update authorization type

* download resource when execute task

* Spark task resource changes (#1990)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* download resource when execute task

* update udf function service

* add resource type in ResourceComponent

* UDF resource tree and change DAG style (#2019)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* add deleteIds method in ResourceMapper and ResourceMapperTest

* Add comments on class and method

* add queryResourceByName method in controller

* update verify-name with change name to full name

* update queryResource with add parameter pid

* update queryResource with add parameter pid

* add resource ids in process definition and delete resource need judge whether it is used by any process definition

* Breadcrumb development (#2033)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix (#2040)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Fix github action rerun failed (#2067)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Change crumb position

* Change crumb position (#2068)

* build resource process definition map

* UDF changed to multiple choice

* UDF changed to multiple choice (#2077)

* Change crumb position

* UDF changed to multiple choice

* build resource process definition map (#2076)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* update resource name also need update all the children full name (#2096)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Limit customization file content to no more than 3000 lines(#2128) (#2140)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList (#2192)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* Modify the main jar package

* Modify the main jar package (#2200)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Modify the main jar package

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* choose main jar with resource tree (#2220)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* getResouDelete checksum and modify parameter namerceId

* revert .env

* remove parameter pid

* Delete request interface

* go back to the last page

* jar interface call

* Fix issue #2234 and #2228

* change resource name with full name

* Fix issue #2234 and #2228 (#2246)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* revert .env

* remove parameter pid

* Fix issue #2234 and #2228

* change resource name with full name

* Fix list query value error

* remove unauth-file with authorize-resource-tree

* Repair data cannot be echoed

* Repair data cannot be echoed

* execute mr and spark task need query resource name before

* Authorized resource interface replacement

* Authorized resource interface replacement

* Filter UDF resources

* Change parameters

* need query all authorized directory children when create task

* Change normalize.scss import method and animation.scss license modification

* Delete file list update processing

* It's fixed that resource not deleted in hdfs when delete it.

* add tooltips

* add tooltips (#2310)

* Echo workflow name and modify udf management name

* [new feature]add resource tree function

* revert front code in order to be same as dev branch

* revert front code in order to be same as dev branch

* revert common.properties and application.properties

* add super method

* update flink parameter test

* update flink parameter and unit test

* update resource service test

* If resource list is empty,need init it

* update flink parameter test

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-06-29 17:36:36 +08:00
gabry.wu
f9c91b8cbe Adapting partial code(file name start with S #2) to the sonar cloud rule (#2270) 2020-06-29 17:36:36 +08:00
gabry.wu
30b98b89a2 Adapting partial code(file name start with T) to the sonar cloud rule (#2271) 2020-06-29 17:36:33 +08:00
songquanhe-gitstudy
fc18d4a216 fix: No operations allowed after statement closed when running sql task (#2265)
* sqlTask failed to run

* get correct attribute value and logger content

Co-authored-by: songqh <songquanhe@foxmail.com>
2020-06-29 17:36:30 +08:00
gabry.wu
e1771de45a Adapting partial code(file name start with R) to the sonar cloud rule (#2243) 2020-06-29 17:36:29 +08:00
Simon
5c30205574 support custom datax configuration (#2224)
* fix #1441

* support custom datax config

* support datax custom config

* support datax custom config

* support datax custom config

Co-authored-by: 张世鸣 <zhangshiming@cvte.com>
2020-06-29 17:36:29 +08:00