Commit Graph

150 Commits

Author SHA1 Message Date
qiaozhanwei
41d8abdafb test 2020-04-07 14:33:38 +08:00
break60
bd6adf284d
Fix the problem of data echo in script edit box (#2366)
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-04-04 12:17:02 +08:00
break60
ef8d64aacc fix 2020-04-02 17:32:00 +08:00
break60
5bff786e7c Fix the problem of data echo in script edit box 2020-04-02 17:00:16 +08:00
break60
8da2a50d16 Corrections to historical jar packages and resource data 2020-04-01 21:05:54 +08:00
break60
8561d65b2b Corrections to historical jar packages and resource data 2020-04-01 17:38:57 +08:00
qiaozhanwei
7cd1f91587 test 2020-03-31 16:55:00 +08:00
qiaozhanwei
152c814126 test 2020-03-31 16:51:19 +08:00
break60
2b17435337
Supplemental licenses and repair resources cannot be renamed (#2340)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines

* dd branch flow node verification

* datax

* datax add custom

* Change normalize.scss import method and animation.scss license modification

* Resource tree code merge

* Modify ans-ui version and timing management style

* Supplemental licenses and repair resources cannot be renamed
2020-03-31 11:11:50 +08:00
break60
b7bf172d62 Supplemental licenses and repair resources cannot be renamed 2020-03-31 10:55:24 +08:00
break60
2114c8d95f
Modify ans-ui version and timing management style (#2339)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines

* dd branch flow node verification

* datax

* datax add custom

* Change normalize.scss import method and animation.scss license modification

* Resource tree code merge

* Modify ans-ui version and timing management style
2020-03-31 10:51:04 +08:00
break60
92d944edde Modify ans-ui version and timing management style 2020-03-31 09:08:30 +08:00
liwenhe1993
526e5c91f1
Connection mode of adding Sid to Oracle (#2254)
* Connection mode of adding Sid to Oracle

* Remove code

* Add asf

* Add unit test

* Add unit test

* Add unit test

* solve the conflict
2020-03-28 23:28:35 +08:00
lgcareer
38c965ddc9
new feature for #404 add resource tree function (#2323)
* add create resource directory

* add create resource directory

* update the resource test

* add upgrade sql in version 1.2.2

* Adding request parameter id to update queryResourceListPaging

* set isDirectory value is false default

* add full name to update updateResource

* remove request parameter isDirectory to update createResource method

* update queryResourceListPaging with change get to post

* update updateResource method with remove fullName

* File management list modification (#1976)

* add resource component

* add resource tree visitor

* return json string

* update queryResourceList

* upload file need fullName

* add method rootNode

* Shell task resources and authorization resources (#1989)

* File management list modification

* Shell task resources and authorization resources

* download resource when execute task

* download resource when execute task

* update authorization type

* download resource when execute task

* Spark task resource changes (#1990)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* download resource when execute task

* update udf function service

* add resource type in ResourceComponent

* UDF resource tree and change DAG style (#2019)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* add deleteIds method in ResourceMapper and ResourceMapperTest

* Add comments on class and method

* add queryResourceByName method in controller

* update verify-name with change name to full name

* update queryResource with add parameter pid

* update queryResource with add parameter pid

* add resource ids in process definition and delete resource need judge whether it is used by any process definition

* Breadcrumb development (#2033)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix (#2040)

* File management list modification

* Shell task resources and authorization resources

* Spark task resource changes

* UDF resource tree and change DAG style

* Breadcrumb development

* Breadcrumb development

* Resource tree bug fix

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Fix github action rerun failed (#2067)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* Change crumb position

* Change crumb position (#2068)

* build resource process definition map

* UDF changed to multiple choice

* UDF changed to multiple choice (#2077)

* Change crumb position

* UDF changed to multiple choice

* build resource process definition map (#2076)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* update resource name also need update all the children full name (#2096)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Limit customization file content to no more than 3000 lines(#2128) (#2140)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList (#2192)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* Modify the main jar package

* Modify the main jar package (#2200)

* Change crumb position

* UDF changed to multiple choice

* Limit customization file content to no more than 3000 lines

* Limit customization file content to no more than 3000 lines(#2128)

* Modify the main jar package

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* choose main jar with resource tree (#2220)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* getResouDelete checksum and modify parameter namerceId

* revert .env

* remove parameter pid

* Delete request interface

* go back to the last page

* jar interface call

* Fix issue #2234 and #2228

* change resource name with full name

* Fix issue #2234 and #2228 (#2246)

* update resource service test

* Fix github action rerun failed

* add status of PARENT_RESOURCE_NOT_EXIST

* build resource process definition map

* update resource name also need update all the children full name

* need add queryResource

* add queryResourceJarList

* add queryResourceJarList

* add queryResourceJarList

* add resource filter in order to get filtered resource

* add comments of resource filter

* update list children by resource

* Return null if query resource list is empty

* update queryResource method change parameter pid to id

* revert .env

* remove parameter pid

* Fix issue #2234 and #2228

* change resource name with full name

* Fix list query value error

* remove unauth-file with authorize-resource-tree

* Repair data cannot be echoed

* Repair data cannot be echoed

* execute mr and spark task need query resource name before

* Authorized resource interface replacement

* Authorized resource interface replacement

* Filter UDF resources

* Change parameters

* need query all authorized directory children when create task

* Change normalize.scss import method and animation.scss license modification

* Delete file list update processing

* It's fixed that resource not deleted in hdfs when delete it.

* add tooltips

* add tooltips (#2310)

* Echo workflow name and modify udf management name

* [new feature]add resource tree function

* revert front code in order to be same as dev branch

* revert front code in order to be same as dev branch

* revert common.properties and application.properties

* add super method

* update flink parameter test

* update flink parameter and unit test

* update resource service test

* If resource list is empty,need init it

* update flink parameter test

Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-03-28 15:43:07 +08:00
break60
c52c92e015
Resource tree code merge (#2322)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines

* dd branch flow node verification

* datax

* datax add custom

* Change normalize.scss import method and animation.scss license modification

* Resource tree code merge
2020-03-28 09:26:36 +08:00
break60
0d70e065ba
Change normalize.scss import method and animation.scss license modification (#2295)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines

* dd branch flow node verification

* datax

* datax add custom

* Change normalize.scss import method and animation.scss license modification
2020-03-24 17:40:09 +08:00
break60
3ced3d594b
datax add custom and add branch flow node verification (#2218)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines

* dd branch flow node verification

* datax

* datax add custom
2020-03-23 10:10:31 +08:00
khadgarmage
cddf803ba0
license and checkoutv2 fix (#2261) 2020-03-21 20:23:13 +08:00
break60
99c28c7485
modify parameters (#2203)
* Delete worker group management page

* Modify workgroupid parameter name

* Worker group parameter modification

* modify parameters
2020-03-18 00:18:19 +08:00
qiaozhanwei
055bb28de4
worker fault tolerance modify (#2212)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-17 18:45:39 +08:00
break60
aa650b4426
Worker group parameter modification (#2188)
* Delete worker group management page

* Modify workgroupid parameter name

* Worker group parameter modification
2020-03-16 11:33:04 +08:00
liwenhe1993
dd2c50a10d
Toolbar of the DAG has not prompt (#2161)
* Toolbar of the DAG has not prompt

* Remove duplication code
2020-03-13 23:38:38 +08:00
gaojun2048
a0088c1137
fix rpmbuild and DS Ambari plugin bug (#2144)
* Update quick-start.md

* 更新demo的地址

* add rpm plugin

* delete the git ignore files

* 测试git ignore

* 测试ignore

* optimize the .gitignore file

* add rpm package, delete unuse copy jar action in assembly

* delete docs dir

* add rpm plugin support

* add rpm plugin support

* add rpm plugin support

* dolphinscheduler-common have not bin dir

* dolphinscheduler-common have not bin dir

* delete unuse config about assembly

* add defineSatement

* add rpm plugin support

* add rpm plugin support

* add rpm plugin support

* add rpm support

* update install dirname , make front and backend together

* update rpm name

* update rpm name

* update rpm config

* add jars excludes

* add jars excludes

* add rpm plugion

* add rpm plugion

* add rpm plugion

* add auto create /opt/soft

* The install dir can not be fixed to apache-dolphinscheduler-incubating-1.2.1 , its version is 1.2.2

* install from rpm will create a link

* the ui install thrown rpm only support jetty not nginx

* the ui install thrown rpm only support jetty not nginx

* sql目录里加上soft_version文件

* env dir need in conf dir

* remove no use module

* add module

* updaate

* rpmbuild add dolphinscheduler-service module
2020-03-13 13:04:33 +08:00
zhukai
68e4e5c4e0
add input cache for sqoop and conditions tasks (#2118)
Co-authored-by: zhukai <zhukai@ygsoft.com>
2020-03-12 18:08:49 +08:00
Rubik-W
9fda4da70e
fix: The toolbar buttons of Dag disable bug (#2124) 2020-03-12 18:00:33 +08:00
break60
712a1bc126
Limit customization file content to no more than 3000 lines(#2128) (#2141)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines
2020-03-12 17:59:16 +08:00
break60
290772dfaa
Modify workgroupid parameter name (#2105)
* Delete worker group management page

* Modify workgroupid parameter name
2020-03-08 15:59:13 +08:00
zhukai
3186545da1
1. add input cache for datax task (#2034)
2. add cache for codemirror component, include python, shell and sql task.
2020-03-04 09:28:23 +08:00
break60
3277b90647
Change DOM label (#2064)
* Change DOM label

* Change name to lowercase
2020-03-03 14:38:21 +08:00
bao liang
0df7c6719d
[New Feature] add conditions task #205 (#2003)
* add funtion conditions task

* update conditions tasks

* update conditions for ui

* update conditions

* update

* revert

* update
2020-02-28 17:08:34 +08:00
qiaozhanwei
2bb5ebaf3a
add TaskInstanceCacheManager receive Worker report result,modify master polling db transfrom to cache (#2021)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access
2020-02-26 18:46:05 +08:00
zhukai
5f3d88e4c2
Fix bug #1986. (#1987)
When in the projects define page, click the toolbar's button(task type) will cause the task details window switch to new one.

For now, the task details window will not switch.
2020-02-25 09:47:51 +08:00
孙朝和
3d6eee5d80
#203 add sqoop task (#1974)
* add sqoop task

* add sqoop task

* add sqoop task test

* add sqoop task test

* add sqoop task test

* modify add sqoop task test

* modify sqoop task test

* modify sqoop task test

* modify sqoop task test

* modify sqoop task test 2

* modify sqoop task test 3

* modify sqoop task test 3

* modify sqoop task test 3

* modify sqoop task test 4
2020-02-24 10:30:17 +08:00
liwenhe1993
34e0a663ed
Refactor dockerfile (#1978)
* Support DS to create user and group in windows environment

* Add unit test

* delete combined-server config in dolphinscheduler-daemon.sh file

* refactor dockerfile

* modify dockerfile

* modify dockerfile
2020-02-20 20:49:38 +08:00
Yelli
619fb30c50
#1310 add executor user to the process instance & task instance pages (#1973)
* add executor name in processDefinitionInstance & taskInstance

* modify processInstanceTest

* modify processInstanceTest

* #1310 add executor user to the process instance & task instance pages

* add processInstanceUT & taskInstanceUT

* add processInstanceUT & taskInstanceUT

* modify processInstanceTest & taskInstanceTest

* Remove duplicate code

* add userServiceUT
2020-02-20 13:25:41 +08:00
zhukai
afd5c75cd2
Fix the bug #1968. (#1969)
When create a new task or edit an exist task.The task details window will be open.
Then if delete the task node or click the DataSource menu. And the task details window still open.

The task details window should be close.
2020-02-17 19:09:10 +08:00
Yelli
d937a6ae94 Add modify user name for process definition (#1919)
* class overrides equals() and should therefore also override hashCode()

* #1862 add modify user in process difinition list

* #1862 add pg-1.2.2 ddl.sql

* modify ScriptRunnerTest

* add updateProessDifinition UT

* modify updateProcessDifinition UT

* modify updateProcessDifinition UT

* modify mysql 1.2.2 ddl.sql&dml.sql

* add scope test to mysql in pom

* modify pg-1.2.2 ddl.sql
2020-02-10 16:37:51 +08:00
zhukai
bcc734713b Newfeature for #1675. (#1908)
Continue to finish the rest works, add the cache feature for dependence,mr,python,sub_process,procedure and shell.
2020-02-10 14:15:20 +08:00
魔方不在手
f942e5ced3
[feature] data synchronization function (#1881)
* feature:data synchronization function

* feature:data synchronization function

* feature:data synchronization function

* fix:add license

* fix:resource close bug

* fix:remove class author

* fix:Improve test coverage

* fix:add UT

* fix:add UT

* fix:data sync node renamed to datax

* fix:add UT

* fix: UT bug

* fix:Optimize variable name

* fix:pom

* fix:Variable name error

* fix:optimize the code
2020-02-06 09:29:23 +08:00
break60
9549910fcc Fix UDF function list delete data without refresh and page data display widened #1851 (#1853)
* fix udfs assignment and task instance page jump #1789

* jest add coverage display

* Fix data echo, style, popup cannot be closed and license

* Fix data echo, style, popup cannot be closed and license

* fix

* Requests the previous page when there is no data on the list page

* Fix DAG task nodes keep requesting API and tooltip background color modification

* Modify list tooltip theme color

* Fix UDF function list delete data without refresh and page data display widened
2020-01-19 11:59:06 +08:00
break60
60bb2deb2a Modify list tooltip theme color (#1845)
* fix udfs assignment and task instance page jump #1789

* jest add coverage display

* Fix data echo, style, popup cannot be closed and license

* Fix data echo, style, popup cannot be closed and license

* fix

* Requests the previous page when there is no data on the list page

* Fix DAG task nodes keep requesting API and tooltip background color modification

* Modify list tooltip theme color
2020-01-17 15:02:47 +08:00
老佛爷
1c77708a89 Supplementary data by schedule (#1830)
* Supplementary data by schedule

* fix sonar check bug

* fix code duplicated blocks

* ut

* loop by day

* MasterExecThread test

* test add licene

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-01-16 15:33:00 +08:00
break60
5739afb95a Fix DAG task nodes keep requesting API and tooltip background color modification (#1837)
* fix udfs assignment and task instance page jump #1789

* jest add coverage display

* Fix data echo, style, popup cannot be closed and license

* Fix data echo, style, popup cannot be closed and license

* fix

* Requests the previous page when there is no data on the list page

* Fix DAG task nodes keep requesting API and tooltip background color modification
2020-01-16 11:27:43 +08:00
zhukai
be4fa728e9 fix the bug #1816 (#1834) 2020-01-15 13:56:02 +08:00
break60
d02422d030 Requests the previous page when there is no data on the list page(#1768) (#1832)
* fix udfs assignment and task instance page jump #1789

* jest add coverage display

* Fix data echo, style, popup cannot be closed and license

* Fix data echo, style, popup cannot be closed and license

* fix

* Requests the previous page when there is no data on the list page
2020-01-15 13:55:25 +08:00
zhukai
512e7c1f6a fix bug, add the cache logic in getInstancedetail. (#1814) 2020-01-15 11:26:57 +08:00
zhukai
289ae6ac96 Implements the spark and flink type task, now they can be cache the previous input. (#1795) 2020-01-13 10:18:43 +08:00
break60
24f362de5a fix udfs assignment and task instance page jump #1789 (#1792)
* fix udfs assignment and task instance page jump #1789

* jest add coverage display

* Fix data echo, style, popup cannot be closed and license

* Fix data echo, style, popup cannot be closed and license

* fix
2020-01-13 10:12:09 +08:00
break60
b6d6bb6416 Fixed some input box length restrictions and udfs not getting values #109 (#1778)
* Fix api url

* Fixed DAG zoom in and zoom out nodes separated from arrows

* Fix front-end code specifications

* Fix front-end code specifications

* Fix front-end code specifications

* jest unit test demo

* jest unit test demo

* Fix delete operation, no query interface request is made on the list page

* Fix delete operation, no query interface request is made on the list page

* Fixed some input box length restrictions and udfs not getting values
2020-01-09 10:18:45 +08:00
break60
74a5506b9a Fix delete operation, no query interface request is made on the list page #1768 (#1777)
* Fix api url

* Fixed DAG zoom in and zoom out nodes separated from arrows

* Fix front-end code specifications

* Fix front-end code specifications

* Fix front-end code specifications

* jest unit test demo

* jest unit test demo

* Fix delete operation, no query interface request is made on the list page

* Fix delete operation, no query interface request is made on the list page
2020-01-08 19:35:27 +08:00