Commit Graph

2396 Commits

Author SHA1 Message Date
tswstarplanet
f4897b563d
fix blocker problem (#2244)
* fix blocker problem

* use error level to log exception
2020-03-19 23:56:26 +08:00
Tboy
0dab5cb042
Refactor worker (#2241)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory

* refactor TaskCallbackService
2020-03-19 13:31:09 +00:00
John Liu
0714971453
Optimizing code (#2240)
* updated

* optimized the code

* Delete ParameterUtils.java

* rollback source code

* rollback source code

* rollback source code

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-03-19 19:26:03 +08:00
Simon
00ff88ef1c
support custom datax configuration (#2224)
* fix #1441

* support custom datax config

* support datax custom config

* support datax custom config

* support datax custom config

Co-authored-by: 张世鸣 <zhangshiming@cvte.com>
2020-03-19 18:56:58 +08:00
break60
bfcdc17d5d datax 2020-03-19 14:00:18 +08:00
dailidong
ce2853ad4c
add master and worker properties and remove data quality config (#2211)
* update logback

* update log

* refactor worker registry (#2107)

* Refactor worker (#2115)

* refactor worker registry

* refactor master server

* Modify workgroupid parameter name (#2105)

* Delete worker group management page

* Modify workgroupid parameter name

* Refactor worker (#2121)

* refactor worker registry

* refactor master server

* refactor MasterSchedulerService

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* not exist in openjdk,just delete

* add master and worker properties

* add master and worker properties

* add master and worker properties

Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-19 00:07:36 +08:00
tswstarplanet
3c5227ac0f
Refactor ConnectionFactory to static inner holder Singleton (#2204)
* refactor ConnectionFactory to static inner holder Singleton

* remove redundant import

* make getSqlSessionFactory() method private

* fix sonar issue

* remove @Ignore of MailUtilsTest

* add MailUtilsTest to pom.xml to test

* fix MailUtilsTest path error in pom.xml

* modify test method name

* add assert and logger to unit test

* add log

* add log

* add AlertDaoTest

* move AlertDaoTest to new module

* modify test in pom.xml

* remove unnecessary log
2020-03-18 23:48:19 +08:00
ruson
880e995654
add repace time when run history job and batch complement job ,not current time (#2196)
* add release notes in 1.2.1 (#1654)

* fix Monitor bug (#1656)

* fix Monitor bug

* fix zk monitor bug

* fix api url (#1673)

* get root path from zookeeper config

* Fixed DAG zoom in and zoom out nodes separated from arrows #1679 (#1685)

* fix api url

* Fixed DAG zoom in and zoom out nodes separated from arrows #1679

* check license and update (#1725)

* Fixed tasks_queue and tasks_kill did not exist in zookeeper #1696 (#1734)

Co-authored-by: elonlo <gamelifedong@gmail.com>

* upgrade jackson from 2.9.8 to 2.10.1 (#1767)

* add out put log when master/worker server start (#1769)

* merge hadoop.properties into common.properties

* merge hadoop,zookeeper.properties into common.properties
remove combined.properties/master.properties/worker.properties

* change db user/pwd to test/test

* rename .dolphinscheduler_en.sh to dolphinscheduler_env.sh
remove some unused in install.sh

* add out put log when master/worker server start...

* add start log when servers start

* add check download resource permission in order to fix issues 1770 (#1788)

* merge dev branch front-end code (#1786)

* fix #1775 bug,delete process definition when process instance is running (#1790)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* fix udfs assignment and task instance page jump #1789 (#1791)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* update method checkDownloadPermission in order to fix issues 1770 (#1794)

* add check download resource permission in order to fix issues 1770

* update method checkDownloadPermission in order to fix issues 1770

* update method listUnauthorizedResource in order to fix issues 1770

* update method listUnauthorizedResource in order to fix issues 1770 (#1797)

* if login user is admin,it will has permission to query all udf function (#1799)

* Fixed space and icon display issues before and after the input box (#1798)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* donot submit task to queue if sub process (#1793)

* dockerfile modify (#1800)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* Fix data echo, style and popup cannot be closed (#1801)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value(#1810) (#1815)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* [Fix issue #1770]check udf and data source in order to fix issue 1770 (#1817)

* if login user is admin,it will has permission to query all udf function

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* check udf and data source in order to fix issue 1770

* revert common.properties

* update the test method name

* remove useless code and import in unit test

* refactor has permission and check permission

* #1813 remove "_001" after the master/server register path in zookeepe (#1824)

* donot submit task to queue if sub process

* [feature] #1813 remove "_001" after the master/server register path in zookeeper (#1820)

* change master/worker register path.

* remove "_" from register path.

* remove install.sh server.servlet.context-path replace (#1823)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface (#1827)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface

* rpm package add UI (#1846)

* [fix #1828] when the executor of process instance is not the owner of udf resouce, the path of the read resource file is incorrect (#1847)

* fix issue 1828:get the udf resource path error when create udf function

* update grantResources

* first verify whether udf resource is bound by udf function

* update grantResources

* update testListAuthorizedUdfFunc

* update getUserInfo in order to run success

* add LoggerServer RPC PORT modified. #1848 (#1849)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* add LoggerServer RPC PORT modified. #1848

* LoggerService Logger RPC PORT get Error modify . #1848 (#1857)

* fix #1775 bug,delete process definition when process instance is running

* revert CONTRIBUTING.md

* dockerfile modify

* dockerfile modify

* dockerfile modify

* remove install.sh server.servlet.context-path replace

* add LoggerServer RPC PORT modified. #1848

* LoggerService Logger RPC PORT get Error modify . #1848

* Fix UDF function list delete data without refresh and page data display widened #1851 (#1852)

* merge dev branch front-end code

* fix udfs assignment

* Fix task instance page jump

* fix udfs assignment and task instance page jump #1789

* Fixed space and icon display issues before and after the input box

* add license

* add license

* Fix data echo, style and popup cannot be closed

* Fix page number loading issue and dag not getting value

* Fix DAG add dependent project value exception # 1816 and keep requesting task interface

* Fix UDF function list delete data without refresh and page data display widened

* [Fix #1828]check whether has permission to download udf file or delete udf function (#1858)

* fix issue 1828:get the udf resource path error when create udf function

* update grantResources

* first verify whether udf resource is bound by udf function

* update grantResources

* update testListAuthorizedUdfFunc

* update getUserInfo in order to run success

* check whether has permission to download udf file or delete udf file

* update listAuthorizedResourceById in ResourceMapper.xml

* add log, don't swallow exception info (#1877)

* Added hints in Resource Center (#1891)

* update jackson version from 2.9.8 to 2.10.1 (#1892)

* update release notes and modify the plugin of rat (#1899)

* modify how to build (#1902)

* update release notes and modify the plugin of rat

* modify how to build

* [maven-release-plugin] prepare release 1.2.1

* [maven-release-plugin] prepare for next development iteration

* get authorized udf resources need exclude all created by self (#1905)

* get authorized udf resources need exclude all created by self

* reset pom version to 1.2.1-SNAPSHOT

* Update ReleaseNotes.md

add detail info

* Update ReleaseNotes.md

* update notice year form 2019 to 2019-2020 (#1907)

* [maven-release-plugin] prepare release 1.2.1

* [maven-release-plugin] prepare for next development iteration

* 当手动重跑历史活着补数的时候对shell和sql脚本中的[YYYYmmddd...]变量赋值与传递的日期,而不是当前日期

Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: bao liang <29528966+lenboo@users.noreply.github.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: elonlo <gamelifedong@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: zhangchunyang <18910529250@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: 谷雨 <lixun@guyudeMacBook-Pro.local>
2020-03-18 23:26:07 +08:00
gabry.wu
0e5fb86bef
Adapting partial code(file name start with N) to the sonar cloud rule (#2215) 2020-03-18 17:02:57 +08:00
break60
e388e343db dd branch flow node verification 2020-03-18 11:26:11 +08:00
break60
b90461fae9 Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev 2020-03-18 11:05:16 +08:00
xingchun-chen
7789e05c20
add LoginTest license (#2217)
Co-authored-by: chenxingchun <438044805@qq.com>
2020-03-18 10:03:03 +08:00
break60
99c28c7485
modify parameters (#2203)
* Delete worker group management page

* Modify workgroupid parameter name

* Worker group parameter modification

* modify parameters
2020-03-18 00:18:19 +08:00
qiaozhanwei
055bb28de4
worker fault tolerance modify (#2212)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-17 18:45:39 +08:00
xingchun-chen
c82758c505
e2e add the case of deleting user and tenant   (#2210)
* add e2e
2020-03-17 17:57:12 +08:00
qiaozhanwei
cc878505cd
1,master and worker register ip use OSUtils.getHost() 2,ProcessInstance host set ip:port format (#2209)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-17 16:38:00 +08:00
t1mon
7dd126da71
Optimize PropertyUtils instantiation. (#2199)
* Optimize PropertyUtils instantiation.

* Fix info error.
2020-03-17 15:01:42 +08:00
gabry.wu
c5e1c55c00
Adapting partial code(file name start with M #2) to the sonar cloud rule (#2179)
* Adapting partial code(file name start with M #2) to the sonar cloud rule

* remove unused import
2020-03-17 14:58:16 +08:00
qiaozhanwei
e31b48fdb2
1,get workergroup from zk modify 2,SpringConnectionFactory repeat load modify (#2201)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-17 11:52:31 +08:00
Rubik-W
69633a29c6
fix: Api server startup fail (#2198) 2020-03-17 11:10:55 +08:00
tswstarplanet
f28ba55014
fix ResultSet not close and reformat code (#2183)
* fix ResultSet not close

* fix ResultSet not close

* extract code of closing resource to a method

* remove redundant if condition

* modify e2e timeout
2020-03-16 18:56:36 +08:00
tswstarplanet
a3767642a5
change static field to instance field (#2190) 2020-03-16 18:06:13 +08:00
t1mon
f6f29f4abb
Replace Charset.forName("UTF-8") with StandardCharsets.UTF_8. (#2193) 2020-03-16 17:10:55 +08:00
Tboy
11dde20311
Refactor worker (#2189)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test

* move mybatis config from application.properties to SpringConnectionFactory

* move mybatis-plus config from application.properties to SpringConnectionFactory
2020-03-16 12:21:28 +08:00
break60
aa650b4426
Worker group parameter modification (#2188)
* Delete worker group management page

* Modify workgroupid parameter name

* Worker group parameter modification
2020-03-16 11:33:04 +08:00
gabry.wu
1658176ef5
Adapting partial code(file name start with P #1) to the sonar cloud rule (#2171) (#2187) 2020-03-15 21:25:52 +08:00
tswstarplanet
e0fc3171da
Cache HadoopUtils instance with specific days expire time (#2181)
* Cache HadoopUtils instance with 7 days expire time

* solve sonar issue

* add kerberos expire time config

* move KERBEROS_EXPIRE_TIME to Constants.java
2020-03-15 21:14:43 +08:00
gabry.wu
fcc57320d6
Adapting partial code(file name start with M #1) to the sonar cloud rule (#2173)
* Adapting partial code(file name start with M #1) to the sonar cloud rule

* remove unused import
2020-03-15 19:05:33 +08:00
Tboy
75efbb5402
Refactor worker (#2186)
* let quartz use the same datasource

* move master/worker config from dao.properties to each config
add master/worker registry test
2020-03-15 19:05:04 +08:00
gabry.wu
61ebbf2946
Adapting partial code(file name start with L) to the sonar cloud rule (#2171) 2020-03-14 19:22:24 +08:00
Rubik-W
e0389aebf6
fix: NullPointException in the case of task queuing (#2175) 2020-03-14 19:20:46 +08:00
gabry.wu
a076ae4c36
Adapting partial code(file name start with J) to the sonar cloud rule (#2165)
* Adapting partial code(file name start with J) to the sonar cloud rule

* fix unit failure
2020-03-14 18:12:14 +08:00
Rubik-W
80c6ce5711
The master and worker server exit exception #2163 (#2176)
* fix: #2163

* fix: format
2020-03-14 18:10:27 +08:00
DS
2e63dc9c96
Delete .asf.yaml (#2168) 2020-03-14 08:47:23 +08:00
DS
e98c6f3008
Update .asf.yaml (#2167) 2020-03-14 08:42:55 +08:00
DS
205af16d55
Create .asf.yaml (#2166) 2020-03-14 08:30:41 +08:00
gabry.wu
daad5efbab
use static method in static class JSON (#2156) 2020-03-13 23:53:22 +08:00
liwenhe1993
dd2c50a10d
Toolbar of the DAG has not prompt (#2161)
* Toolbar of the DAG has not prompt

* Remove duplication code
2020-03-13 23:38:38 +08:00
qiaozhanwei
71b11e2c08
DataxTask process test modify (#2162)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency,MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-13 15:28:22 +08:00
gaojun2048
a0088c1137
fix rpmbuild and DS Ambari plugin bug (#2144)
* Update quick-start.md

* 更新demo的地址

* add rpm plugin

* delete the git ignore files

* 测试git ignore

* 测试ignore

* optimize the .gitignore file

* add rpm package, delete unuse copy jar action in assembly

* delete docs dir

* add rpm plugin support

* add rpm plugin support

* add rpm plugin support

* dolphinscheduler-common have not bin dir

* dolphinscheduler-common have not bin dir

* delete unuse config about assembly

* add defineSatement

* add rpm plugin support

* add rpm plugin support

* add rpm plugin support

* add rpm support

* update install dirname , make front and backend together

* update rpm name

* update rpm name

* update rpm config

* add jars excludes

* add jars excludes

* add rpm plugion

* add rpm plugion

* add rpm plugion

* add auto create /opt/soft

* The install dir can not be fixed to apache-dolphinscheduler-incubating-1.2.1 , its version is 1.2.2

* install from rpm will create a link

* the ui install thrown rpm only support jetty not nginx

* the ui install thrown rpm only support jetty not nginx

* sql目录里加上soft_version文件

* env dir need in conf dir

* remove no use module

* add module

* updaate

* rpmbuild add dolphinscheduler-service module
2020-03-13 13:04:33 +08:00
gabry.wu
93aa160698
Adapting partial code(file name start with I) to the sonar cloud rule (#2155) 2020-03-13 13:02:51 +08:00
qiaozhanwei
607ec3d174
ProcedureTask process test modify (#2159)
* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-13 11:19:12 +08:00
dailidong
455d8c7a7c
Delete .asf.yaml (#2158) 2020-03-13 11:13:17 +08:00
Yelli
91ecc549a1
fix bug: data too long for column 'app_link' (#2146)
* fix bug: data to long for app_link field

* change app_link field length in 1.2.2 ddl

* change app_link field type to text
2020-03-12 21:52:48 +08:00
zhukai
68e4e5c4e0
add input cache for sqoop and conditions tasks (#2118)
Co-authored-by: zhukai <zhukai@ygsoft.com>
2020-03-12 18:08:49 +08:00
gabry.wu
7473d5e928
Adapting partial code(file name start with H) to the sonar cloud rule (#2145) 2020-03-12 18:06:10 +08:00
Rubik-W
9fda4da70e
fix: The toolbar buttons of Dag disable bug (#2124) 2020-03-12 18:00:33 +08:00
break60
712a1bc126
Limit customization file content to no more than 3000 lines(#2128) (#2141)
* Change DOM label

* Change name to lowercase

* Limit customization file content to no more than 3000 lines
2020-03-12 17:59:16 +08:00
t1mon
46e30b1e50
Add mail server configuration description information (#2143)
* Add mail server configuration description information

* Remove mail server configuration note in alert.properties
2020-03-12 17:50:37 +08:00
t1mon
16eeaf50b4
Fix javadoc and DataxTaskTest UT error (#2147)
* Fix javadoc and DataxTaskTest UT error

* Fix UT error
* LoggerServiceTest UT error
* WorkerGroupServiceTest UT error
Modify Host.of()
* Add address NPE check

* Trigger actions rerun

* Try to solve maven VM crash

* Try to solve maven VM crash 1

* Try to solve maven VM crash 2

* Try to solve maven VM crash 3
2020-03-12 17:41:32 +08:00