tswstarplanet
afb907b4b0
fix unequal number of arguments and placeholder ( #2149 )
2020-03-12 17:41:10 +08:00
tswstarplanet
e3065dbfcc
fix field access modifier ( #2150 )
2020-03-12 17:39:25 +08:00
tswstarplanet
e27ba3fd97
use logger to print exception ( #2151 )
2020-03-12 17:38:38 +08:00
liwenhe1993
ebfacbfcfa
docker run failed when docker command was docker run -d ( #2125 )
...
* docker run failed when docker command was `docker run -d`
1. add tini
2. replace `tee` command by `tail` command in startup.sh
3. api-server need to support zookeeper
* modify `check.sh` in dockerfile
* add init directories and log files
2020-03-12 10:11:07 +08:00
Jave-Chen
b03e0a02c5
fix wrong sonar analysis with checkout V2. ( #2148 )
2020-03-12 10:09:13 +08:00
break60
84bcdc7896
Limit customization file content to no more than 3000 lines
2020-03-11 11:03:47 +08:00
AhahaGe
1db4ac6b63
edit README.md modify word from expand to extend ( #2138 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
2020-03-10 22:28:40 +08:00
Tboy
7f72df4ca2
let quartz use the same datasource ( #2137 )
2020-03-10 22:23:23 +08:00
AhahaGe
ce17531024
add .asf.yaml file ( #2136 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
2020-03-10 22:00:40 +08:00
gabry.wu
ebf7a980d2
Adapting partial code(file name start with F) to the sonar cloud rule ( #2045 )
...
* Adapting partial code(file name start with F) to the sonar cloud rule
* add more unit test
* add License
* add includes configuration to maven-surefire-plugin
* fix getResourceFilesList incorrect logic
2020-03-10 18:59:45 +08:00
qiaozhanwei
fd2c2eeb32
1，encapsulate the parameters required by sqltask 2，SQLTask optimization ( #2135 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-10 17:15:27 +08:00
lgcareer
bbe2cd4864
refactor worker group ( #2132 )
...
* replace worker group id with worker group
* add worker group field in ddl
* remove worker group id
2020-03-10 17:10:47 +08:00
Tboy
621ebf6090
Merge branch 'refactor-worker' into refactor-worker
2020-03-10 11:06:44 +08:00
tison
9224b49b58
access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability ( #2113 )
...
* access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability
Current implementation relies on `sun.misc.JavaIOFileDescriptorAccess`
which is only accessible on oraclejdk8.
Basically the demand is getting & setting `handle` field of
`FileDescriptor`, so we can directly do that with reflection.
Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.
* harden initialization of ProcessImplForWin32
* ignore ShellTaskTest#testHandleForWindows outside Windows
2020-03-09 19:06:41 +08:00
dailidong
f551913249
split logback config into seperate child module logback ( #2120 )
...
* update logback
* update log
2020-03-09 19:03:51 +08:00
qiaozhanwei
20af8ff93e
cancelTaskInstance set TaskExecutionContext host,logPath,executePath ( #2126 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-09 16:09:25 +08:00
break60
d7ad1d8b05
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
2020-03-09 16:04:55 +08:00
Tboy
fd6f13fff7
Refactor worker ( #2121 )
...
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
2020-03-09 10:29:38 +08:00
Technoboy-
932fc72794
Merge branch 'refactor-worker' of github.com:Technoboy-/incubator-dolphinscheduler into refactor-worker
2020-03-09 10:23:39 +08:00
Technoboy-
cc97052cc0
refactor MasterSchedulerService
2020-03-09 10:23:25 +08:00
Tboy
645529e4cc
Merge pull request #24 from apache/refactor-worker
...
Refactor worker
2020-03-09 10:18:10 +08:00
gabry.wu
450a1f56fc
Adapting partial code(file name start with E) to the sonar cloud rule ( #1999 )
...
* Adapting partial code(file name start with E) to the sonar cloud rule
* remove isEmpty invoke
* resolve conflicts
2020-03-08 21:28:49 +08:00
gabry.wu
ad381fcb71
move UT class to reasonable package ( #2116 )
2020-03-08 21:22:07 +08:00
Rubik-W
275df85250
fix: remove redundant import class ( #2110 )
2020-03-08 16:00:47 +08:00
break60
290772dfaa
Modify workgroupid parameter name ( #2105 )
...
* Delete worker group management page
* Modify workgroupid parameter name
2020-03-08 15:59:13 +08:00
tswstarplanet
b2268b1af9
use stringbuilder to append string ( #2108 )
2020-03-08 15:57:06 +08:00
liwenhe1993
89b42db3b4
Add docker internal server check script ( #2092 )
...
* add environment variables add checkpoint.sh file to docker image
1. add `POSTGRESQL_DATABASE` environment variable
2. add `DOLPHINSCHEDULER_DATA_BASEDIR_PATH` environment variable
3. add `DOLPHINSCHEDULER_DATA_DOWNLOAD_BASEDIR_PATH` environment variable
4. add `DOLPHINSCHEDULER_PROCESS_EXEC_BASEPATH` environment variable
5. add checkpoint.sh file to check server process
6. modify `README.md` and `README_zh_CN.md`
* add environment variables add checkpoint.sh file to docker image
1. add `POSTGRESQL_DATABASE` environment variable
2. add `DOLPHINSCHEDULER_DATA_BASEDIR_PATH` environment variable
3. add `DOLPHINSCHEDULER_DATA_DOWNLOAD_BASEDIR_PATH` environment variable
4. add `DOLPHINSCHEDULER_PROCESS_EXEC_BASEPATH` environment variable
5. add checkpoint.sh file to check server process
6. modify `README.md` and `README_zh_CN.md`
* add logger port and modify dolphinscheduler_env to dolphinscheduler_env.sh
2020-03-08 15:56:24 +08:00
Tboy
0febd9530c
Refactor worker ( #2115 )
...
* refactor worker registry
* refactor master server
2020-03-08 15:54:55 +08:00
Technoboy-
a4c2dfa311
refactor master server
2020-03-08 13:26:09 +08:00
Tboy
c3293887fa
Merge pull request #23 from apache/refactor-worker
...
refactor worker registry (#2107 )
2020-03-08 13:22:04 +08:00
Tboy
d617f9df32
refactor worker registry ( #2107 )
2020-03-08 12:06:31 +08:00
gabry.wu
0e1dd8d7d9
remove incorrect or unnecessary URL tag ( #2099 )
2020-03-07 18:07:53 +08:00
Technoboy-
76fd384a2e
refactor worker registry
2020-03-06 21:13:48 +08:00
Tboy
a51f3ea7c7
Merge pull request #22 from apache/refactor-worker
...
Refactor worker
2020-03-06 20:47:04 +08:00
Tboy
5c621009b5
Merge branch 'refactor-worker' into refactor-worker
2020-03-06 20:46:54 +08:00
qiaozhanwei
043df7e189
WorkerServer refactor ( #2106 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-06 18:58:53 +08:00
Tboy
4eebde8355
Refactor worker ( #2103 )
...
* refactor kill logic
* refactor ExecutionContext
* refactor worker
2020-03-06 18:18:49 +08:00
Technoboy-
aeb8b852e7
refactor worker
2020-03-06 18:06:42 +08:00
Tboy
b42357a16d
Merge pull request #21 from apache/refactor-worker
...
TaskPriority refactor (#2097 )
2020-03-06 17:12:30 +08:00
qiaozhanwei
eb45ff9e83
TaskPriority refactor ( #2097 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
2020-03-06 12:02:14 +08:00
dailidong
4bb97dfb8b
fix actions/checkout@v2 problem ( #2095 )
2020-03-05 23:53:12 +08:00
dailidong
6070738acb
Support worker server to run bat script ( #2023 )
...
* Support worker server to run bat script
1. Reimplement ProcessImpl.java, ProcessEnvironment.java and ProcessBuilder.java for Windows
2. Modify shell task code for windows
3. Add ASF License
* Add Unit Test
2020-03-05 23:34:37 +08:00
dailidong
1f92b4c4db
Optimize ParameterUtils.curingGlobalParams() execution efficiency ( #2090 )
...
* Optimize ParameterUtils.curingGlobalParams() execution efficiency
* Remove excess null check
2020-03-05 23:21:10 +08:00
Tboy
9964cf9057
Merge pull request #20 from apache/refactor-worker
...
task prioriry refator (#2094 )
2020-03-05 19:01:47 +08:00
Tboy
7ed28af4fe
Upgrade curator version ( #2083 )
...
* upgrade curator version
issue: https://github.com/apache/incubator-dolphinscheduler/issues/2020
curator issue: https://github.com/apache/curator/pull/297
* add DefaultEnsembleProvider override
* add some unit test
* add License
2020-03-05 19:00:06 +08:00
qiaozhanwei
8aaee39aa2
task prioriry refator ( #2094 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
2020-03-05 18:25:48 +08:00
dailidong
4230d5a737
add embedded database h2 and update embedded zookeeper ( #2082 )
...
add embedded database h2 and update embedded zookeeper
2020-03-05 17:32:40 +08:00
dailidong
6ec85540bf
fix typo ( #2093 )
2020-03-05 16:36:20 +08:00
Jave-Chen
d33b883420
Merge pull request #2085 from khadgarmage/ci-action-alter
...
action push delete
2020-03-05 14:30:19 +08:00
xiaochun.liu
18d7ff274b
action push delete
2020-03-05 11:33:24 +08:00
