Commit Graph

509 Commits

Author SHA1 Message Date
Tq
2486451221
remove AbstractTask.getCurTaskParamsClass()and replace with TaskParametersUtils.getParameters() (#4262)
* remove getCurTaskParamsClass() in AbstractTask.java and replace it with TaskParametersUtils.getParameters()

* remove unused imports in AbstractTask.java

* reformat
2020-12-20 21:52:52 +08:00
zhuangchong
3bcd3fbcd3
[Improvement-4069][server] When the tenant does not exist, the task execution should throw an exception (#4108)
* when  the tenant does not exist, the task execution should throw an exception

* remote method createWorkDirAndUserIfAbsent

* set the task status failed when the tenant code does not exist.

* add taskLog.

* update check os user exists

* update TaskExecuteThreadTest test method.

* solving sonar fail.
2020-12-20 21:17:39 +08:00
lgcareer
9fe5c3717d
[Fix-4222][Master]Add the priority queue to ensure that tasks are submitted according to priority. (#4250)
* [Fix-4222][Master]Add the priority queue to ensure that tasks are submitted according to priority.

* [Fix-4222][Master]Add the priority queue to ensure that tasks are submitted according to priority.

* [Fix-4222][Master]Remove useless import

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]add PeerTaskInstancePriorityQueueTest

* [Fix-4222][Master]Fix code smell

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]Fix code smell

Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-12-18 17:28:37 +08:00
Kirs
d7b3936205
[FIX-3177]Task time parameter parsing error (#4228)
* [FIX-3177]Task time parameter parsing error
when system.datetime !=null $[datetime] = system.datetime
else $[datetime] = current time

* remove unused import
2020-12-15 16:24:31 +08:00
Kirs
8919e3838f
[FIX-#3177]Task time parameter parsing error (#4224)
* [FIX-#3177]Task time parameter parsing error

rerun schedule time error

this closes #3177

* [FIX-#3177]Task time parameter parsing error

rerun schedule time error

this closes #3177
2020-12-15 10:30:36 +08:00
Kirs
a13e737eb4
[FIX-#4172][server-worker] kill task NPE (#4182)
* [FIX-#4172][server-worker] kill task NPE

The cache task will be sent when the Process is generated. Before that, if a kill task appears, then NPE will appear
Modification method: write into the cache when the task is received, and mark it as preData
If the task is killed before the Process is generated, delete the cache directly at this time
It will be judged before the process is generated. If the task has been killed, it will not be executed.
After the new process is created, write it into the cache, and judge again, if kill, then kill the process.

this closes #4172

* Delete the commented out code
Add spring beans

* code smell

* add test

* add test

* fix error

* test

* test

* revert

* fix error
2020-12-11 15:41:35 +08:00
BoYiZhang
b3120a74d2
[Feature-4138][Master] dispatch workgroup error add sleep time (#4139)
* When there are tasks with assignment failure and the number of tasks in the current task queue is less than 10, sleep for 1 second

* When there are tasks with assignment failure and the number of tasks in the current task queue is less than 10, sleep for 1 second

* fix code smell & code style

* fix code smell & code style

Co-authored-by: zhanglong <zhanglong@ysstech.com>
2020-12-11 15:35:40 +08:00
Shiwen Cheng
cbc30b4900
[Fix-3457][flink] fix flink args build problem (#4166)
* [Fix][Flink] fix flink args build problem

* [Fix][Flink] fix FlinkArgsUtilsTest

* [Improvement][UI] hide version and cluster input when deployMode is local
2020-12-10 22:37:21 +08:00
Kirs
aa19a06abb
[FIX-PR-4097][server-master]task ack miss (#4189)
When the message of successful execution arrives earlier than
the message of ack,
the message of ack will be discarded,
resulting in some information missing
2020-12-10 10:57:12 +08:00
Yelli
aa0974fd1f
split sqoop import hive database and table (#4141) 2020-12-02 15:50:50 +08:00
felix.wang
3e411d075f
[Improvement][Code style] FIX SPELL WAITTING TO WAITING , etc. (#4118)
* FIX SPELL

* FIX SPELL AND  Optimizing code conventions

* add ut  cannot construct process instance, return null;

* add ut testExportProcessMetaData

* add ut testExportProcessMetaData

* add ut testImportProcessSchedule

* add ut MasterExecThreadTest

* add ut MasterExecThreadTest

* add ut testSubProcessViewTree

* add ut testComplementWithStartNodeList

* add ut testRecurseFindSubProcessId

* add ut testRecurseFindSubProcessId

* add ut testRecurseFindSubProcessId
2020-12-01 08:58:55 +08:00
Yelli
1cf40e1d1e
fix sqoop task jdbc string contains special char (#4105) 2020-11-27 09:46:23 +08:00
dailidong
656ec295b9
Merge pull request #4097 from CalvinKirs/ack_status
[FIX-#4084][server]fix taskInstance state change error
2020-11-26 14:30:29 +08:00
CalvinKirs
c902268021 fix task instance status judgment error 2020-11-26 11:01:34 +08:00
CalvinKirs
0586e7b160 test 2020-11-25 22:03:26 +08:00
CalvinKirs
cc14b24326 test 2020-11-25 18:11:55 +08:00
CalvinKirs
d5769999d5 remove assert 2020-11-25 15:16:19 +08:00
CalvinKirs
2a9e789f68 test 2020-11-25 14:47:00 +08:00
CalvinKirs
b3991d9dee Time is too small and the task is not finished 2020-11-25 13:43:14 +08:00
CalvinKirs
c6a686a710 Time is too small and the task is not finished 2020-11-25 13:27:04 +08:00
Yelli
145314b782
[FIX-4034][server] fix sqoop import fail (#4036)
* fix #4043, sqoop import query fail

* fix #4043, sqoop task hard code & code style

* add license for SqoopConstants

* add private constructor for SqoopConstants

* fixed sqoop mysql pwd have special character

* fix checkstyle

* fix sqoop task log

* remove unused constants
2020-11-25 11:11:44 +08:00
CalvinKirs
a164d5f990 Time is too small and the task is not finished 2020-11-25 10:23:54 +08:00
CalvinKirs
8286af79d3 Time is too small and the task is not finished 2020-11-25 09:38:58 +08:00
CalvinKirs
858c4f3287 code smell 2020-11-25 09:20:58 +08:00
CalvinKirs
6c1a2ebeab add taskResponseTest 2020-11-24 22:49:39 +08:00
CalvinKirs
5355927b79 [FIX-#4083][server]fix taskInstance state change error
Concurrent processing of ack message and result message causes the execution sequence to be wrong

# this close # 4083
2020-11-24 16:59:26 +08:00
Eights-LI
802de19cc0 using OSUtils.execCmd when kill yarn app 2020-11-20 21:40:22 +08:00
Eights-LI
c24dbc448d deal with magic value 2020-11-17 22:25:25 +08:00
Eights-LI
d3397b8ddb Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev-kill-yarn-job 2020-11-17 21:39:56 +08:00
Yelli
1c5be9acf1
[Feature-4050][server] Spark task support udv inject (#4061)
* #4050 spark task support udv inject

* modify spark task UT

* modify sparkTaskExecutionCtx

* add exp for spark task get main jar method
2020-11-17 15:24:55 +08:00
Eights-LI
cb669b5900 deal with magic value 2020-11-16 20:34:32 +08:00
Eights-LI
78445084ef Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev-kill-yarn-job 2020-11-16 20:25:37 +08:00
BoYiZhang
fe3026627f
[Feature-3985][Datax] Datax supports setting up running memory (#3986)
* Datax supports setting up running memory

* Datax supports setting up running memory

* Datax supports setting up running memory

* When running a task, the resource file is lost, which results in an error

* add unit test

* add unit test

* add unit test

* add test unit

* add test unit

* add test unit

* fix code smell

* add test unit

* add test unit
2020-11-16 10:55:20 +08:00
Kirs
8a92105f67
[FIX_BUG][server-test] dismiss of server warm-up time in RoundRobinSelectorTest (#4067) 2020-11-14 14:59:20 +08:00
Eights-LI
aac762a267 modify ProcessUtils & ProcessUtilsTest 2020-11-13 16:20:04 +08:00
Eights-LI
1280155956 add ProcessUtils UT 2020-11-12 12:18:52 +08:00
Eights-LI
134cef36ff Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev 2020-11-12 12:17:49 +08:00
Eights-LI
2cad271749 add ProcessUtils UT 2020-11-11 22:58:51 +08:00
baoliang
c73360c78b merge from 1.3.3-release 2020-11-11 16:15:26 +08:00
baoliang
e142b8949d merge from 1.3.3-release 2020-11-11 15:00:44 +08:00
baoliang
98d4e1f5f2 refactor 2020-11-10 15:30:51 +08:00
baoliang
3f88a00174 refactor ut 2020-11-10 11:07:49 +08:00
baoliang
db3d9fc27a refactor ut 2020-11-10 10:15:41 +08:00
Eights-LI
8fdd6f547d fix #3900 kill multi yarn app in one job 2020-11-09 22:02:07 +08:00
baoliang
3bb0f3bfb2 refactor ut test 2020-11-09 18:51:00 +08:00
baoliang
30cf144421 refactor ut test 2020-11-09 18:33:57 +08:00
baoliang
b3dbc63c72 merge from 1.3.3-release 2020-11-09 17:33:06 +08:00
baoliang
d0c8dadc94 merge from 1.3.3-release 2020-11-09 17:27:57 +08:00
baoliang
ce4d123347 merge from 1.3.3-release 2020-11-09 17:24:44 +08:00
baoliang
8f5f1a61c5 merge from 1.3.3-release 2020-11-09 16:58:58 +08:00