Commit Graph

518 Commits

Author SHA1 Message Date
李丛阳
ede7895083
Dev imp server process utils (#4263)
* Adds the comment on the WINDOWSATTERN

Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: 0002939 <licongyang@mininglamp.com>
2020-12-24 17:26:51 +08:00
Shiwen Cheng
f0ada2aa2d
[Fix-4289][*] Flink name with disappeared and unescaped problem (#4290)
* fix flink name not display in process definition editor

* fix flink name not escape problem

* simplify escape method
2020-12-23 21:19:08 +08:00
hailin0
3a28a71b00
[Improvement] Refactor code to support distributed tracing (#4270)
* Refactor code to support tracing

* Extension network protocol, support context and version
* Extension master asynchronous queue support context
* Extract scan task method from MasterSchedulerService for tracing

* fix

* fix

* add test case

* fix

* fix

Co-authored-by: hailin0 <hailin0@yeah.net>
2020-12-22 15:32:41 +08:00
xingchun-chen
55447348ed
Merge pull request #4237 from CalvinKirs/fix_date_convert_null
[FIX-3177]Task time parameter parsing error
2020-12-21 15:54:40 +08:00
Shiwen Cheng
c4d75443e1
[Fix-4271][server] Fix IOException or NoSuchFileException in logger server (#4272) 2020-12-21 09:07:49 +08:00
Tq
2486451221
remove AbstractTask.getCurTaskParamsClass()and replace with TaskParametersUtils.getParameters() (#4262)
* remove getCurTaskParamsClass() in AbstractTask.java and replace it with TaskParametersUtils.getParameters()

* remove unused imports in AbstractTask.java

* reformat
2020-12-20 21:52:52 +08:00
zhuangchong
3bcd3fbcd3
[Improvement-4069][server] When the tenant does not exist, the task execution should throw an exception (#4108)
* when  the tenant does not exist, the task execution should throw an exception

* remote method createWorkDirAndUserIfAbsent

* set the task status failed when the tenant code does not exist.

* add taskLog.

* update check os user exists

* update TaskExecuteThreadTest test method.

* solving sonar fail.
2020-12-20 21:17:39 +08:00
lgcareer
9fe5c3717d
[Fix-4222][Master]Add the priority queue to ensure that tasks are submitted according to priority. (#4250)
* [Fix-4222][Master]Add the priority queue to ensure that tasks are submitted according to priority.

* [Fix-4222][Master]Add the priority queue to ensure that tasks are submitted according to priority.

* [Fix-4222][Master]Remove useless import

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]add PeerTaskInstancePriorityQueueTest

* [Fix-4222][Master]Fix code smell

* [Fix-4222][Master]Reformat code style

* [Fix-4222][Master]Fix code smell

Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-12-18 17:28:37 +08:00
CalvinKirs
0f42f1588f add ut 2020-12-16 21:43:15 +08:00
CalvinKirs
c11a0c42f3 reformat 2020-12-16 16:41:52 +08:00
CalvinKirs
54ae6437bb fix time cover 2020-12-16 13:31:59 +08:00
CalvinKirs
7df03d91a1 fix time cover 2020-12-16 13:30:09 +08:00
Kirs
d7b3936205
[FIX-3177]Task time parameter parsing error (#4228)
* [FIX-3177]Task time parameter parsing error
when system.datetime !=null $[datetime] = system.datetime
else $[datetime] = current time

* remove unused import
2020-12-15 16:24:31 +08:00
Kirs
8919e3838f
[FIX-#3177]Task time parameter parsing error (#4224)
* [FIX-#3177]Task time parameter parsing error

rerun schedule time error

this closes #3177

* [FIX-#3177]Task time parameter parsing error

rerun schedule time error

this closes #3177
2020-12-15 10:30:36 +08:00
Kirs
a13e737eb4
[FIX-#4172][server-worker] kill task NPE (#4182)
* [FIX-#4172][server-worker] kill task NPE

The cache task will be sent when the Process is generated. Before that, if a kill task appears, then NPE will appear
Modification method: write into the cache when the task is received, and mark it as preData
If the task is killed before the Process is generated, delete the cache directly at this time
It will be judged before the process is generated. If the task has been killed, it will not be executed.
After the new process is created, write it into the cache, and judge again, if kill, then kill the process.

this closes #4172

* Delete the commented out code
Add spring beans

* code smell

* add test

* add test

* fix error

* test

* test

* revert

* fix error
2020-12-11 15:41:35 +08:00
BoYiZhang
b3120a74d2
[Feature-4138][Master] dispatch workgroup error add sleep time (#4139)
* When there are tasks with assignment failure and the number of tasks in the current task queue is less than 10, sleep for 1 second

* When there are tasks with assignment failure and the number of tasks in the current task queue is less than 10, sleep for 1 second

* fix code smell & code style

* fix code smell & code style

Co-authored-by: zhanglong <zhanglong@ysstech.com>
2020-12-11 15:35:40 +08:00
Shiwen Cheng
cbc30b4900
[Fix-3457][flink] fix flink args build problem (#4166)
* [Fix][Flink] fix flink args build problem

* [Fix][Flink] fix FlinkArgsUtilsTest

* [Improvement][UI] hide version and cluster input when deployMode is local
2020-12-10 22:37:21 +08:00
Kirs
aa19a06abb
[FIX-PR-4097][server-master]task ack miss (#4189)
When the message of successful execution arrives earlier than
the message of ack,
the message of ack will be discarded,
resulting in some information missing
2020-12-10 10:57:12 +08:00
Yelli
aa0974fd1f
split sqoop import hive database and table (#4141) 2020-12-02 15:50:50 +08:00
felix.wang
3e411d075f
[Improvement][Code style] FIX SPELL WAITTING TO WAITING , etc. (#4118)
* FIX SPELL

* FIX SPELL AND  Optimizing code conventions

* add ut  cannot construct process instance, return null;

* add ut testExportProcessMetaData

* add ut testExportProcessMetaData

* add ut testImportProcessSchedule

* add ut MasterExecThreadTest

* add ut MasterExecThreadTest

* add ut testSubProcessViewTree

* add ut testComplementWithStartNodeList

* add ut testRecurseFindSubProcessId

* add ut testRecurseFindSubProcessId

* add ut testRecurseFindSubProcessId
2020-12-01 08:58:55 +08:00
Yelli
1cf40e1d1e
fix sqoop task jdbc string contains special char (#4105) 2020-11-27 09:46:23 +08:00
dailidong
656ec295b9
Merge pull request #4097 from CalvinKirs/ack_status
[FIX-#4084][server]fix taskInstance state change error
2020-11-26 14:30:29 +08:00
CalvinKirs
c902268021 fix task instance status judgment error 2020-11-26 11:01:34 +08:00
CalvinKirs
0586e7b160 test 2020-11-25 22:03:26 +08:00
CalvinKirs
cc14b24326 test 2020-11-25 18:11:55 +08:00
CalvinKirs
d5769999d5 remove assert 2020-11-25 15:16:19 +08:00
CalvinKirs
2a9e789f68 test 2020-11-25 14:47:00 +08:00
CalvinKirs
b3991d9dee Time is too small and the task is not finished 2020-11-25 13:43:14 +08:00
CalvinKirs
c6a686a710 Time is too small and the task is not finished 2020-11-25 13:27:04 +08:00
Yelli
145314b782
[FIX-4034][server] fix sqoop import fail (#4036)
* fix #4043, sqoop import query fail

* fix #4043, sqoop task hard code & code style

* add license for SqoopConstants

* add private constructor for SqoopConstants

* fixed sqoop mysql pwd have special character

* fix checkstyle

* fix sqoop task log

* remove unused constants
2020-11-25 11:11:44 +08:00
CalvinKirs
a164d5f990 Time is too small and the task is not finished 2020-11-25 10:23:54 +08:00
CalvinKirs
8286af79d3 Time is too small and the task is not finished 2020-11-25 09:38:58 +08:00
CalvinKirs
858c4f3287 code smell 2020-11-25 09:20:58 +08:00
CalvinKirs
6c1a2ebeab add taskResponseTest 2020-11-24 22:49:39 +08:00
CalvinKirs
5355927b79 [FIX-#4083][server]fix taskInstance state change error
Concurrent processing of ack message and result message causes the execution sequence to be wrong

# this close # 4083
2020-11-24 16:59:26 +08:00
Eights-LI
802de19cc0 using OSUtils.execCmd when kill yarn app 2020-11-20 21:40:22 +08:00
Eights-LI
c24dbc448d deal with magic value 2020-11-17 22:25:25 +08:00
Eights-LI
d3397b8ddb Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev-kill-yarn-job 2020-11-17 21:39:56 +08:00
Yelli
1c5be9acf1
[Feature-4050][server] Spark task support udv inject (#4061)
* #4050 spark task support udv inject

* modify spark task UT

* modify sparkTaskExecutionCtx

* add exp for spark task get main jar method
2020-11-17 15:24:55 +08:00
Eights-LI
cb669b5900 deal with magic value 2020-11-16 20:34:32 +08:00
Eights-LI
78445084ef Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev-kill-yarn-job 2020-11-16 20:25:37 +08:00
BoYiZhang
fe3026627f
[Feature-3985][Datax] Datax supports setting up running memory (#3986)
* Datax supports setting up running memory

* Datax supports setting up running memory

* Datax supports setting up running memory

* When running a task, the resource file is lost, which results in an error

* add unit test

* add unit test

* add unit test

* add test unit

* add test unit

* add test unit

* fix code smell

* add test unit

* add test unit
2020-11-16 10:55:20 +08:00
Kirs
8a92105f67
[FIX_BUG][server-test] dismiss of server warm-up time in RoundRobinSelectorTest (#4067) 2020-11-14 14:59:20 +08:00
Eights-LI
aac762a267 modify ProcessUtils & ProcessUtilsTest 2020-11-13 16:20:04 +08:00
Eights-LI
1280155956 add ProcessUtils UT 2020-11-12 12:18:52 +08:00
Eights-LI
134cef36ff Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev 2020-11-12 12:17:49 +08:00
Eights-LI
2cad271749 add ProcessUtils UT 2020-11-11 22:58:51 +08:00
baoliang
c73360c78b merge from 1.3.3-release 2020-11-11 16:15:26 +08:00
baoliang
e142b8949d merge from 1.3.3-release 2020-11-11 15:00:44 +08:00
baoliang
98d4e1f5f2 refactor 2020-11-10 15:30:51 +08:00