test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-03 12:48:46 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
6,850 Commits 27 Branches 59 Tags 276 MiB
d94797dfbc
Commit Graph

127 Commits

Author SHA1 Message Date
Kerwin
d94797dfbc
Remove logger header in task log file (#11555) (#11968) 2022-09-16 13:03:22 +08:00
caishunfeng
43d9c00259
[Cherry-pick][Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 #11864 (#11949) 
Co-authored-by: Yann Ann <2993643785@qq.com>
 2022-09-15 10:36:34 +08:00
Jiajie Zhong
889fd580e3 [maven-release-plugin] prepare for next development iteration 2022-08-05 18:16:57 +08:00
Jiajie Zhong
9efd1ace78 [maven-release-plugin] prepare release 3.0.0 2022-08-05 18:16:57 +08:00
Wenjun Ruan
230adbf621
Fix TaskGroup cannot work and will cause master dead loop (#11254) (#11318) 
* Fix TaskGroup cannot work and will cause master dead loop

* Remove acquireTaskGroupAgain in ProcessServiceImpl

(cherry picked from commit acd3d3fab1)
 2022-08-05 17:40:22 +08:00
xiangzihao
a05d7cc8f4 fix_10514 (#10568) 
(cherry picked from commit 0f38217b12)
 2022-08-03 19:41:14 +08:00
xiangzihao
5e129d478e add datasource health check to the healthcheck endpoint (#10588) 
(cherry picked from commit 1111371c9a)
 2022-08-03 19:41:14 +08:00
xiangzihao
b0113a9b64 dependent node log output optimization and fix checkstyle (#10593) 
(cherry picked from commit 397e51fa63)
 2022-08-03 19:41:14 +08:00
Wenjun Ruan
072672a127 Fix recovery from failed task will dead loop (#11239) 
(cherry picked from commit 04f3aa9713)
 2022-08-02 19:07:08 +08:00
caishunfeng
034475bfbe [Bug-11101] fix task failover NPE (#11168) 
(cherry picked from commit 6c7e00c4aa)
 2022-07-28 11:14:33 +08:00
caishunfeng
cec5e88319
[Bug-11071] Fix task local params NPE (#11166) 
* fix local param NPE

* delete empty files
 2022-07-28 09:43:58 +08:00
caishunfeng
b9e3187ad5 [Feature-10871] add workflow executing data query (#10875) 
* add workflow executing data query
* fix sonar check for interrupted
 2022-07-19 16:10:42 +08:00
Wenjun Ruan
5a2ea0b76b Fix compile error 2022-07-19 12:30:26 +08:00
Wenjun Ruan
648d469b62 Return status 2022-07-19 12:27:49 +08:00
Wenjun Ruan
f3250bf5fa Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979) 
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed

* Move heartbeat error threshold to applicaiton.yml

(cherry picked from commit 2be1d4bf0a)
 2022-07-19 12:23:20 +08:00
Wenjun Ruan
17f0be5bfb Kill task when we do master failover to avoid task lost (#10997) 
(cherry picked from commit 998e4d74dd)
 2022-07-19 12:22:26 +08:00
Wenjun Ruan
44ddb6908e Fix kill yarn job error when failover caused by doesn't set ProcessDefinition (#10948) 
(cherry picked from commit b245e7c973)
 2022-07-19 12:22:13 +08:00
Wenjun Ruan
812d7a8f26 Fix taskInstance's host is not worker nettyServer address (#10926) 
* Fix taskInstance's host is not worker nettyServer address

* Remove unnecessary mock

(cherry picked from commit df0416c193)
 2022-07-19 12:21:59 +08:00
Wenjun Ruan
527ee472fb Catch exception when check state in StateWheelExecuteThread (#10908) 
* Catch exception when check state

(cherry picked from commit 2a67866718)
 2022-07-19 12:21:20 +08:00
Wenjun Ruan
9ee20cffef [Fix-10827] Fix network error cause worker cannot send message to master (#10886) 
* Fix network error cause worker cannot send message to master

(cherry picked from commit cade66a9b6)
 2022-07-19 12:20:18 +08:00
Wenjun Ruan
b259deb196 [Fix-10854] Fix database restart may lost task instance status (#10866) 
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status

(cherry picked from commit f639a2eed4)
 2022-07-19 12:12:57 +08:00
Wenjun Ruan
71edaf41a2 [Fix-10842] Fix master/worker failover will cause status incorrect (#10839) 
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance

(cherry picked from commit 3f69ec8f28)
 2022-07-19 12:08:32 +08:00
Wenjun Ruan
4fc9bce444 [Fix-10785] Fix state event handle error will not retry (#10786) 
* Fix state event handle error will not retry

* Use state event handler to deal with the event

(cherry picked from commit 67d14fb7b3)
 2022-07-19 12:03:15 +08:00
Wenjun Ruan
04c47034d4 Add task prepare metrics 2022-07-19 11:52:18 +08:00
Wenjun Ruan
7500e99682 [Fix-10785] Fix state event handle error will not retry (#10786) 
* Fix state event handle error will not retry

* Use state event handler to deal with the event

(cherry picked from commit 67d14fb7b3)
 2022-07-19 11:52:06 +08:00
Wenjun Ruan
3b923e5933 [Fix-10666] Workflow submit failed will still in memory and never retry (#10667) 
* Workflow submit failed will still in memory and never retry

(cherry picked from commit 35a10d092f)
 2022-07-19 11:47:04 +08:00
Wenjun Ruan
6c83967ebe [Improvement-10617] Add comment in slot check (#10618) 
(cherry picked from commit 247ca4ae8a)
 2022-07-19 11:42:31 +08:00
Wenjun Ruan
4b224ae2e5 Validate master/worker config (#10649) 
(cherry picked from commit 35b25da863)
 2022-07-19 11:42:17 +08:00
Wenjun Ruan
db31deb54f [Bug] [Master] Worker failover will cause task cannot be failover (#10631) 
* fix worker failover may lose event

(cherry picked from commit 66624c5c86)
 2022-07-19 11:41:25 +08:00
Wenjun Ruan
fc1c1f6ad1 add CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST 2022-07-19 11:30:47 +08:00
Wenjun Ruan
3ab9ee13fc Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516) 
* Optimize master log, add workflow instance id and task instance id in log

* Use MDC to set the workflow info in log4j

* Add workflowInstanceId and taskInstanceId in MDC

(cherry picked from commit db595b3eff)
 2022-07-19 11:29:41 +08:00
Wenjun Ruan
9a28d32057 Remove the schedule thread in LowerWeightHostManager (#10310) 
(cherry picked from commit b100f6c489)
 2022-07-19 11:23:59 +08:00
Wenjun Ruan
90c87f0121 [Fix-10413] Fix Master startup failure the server still hang (#10500) 
* Fix Master startup failure the server still hang

(cherry picked from commit 117f78ec4b)
 2022-07-19 11:22:53 +08:00
Wenjun Ruan
9a59054655 Fix PeerTaskInstancePriorityQueue cannot contains method use taskInstanceId to check (#10491) 
(cherry picked from commit 0bdfa0cff9)
 2022-07-19 11:22:33 +08:00
Wenjun Ruan
9a4c7f876a Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479) 
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

(cherry picked from commit ad2646ff1f)
 2022-07-19 11:22:16 +08:00
Wenjun Ruan
52815975bc Add some warning log in master (#10383) 
* Add some warn log in master

* fix may skip sleep

(cherry picked from commit b0d9d3f9ab)
 2022-07-19 11:13:39 +08:00
Wenjun Ruan
318a8e3ae0 [Feature][metrics] Add master, worker metrics (#10326) 
* Add mater metrics

* fix UT

* Add url to mysql profile

* Add worker metrics

* Update grafana config

* Add system metrics doc

* Add process failover counter

* Add metrics image

* Change jpg to png

* Add command insert metrics

* Fix UT

* Revert UT

(cherry picked from commit e21d7b1551)
 2022-07-19 11:12:28 +08:00
Wenjun Ruan
81cadd15d2 Optimize MasterServer, add MasterRPCService (#10371) 
* Optimize MasterServer, avoid NPE

(cherry picked from commit 3ecbee3885)
 2022-07-19 11:10:08 +08:00
Wenjun Ruan
4ceb420873 Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479) 
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

(cherry picked from commit ad2646ff1f)
 2022-07-19 11:02:57 +08:00
WangJPLeo
7b7ec0f20f Complement numbers will run in a loop under the serial strategy fixed. (#10862) 
* Complement numbers will run in a loop under the serial strategy fixed.

* e2e rerun
 2022-07-18 17:02:42 +08:00
devosend
7ddaa2f47d [maven-release-plugin] prepare for next development iteration 2022-06-28 16:41:31 +08:00
devosend
0a1b9bdd52 [maven-release-plugin] prepare release 3.0.0-beta-2 2022-06-28 16:41:30 +08:00
devosend
d68dcda2bb [chore] pre-release change pom.xml 2022-06-28 08:24:21 +08:00
Jiajie Zhong
57ade38939 [maven-release-plugin] prepare release 3.0.0-beta-1 2022-05-26 20:26:32 +08:00
旺阳
e08b08efdd [improve] Change Mysql Driver (#10220) 
(cherry picked from commit aba5f8a40e)
 2022-05-25 23:42:28 +08:00
BaoLiang
b016037a6f [BUG][TaskGroup] Task group does not take effect (#10093) 
* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

(cherry picked from commit ee2b855ced)
 2022-05-23 14:46:14 +08:00
xiangzihao
98576cb509 [Fix-10049] Conditions Task branch flow failed (#10077) 
(cherry picked from commit 225cb332d1)
 2022-05-23 14:43:52 +08:00
caishunfeng
aa51c66d91 [Bug][Master] fix master task failover (#10065) 
* fix master task failover

* ui

(cherry picked from commit 0cc0ee77fa)
 2022-05-23 14:42:41 +08:00
WangJPLeo
7b0e6fe5ec [Fix-9975] The selected task instance was recreated when the Master service fail… (#9976) 
* The selected task instance was recreated when the Master service failed over.

* Returns the expression result directly.

* Use Recovery to determine whether to use the old task instance.

(cherry picked from commit dbdbfeaeee)
 2022-05-16 20:11:23 +08:00
Tq
104f67d306 [Bug] [MASTER-9811]fix cmd param to overwrite global param when executing complement (#9952) 
* fix cmd param to overwrite global param when executing complement

* fix cmd param to overwrite global param when executing complement

(cherry picked from commit d4aeee16e5)
 2022-05-16 20:09:10 +08:00