caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
Wenjun Ruan
acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop ( #11254 )
...
* Fix TaskGroup cannot work and will cause master dead loop
* Remove acquireTaskGroupAgain in ProcessServiceImpl
2022-08-04 14:43:45 +08:00
Eric Gao
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server ( #11240 )
...
* [Improvement][Metrics] Add metrics for alert server (#11131 )
* Update related docs of metrics
* Add grafana demo dashboards for alert server metrics
* Refactor metric classes with UtilityClass annotation
* Refactor meter names in camelCase for checkstyle
2022-08-03 15:42:06 +08:00
Wenjun Ruan
04f3aa9713
Fix recovery from failed task will dead loop ( #11239 )
2022-08-02 09:30:18 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db ( #11221 )
2022-08-01 13:04:27 +08:00
caishunfeng
6c7e00c4aa
[Bug-11101] fix task failover NPE ( #11168 )
2022-07-27 19:21:21 +08:00
Eric Gao
0e15ce3389
[Improvement][Metrics] Switch to use tags to indicate task / workflow execution status for metrics ( #11128 )
...
* [Improvement][Metrics] Switch to use tags to indicate task / workflow execution status for metrics (#10867 )
* Update docs and grafana demo dashboards
2022-07-26 13:44:02 +08:00
zhuxt2015
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2022-07-22 13:34:31 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
Wenjun Ruan
e1c78d1a49
Move utils, builder under master ( #11000 )
2022-07-16 23:18:00 +08:00
Wenjun Ruan
083ab2b5c9
Remove dao in worker ( #10994 )
2022-07-15 20:07:18 +08:00
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
2022-07-15 20:06:53 +08:00
Wenjun Ruan
998e4d74dd
Kill task when we do master failover to avoid task lost ( #10997 )
2022-07-15 18:44:42 +08:00
Wenjun Ruan
b245e7c973
Fix kill yarn job error when failover caused by doesn't set ProcessDefinition ( #10948 )
2022-07-13 22:32:30 +08:00
Wenjun Ruan
df0416c193
Fix taskInstance's host is not worker nettyServer address ( #10926 )
...
* Fix taskInstance's host is not worker nettyServer address
* Remove unnecessary mock
2022-07-13 20:46:33 +08:00
WangJPLeo
9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. ( #10922 )
...
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
Wenjun Ruan
2a67866718
Catch exception when check state in StateWheelExecuteThread ( #10908 )
...
* Catch exception when check state
2022-07-13 10:51:20 +08:00
Wenjun Ruan
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master ( #10886 )
...
* Fix network error cause worker cannot send message to master
2022-07-12 14:08:42 +08:00
caishunfeng
aa8b88a875
[Feature-10871] add workflow executing data query ( #10875 )
...
* add workflow executing data query
* fix sonar check for interrupted
2022-07-11 16:55:55 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ( #10866 )
...
* Fix database update error doesn't rollback the task instance status
* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
WangJPLeo
b5653ea7f2
Complement numbers will run in a loop under the serial strategy fixed. ( #10862 )
...
* Complement numbers will run in a loop under the serial strategy fixed.
* e2e rerun
2022-07-09 16:41:51 +08:00
Wenjun Ruan
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect ( #10839 )
...
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
2022-07-09 11:54:59 +08:00
Wenjun Ruan
426567348e
Remove quartz in service ( #10748 )
...
* Remove quartz in service
2022-07-06 15:43:55 +08:00
Wenjun Ruan
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2022-07-06 14:53:28 +08:00
Wenjun Ruan
35a10d092f
[Fix-10666] Workflow submit failed will still in memory and never retry ( #10667 )
...
* Workflow submit failed will still in memory and never retry
2022-07-04 22:08:15 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2022-06-30 22:45:25 +08:00
Wenjun Ruan
247ca4ae8a
[Improvement-10617] Add comment in slot check ( #10618 )
2022-06-30 15:58:49 +08:00
Wenjun Ruan
35b25da863
Validate master/worker config ( #10649 )
2022-06-28 20:17:43 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2022-06-28 16:08:35 +08:00
WangJPLeo
b5184138fa
[Feature] Time function analysis extension. ( #10624 )
...
* Time function analysis extension.
* param add.
* clear useless logs and update method notes
* permission omission fix.
* extending time functions to optimize static methods
* e2e rerun.
2022-06-28 11:24:51 +08:00
pinkhello
628c8ab2a1
[Improvement][Master] batch remove TaskInstaceId and workflowInstanceId ( #10620 )
2022-06-26 10:07:35 +08:00
xiangzihao
397e51fa63
dependent node log output optimization and fix checkstyle ( #10593 )
2022-06-24 16:48:58 +08:00
xiangzihao
1111371c9a
add datasource health check to the healthcheck endpoint ( #10588 )
2022-06-24 13:29:49 +08:00
xiangzihao
0f38217b12
fix_10514 ( #10568 )
2022-06-23 16:15:08 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
WangJPLeo
f46faa02c3
[Fix-10452] Serial wait for policy recovery ( #10453 )
...
* Serial wait for policy recovery
* processInstance state check null
* add sendresult(Host host, Command command) method and replace the original.
2022-06-22 14:28:50 +08:00
Eric Gao
cc06eaaf54
[Improvement][Metrics] Apply micrometer naming convention to metrics ( #10477 )
...
* Apply micrometer naming convention to worker metrics
* Apply micrometer naming convention all current metrics
* Fix remaining metrics names, update English docs and add Chinese docs
* Fix metrics names in grafana-demo dashboards
2022-06-21 14:27:06 +08:00
Wenjun Ruan
b100f6c489
Remove the schedule thread in LowerWeightHostManager ( #10310 )
2022-06-21 13:30:07 +08:00
Wenjun Ruan
117f78ec4b
[Fix-10413] Fix Master startup failure the server still hang ( #10500 )
...
* Fix Master startup failure the server still hang
2022-06-20 22:35:06 +08:00
Wenjun Ruan
0bdfa0cff9
Fix PeerTaskInstancePriorityQueue cannot contains method use taskInstanceId to check ( #10491 )
2022-06-17 16:14:56 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
Wenjun Ruan
78c5fcc6ac
Add mysql registry plugin ( #10406 )
...
* Add mysql registry plugin
2022-06-13 11:24:42 +08:00
hstdream
f6fea06f10
[Improve] Enhance complement function transformation ( #10376 )
2022-06-12 12:35:08 +08:00
Wenjun Ruan
b0d9d3f9ab
Add some warning log in master ( #10383 )
...
* Add some warn log in master
* fix may skip sleep
2022-06-10 19:38:20 +08:00
Wenjun Ruan
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2022-06-09 10:55:39 +08:00
Wenjun Ruan
3ecbee3885
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
2022-06-09 10:00:05 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
2022-06-08 19:42:05 +08:00
Wenjun Ruan
2d3be6b36c
Add dolphinscheduler-scheduler module ( #10360 )
...
* Add dolphinscheduler-scheduler module
2022-06-04 16:39:33 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2022-05-31 15:22:41 +08:00
