ZhenjiLiu
7cdb926a5f
[Improvement][Batch Query] Batch query ProcessDefinitions belongs to need failover ProcessInstance. ( #12506 )
2022-11-03 09:15:19 +08:00
Wenjun Ruan
9e0c9af1a5
Fix the waiting strategy cannot recovery if the serverstate is already in running ( #12651 )
2022-11-02 14:06:01 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
Wenjun Ruan
e6da1ccf81
Add worker-group-refresh-interval in master config ( #12601 )
...
* Add worker-group-refresh-interval in master config
* Set interval cannot smaller than 10s
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-10-31 09:37:26 +08:00
kezhenxu94
065d5caccc
Only expose necessary actuator endpoints ( #12571 )
2022-10-28 07:40:32 +08:00
HanayoZz
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
Yann Ann
04aa125ba2
[Refactor]migrate some task dao functions from ProcessServiceImpl to dao modules. ( #12505 )
...
* migrate some task DAO to dao modules.
* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/repository/impl/ProcessInstanceMapDaoImpl.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 11:51:58 +08:00
liqiquan
4aa9d60683
[Bug][Master] Add the aws-java-sdk-s3 jar package to the master module ( #12259 ) ( #12512 )
...
Co-authored-by: qiquan.li <qiquan.li@linkflowtech.com>
2022-10-25 14:53:18 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
Yann Ann
b936b882bb
[Refactor] Migrate all command-related interface functions from ProcessServiceImpl ( #12474 )
...
* migrate all command-related interface functions to CommonService
2022-10-22 12:39:25 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
Yann Ann
38b643f69b
make sure all failed task will save in errorTaskMap ( #12424 )
2022-10-19 09:36:47 +08:00
kezhenxu94
4ea7eec750
Use temurin Java docker images instead of deprecated ones ( #12334 )
2022-10-17 15:46:21 +08:00
Yann Ann
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules ( #12362 )
...
* migrate junit4 -> junit5 in master, worker, and remote modules
2022-10-15 21:58:31 +08:00
Eric Gao
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
rickchengx
05099bd8f7
[Improvement-12296][Test] Close static mock ( #12300 )
2022-10-11 16:13:05 +08:00
sgw
165b9a58de
[DS-12131][master] Optimize the log printing of the master module acc… ( #12152 )
...
* [DS-12131][master] Optimize the log printing of the master module according to the log specification.
2022-10-09 13:56:50 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
xuhhui
55388be21f
[Improvement][Master] LowerWeightRoundRobin doSelect may be null ( #12159 )
2022-09-30 20:00:23 +08:00
caishunfeng
40090d459e
[Improvement] remove log-server and server module ( #12206 )
...
* remove log-server module
* remove dolphinscheduler-server module
2022-09-30 16:41:41 +08:00
Wenjun Ruan
aacabc6ec7
Remove workerGroup in registry ( #12217 )
2022-09-30 13:58:53 +08:00
fuchanghai
e27c79974d
[fix-#11753] send alert error alert data id ( #11774 )
...
* [fix-#11753] send alert error alert data id
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-09-29 15:34:40 +08:00
Wenjun Ruan
d526dd89f6
[hotfix][UI] Remove SURVEIL and NEXT_LOOP in ui ( #12160 )
...
* Remove SURVEIL and NEXT_LOOP in ui
* Fix UT
2022-09-28 09:17:12 +08:00
Eric Gao
cc82206c33
Remove powermock in dolphinscheduler-master module ( #12143 )
...
* Remove powermock related code in dolphinscheduler-master module
* Remove redundant comments, useless code and unnecessary stubbings
2022-09-26 17:02:14 +08:00
Tq
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts ( #12126 )
...
* fix htrace dependency conflicts
* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:08 +08:00
Yann Ann
ada7cf71d5
[fix#12000]Cannot remove the WorkerGroup from the master service ( #12050 )
...
* [Bug] [Master] Cannot remove the WorkerGroup from the master service. #12000
* remove unnecessary locks
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/ServerNodeManager.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-24 18:57:20 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
Yann Ann
e938fdbe96
[Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 ( #11864 )
...
* [Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838
2022-09-15 09:46:30 +08:00
caishunfeng
6868876a29
fix workflow keep running when task fail ( #11930 )
2022-09-15 09:06:10 +08:00
Wenjun Ruan
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2022-09-14 16:18:08 +08:00
Daniel Y
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 ( #11835 )
...
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9
* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-14 09:29:45 +08:00
LiuBodong
3ca9680b20
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly ( #11870 )
2022-09-11 19:11:03 +08:00
kezhenxu94
105773f54d
Remove htrace dependencies ( #11881 )
...
* Remove htrace dependencies
* Update pom.xml
2022-09-10 22:54:38 +08:00
Kengo Seki
3664d85143
[Improve] Upgrade Hadoop to 3.2.4 ( #11832 )
2022-09-09 10:47:03 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2022-09-08 09:05:05 +08:00
Kengo Seki
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
caishunfeng
f8d46a26c1
[Feature-11530] add state history for process instance ( #11757 )
...
* [Feature] add state history for process instance (#97 )
* add state history for process instance
* upsertProcessInstance
* remove unuse method
* fix UT
Co-authored-by: caishunfeng <534328519@qq.com>
2022-09-03 10:14:05 +08:00
kezhenxu94
277f137358
Add Kubernetes configmap reload to all components ( #11730 )
2022-09-02 12:03:19 +08:00
caishunfeng
f95cdd350a
reduce task dependencies ( #11731 )
2022-09-01 17:04:07 +08:00
Wenjun Ruan
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2022-08-31 16:20:23 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2022-08-24 19:51:47 +08:00
JinYong Li
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug ( #11232 )
...
* fix heartBeat bug
* modify class name
* fix conflict
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2022-08-23 11:30:13 +08:00
insist777
abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters ( #11542 )
...
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [improvement] The default worker grouping supports editing
* [improvement] The default worker grouping supports editing
* Update WorkerGroupServiceImpl.java
* Update WorkerGroupServiceImpl.java
* The default worker grouping supports editing
* update sql file
* update
* update
* update
* update
2022-08-23 10:54:17 +08:00
Wenjun Ruan
4d13a5104b
Fix kill task failed will cause the taskGroup cannot release and add taskGroup log ( #11469 )
2022-08-23 10:05:03 +08:00
Wenjun Ruan
a41c6824fc
Add dependent task instance log ( #11541 )
...
* Add dependent task instance log
* Optimize log
* Fix dependent task initialize failed will throw exception
2022-08-23 09:55:38 +08:00
Wenjun Ruan
8886d5a126
Fix quartz threadPriority config name error ( #11596 )
...
* Fix quartz threadPriority config name error
* Add batchTriggerAcquisitionMaxCount config
2022-08-23 09:32:05 +08:00
Wenjun Ruan
3516533017
Remove logger header in task log file ( #11555 )
2022-08-19 14:01:52 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2022-08-16 20:25:17 +08:00
Wenjun Ruan
4362fb8448
If the task processor is not found need to throw error rather than exception ( #11461 )
2022-08-13 22:01:39 +08:00
Wenjun Ruan
05589606a2
Set master's task running status in runTask to avoid the task group acquire failed, but the task status is in running ( #11451 )
2022-08-13 14:19:13 +08:00
Wenjun Ruan
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2022-08-13 09:52:03 +08:00
Wenjun Ruan
0ca3086296
Fix workflow instance may failover many times due to doesn't check the restart time ( #11445 )
2022-08-13 00:05:37 +08:00
Wenjun Ruan
a2b5219315
Fix find last process instance in dependent task is not correct ( #11424 )
2022-08-12 14:11:28 +08:00
Wenjun Ruan
47b50067ef
Clear thread name in BaseTaskProcessor ( #11422 )
2022-08-12 09:38:48 +08:00
Wenjun Ruan
496c2d4bfa
[Fix-11413] Cannot set task status to kill if the task is not in running( #11414 )
2022-08-11 13:38:49 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2022-08-11 09:25:53 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
Wenjun Ruan
acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop ( #11254 )
...
* Fix TaskGroup cannot work and will cause master dead loop
* Remove acquireTaskGroupAgain in ProcessServiceImpl
2022-08-04 14:43:45 +08:00
Eric Gao
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server ( #11240 )
...
* [Improvement][Metrics] Add metrics for alert server (#11131 )
* Update related docs of metrics
* Add grafana demo dashboards for alert server metrics
* Refactor metric classes with UtilityClass annotation
* Refactor meter names in camelCase for checkstyle
2022-08-03 15:42:06 +08:00
Wenjun Ruan
04f3aa9713
Fix recovery from failed task will dead loop ( #11239 )
2022-08-02 09:30:18 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db ( #11221 )
2022-08-01 13:04:27 +08:00
caishunfeng
6c7e00c4aa
[Bug-11101] fix task failover NPE ( #11168 )
2022-07-27 19:21:21 +08:00
Eric Gao
0e15ce3389
[Improvement][Metrics] Switch to use tags to indicate task / workflow execution status for metrics ( #11128 )
...
* [Improvement][Metrics] Switch to use tags to indicate task / workflow execution status for metrics (#10867 )
* Update docs and grafana demo dashboards
2022-07-26 13:44:02 +08:00
zhuxt2015
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2022-07-22 13:34:31 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
Wenjun Ruan
e1c78d1a49
Move utils, builder under master ( #11000 )
2022-07-16 23:18:00 +08:00
Wenjun Ruan
083ab2b5c9
Remove dao in worker ( #10994 )
2022-07-15 20:07:18 +08:00
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
2022-07-15 20:06:53 +08:00
Wenjun Ruan
998e4d74dd
Kill task when we do master failover to avoid task lost ( #10997 )
2022-07-15 18:44:42 +08:00
Wenjun Ruan
b245e7c973
Fix kill yarn job error when failover caused by doesn't set ProcessDefinition ( #10948 )
2022-07-13 22:32:30 +08:00
Wenjun Ruan
df0416c193
Fix taskInstance's host is not worker nettyServer address ( #10926 )
...
* Fix taskInstance's host is not worker nettyServer address
* Remove unnecessary mock
2022-07-13 20:46:33 +08:00
WangJPLeo
9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. ( #10922 )
...
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
Wenjun Ruan
2a67866718
Catch exception when check state in StateWheelExecuteThread ( #10908 )
...
* Catch exception when check state
2022-07-13 10:51:20 +08:00
Wenjun Ruan
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master ( #10886 )
...
* Fix network error cause worker cannot send message to master
2022-07-12 14:08:42 +08:00
caishunfeng
aa8b88a875
[Feature-10871] add workflow executing data query ( #10875 )
...
* add workflow executing data query
* fix sonar check for interrupted
2022-07-11 16:55:55 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ( #10866 )
...
* Fix database update error doesn't rollback the task instance status
* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
WangJPLeo
b5653ea7f2
Complement numbers will run in a loop under the serial strategy fixed. ( #10862 )
...
* Complement numbers will run in a loop under the serial strategy fixed.
* e2e rerun
2022-07-09 16:41:51 +08:00
Wenjun Ruan
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect ( #10839 )
...
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
2022-07-09 11:54:59 +08:00
Wenjun Ruan
426567348e
Remove quartz in service ( #10748 )
...
* Remove quartz in service
2022-07-06 15:43:55 +08:00
Wenjun Ruan
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2022-07-06 14:53:28 +08:00
Wenjun Ruan
35a10d092f
[Fix-10666] Workflow submit failed will still in memory and never retry ( #10667 )
...
* Workflow submit failed will still in memory and never retry
2022-07-04 22:08:15 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2022-06-30 22:45:25 +08:00
Wenjun Ruan
247ca4ae8a
[Improvement-10617] Add comment in slot check ( #10618 )
2022-06-30 15:58:49 +08:00
Wenjun Ruan
35b25da863
Validate master/worker config ( #10649 )
2022-06-28 20:17:43 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2022-06-28 16:08:35 +08:00
WangJPLeo
b5184138fa
[Feature] Time function analysis extension. ( #10624 )
...
* Time function analysis extension.
* param add.
* clear useless logs and update method notes
* permission omission fix.
* extending time functions to optimize static methods
* e2e rerun.
2022-06-28 11:24:51 +08:00
pinkhello
628c8ab2a1
[Improvement][Master] batch remove TaskInstaceId and workflowInstanceId ( #10620 )
2022-06-26 10:07:35 +08:00
xiangzihao
397e51fa63
dependent node log output optimization and fix checkstyle ( #10593 )
2022-06-24 16:48:58 +08:00
xiangzihao
1111371c9a
add datasource health check to the healthcheck endpoint ( #10588 )
2022-06-24 13:29:49 +08:00
xiangzihao
0f38217b12
fix_10514 ( #10568 )
2022-06-23 16:15:08 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
WangJPLeo
f46faa02c3
[Fix-10452] Serial wait for policy recovery ( #10453 )
...
* Serial wait for policy recovery
* processInstance state check null
* add sendresult(Host host, Command command) method and replace the original.
2022-06-22 14:28:50 +08:00
Eric Gao
cc06eaaf54
[Improvement][Metrics] Apply micrometer naming convention to metrics ( #10477 )
...
* Apply micrometer naming convention to worker metrics
* Apply micrometer naming convention all current metrics
* Fix remaining metrics names, update English docs and add Chinese docs
* Fix metrics names in grafana-demo dashboards
2022-06-21 14:27:06 +08:00
Wenjun Ruan
b100f6c489
Remove the schedule thread in LowerWeightHostManager ( #10310 )
2022-06-21 13:30:07 +08:00
Wenjun Ruan
117f78ec4b
[Fix-10413] Fix Master startup failure the server still hang ( #10500 )
...
* Fix Master startup failure the server still hang
2022-06-20 22:35:06 +08:00
Wenjun Ruan
0bdfa0cff9
Fix PeerTaskInstancePriorityQueue cannot contains method use taskInstanceId to check ( #10491 )
2022-06-17 16:14:56 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
Wenjun Ruan
78c5fcc6ac
Add mysql registry plugin ( #10406 )
...
* Add mysql registry plugin
2022-06-13 11:24:42 +08:00
hstdream
f6fea06f10
[Improve] Enhance complement function transformation ( #10376 )
2022-06-12 12:35:08 +08:00
Wenjun Ruan
b0d9d3f9ab
Add some warning log in master ( #10383 )
...
* Add some warn log in master
* fix may skip sleep
2022-06-10 19:38:20 +08:00
Wenjun Ruan
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2022-06-09 10:55:39 +08:00
Wenjun Ruan
3ecbee3885
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
2022-06-09 10:00:05 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
2022-06-08 19:42:05 +08:00
Wenjun Ruan
2d3be6b36c
Add dolphinscheduler-scheduler module ( #10360 )
...
* Add dolphinscheduler-scheduler module
2022-06-04 16:39:33 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2022-05-31 15:22:41 +08:00
JinYong Li
49979c658e
[Fix-8828] [Master] Assign tasks to worker optimization ( #9919 )
...
* fix 9584
* master recall
* fix ut
* update logger
* update delay queue
* fix ut
* remove sleep
Co-authored-by: 进勇 <lijinyong@cai-inc.com>
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2022-05-31 11:49:54 +08:00
旺阳
aba5f8a40e
[improve] Change Mysql Driver ( #10220 )
2022-05-25 14:09:15 +08:00
BaoLiang
ee2b855ced
[BUG][TaskGroup] Task group does not take effect ( #10093 )
...
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
2022-05-18 18:40:36 +08:00
He Zhao
5bb1eb04fc
[Feature][plugin] Add k8s task in task plugin ( #9425 )
...
* [Feature][plugin] Add k8s task in task plugin
* [Feature][plugin] fix dos and code problems
* [Feature][plugin] refactor some code based on sonar
* [Feature][UI] front-end for k8s task plugin
* [Feature][plugin] delete some front files
* [Feature][plugin] update document
* Update docs/docs/zh/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* [Feature][UI] front-end change from review comments
* [Feature][UI] replace get namespace list api
* [Feature][plugin] change file name
* Add kubernetes to zh task list
Co-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-18 17:23:42 +08:00
xiangzihao
225cb332d1
[Fix-10049] Conditions Task branch flow failed ( #10077 )
2022-05-18 11:40:32 +08:00
caishunfeng
0cc0ee77fa
[Bug][Master] fix master task failover ( #10065 )
...
* fix master task failover
* ui
2022-05-17 16:36:24 +08:00
WangJPLeo
dbdbfeaeee
[Fix-9975] The selected task instance was recreated when the Master service fail… ( #9976 )
...
* The selected task instance was recreated when the Master service failed over.
* Returns the expression result directly.
* Use Recovery to determine whether to use the old task instance.
2022-05-12 16:31:53 +08:00
Tq
d4aeee16e5
[Bug] [MASTER-9811]fix cmd param to overwrite global param when executing complement ( #9952 )
...
* fix cmd param to overwrite global param when executing complement
* fix cmd param to overwrite global param when executing complement
2022-05-11 18:37:03 +08:00
Paul Zhang
8562f6a878
[Feature][Log]Add timezone information in log output ( #9913 )
2022-05-06 17:31:44 +08:00
WangJPLeo
a1b6b033ad
[Fix-9906] After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance. ( #9907 )
...
* After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance.
* clear logic
* Resource overloading
2022-05-06 15:01:21 +08:00
WangJPLeo
fb0f96ed94
[Fix-9868] A task flow definition isolates the runs of different execution strategies by version numbers. ( #9869 )
...
* The thread cache task flow definition should get the latest version.
* Coverage on New Code
* Coverage on New Code
* Coverage on New Code
* use an existing method.
* Increase unit test coverage.
* Task flow definitions enforce policy isolation.
2022-05-03 22:52:42 +08:00
WangJPLeo
31cd1b5e61
Serial wait for subsequent execution ( #9847 )
2022-04-29 15:44:05 +08:00
WangJPLeo
3cea039239
Task queue status update. ( #9832 )
2022-04-28 16:41:00 +08:00
WangJPLeo
5c0be8a3d7
A task instance that normally queries the serial wait state. ( #9777 )
...
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-26 15:47:01 +08:00
WangJPLeo
897d7cb555
Add the host address of the execution server to the sub task task instance. ( #9758 )
...
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 18:17:48 +08:00
Jiajie Zhong
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server ( #9726 )
...
* [common] Make dolphinscheduler_env.sh work
* Change dist tarball `dolphinscheduler_env.sh` location
from `bin/` to `conf/`, which users could finish their
change configuration operation in one single directory.
and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
when we start our sever instead of adding both
`$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
WangJPLeo
7bcec7115a
[Fix-9717] The failure policy of the task flow takes effect ( #9718 )
...
* Failure policy takes effect.
* Coverage on New Code
* correct description logic
* Compatible with all scenarios
* clearer logic
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 15:29:18 +08:00
caishunfeng
5657cb9aec
[Bug-9719][Master] fix failover fail because task plugins has not been loaded ( #9720 )
2022-04-24 20:34:21 +08:00
gaojun2048
ebc4253d50
[fix][Service] BusinessTime should format with schedule timezone ( #9714 )
...
* BusinessTime should format with schedule timezone
* fix test error
* fix test error
* fix test error
2022-04-24 19:21:21 +08:00
caishunfeng
88d2803fe1
fix task dispatch error overload resource pool of task group ( #9667 )
2022-04-22 18:39:40 +08:00
caishunfeng
63638601b0
fix process pause and rerun ( #9568 )
2022-04-19 10:23:56 +08:00
sparklezzz
508ed9769a
[Fix][Master Server] handle warn+failed timeout strategy in workflow execute thread of master server ( #8077 ) ( #9485 )
...
Co-authored-by: xudong.zhang <xudong.zhang@nio.com>
2022-04-18 20:34:22 +08:00
Paul Zhang
3815a86a3b
[Improvement][Master] Fix typo for MasterTaskExecThreadTest ( #9513 )
2022-04-15 11:32:22 +08:00
caishunfeng
66d148872d
[Bug-9501][Worker] fix kill task error before running ( #9509 )
2022-04-14 21:34:02 +08:00
czeming
706cdb6a8e
[Fix-9174] [Alert] Fix deduplication of alarm information ( #9371 )
...
* feat(issue #9174 ):
Fix-9174
2022-04-14 10:30:25 +08:00
worry
69923546a1
[DS-9263][Improvement][master]optimize failover ( #9281 )
...
- add FailoverService.java
- move failover method from MasterRegistryClient to FailoverService
- move failover code from FailoverExecuteThread to FailoverService
This closes #9263
2022-04-12 11:53:18 +08:00
caishunfeng
b285ccf930
[Future-9396]Support output parameters transfer from parent workflow to child work flow ( #9410 )
...
* [Future-9396]Support output parameters transfer from parent workflow to child work flow
* fix note
2022-04-11 20:03:16 +08:00
worry
dce3c132ca
[DS-9387][refactor]Remove the lock in the start method of the MasterRegistryClient class ( #9389 )
2022-04-08 21:06:28 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
2022-04-07 18:15:19 +08:00
caishunfeng
40e44f6bc4
fix repeated submit task ( #9323 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-04-02 14:06:35 +08:00
guoshupei
3e5781b6e1
[Fix-9065] [master] when task submit failed, remove from queue and add state event to thread ( #9186 )
...
* [Fix-9065] [master] when task submit failed, remove from queue and add state event to thread
This closes #9065
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
yes, your idea is better
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [Bug-9065] [master] update logger info
Co-authored-by: guoshupei <guoshupei@lixiang.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-04-01 22:23:35 +08:00
caishunfeng
1073fcae44
[Bug-9295][Master] fix repeated submit task ( #9304 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-31 21:10:09 +08:00
caishunfeng
52ba2c6475
[Improvement-9288][Master] add task event thread pool ( #9293 )
...
* add task event thread
* license heander
* ci test
* delete unuse file
* fix CI test
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-31 14:13:33 +08:00
caishunfeng
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback ( #8719 )
...
* rebase dev
* change task state to dispatch if dispatch success
* update task host when dispatch
* add dispatch task event
* test
* check tenant after enable auto create
* handle dispatch state
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 17:35:05 +08:00
worry
b4b5241797
[DS-9266][feat][server]Implement show dolphinscheduler banner at launch ( #9273 )
...
- add banner.txt
- modify application.yaml
This closes #9266
2022-03-30 16:45:24 +08:00
wangxj3
fd76436d87
[Feature-#8318] [master-server] Sub process transfer parameters to father process ( #8522 )
...
* load param from subprocess
* unit test
* modify code style
Co-authored-by: wangxj <wangxj31>
2022-03-30 14:55:59 +08:00
caishunfeng
df519f8044
remove common task log ( #9274 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 11:42:47 +08:00
guoshupei
e2c1cc0579
[Fix-9222] [master] Support for deploying multiple masters on one node ( #9240 )
...
This closes #9222
Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-29 22:46:02 +08:00
JinYong Li
85e56b5e2a
fix 9243 ( #9244 )
2022-03-28 20:45:01 +08:00
worry
d3251c9bcc
[Improvement-9227][master]implement use the slot to scan the database ( #9228 )
...
when the master assigns tasks by slot,implement use the slot to scan the database.
This closes #9227
2022-03-28 09:52:12 +08:00
caishunfeng
263791a63e
[Bug-9137][Server] fix timezone init ( #9219 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
guoshupei
258285e6bb
[Fix-8980] [master] fixed repeat processing command ( #9220 )
...
This closes #8980
Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-27 23:33:30 +08:00
