test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-12 12:25:16 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,269 Commits 27 Branches 59 Tags 275 MiB
fe3e93861f
Commit Graph

102 Commits

Author SHA1 Message Date
Wenjun Ruan
fe3e93861f
Add switch for checkOwn (#197) (#12803) 2022-11-08 10:44:17 +08:00
zhuangchong
145c5ac8cb [maven-release-plugin] prepare for next development iteration 2022-11-07 20:43:01 +08:00
zhuangchong
525a6846ac [maven-release-plugin] prepare release 3.1.1 2022-11-07 20:43:00 +08:00
Kerwin
2c93dc6041
cherry-pick (#12608) 
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE
#12449

Co-authored-by: fuchanghai <33984497+fuchanghai@users.noreply.github.com>
 2022-10-29 21:48:37 +08:00
Kerwin
94ef02e9de
cherry-pick [Improvement] Merge spi.utils into common.utils (#12607) 
#12502

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
 2022-10-29 21:30:28 +08:00
Wenjun Ruan
75f6c416fb Add task executor threads full policy config in worker (#12510) 2022-10-27 14:52:15 +08:00
Wenjun Ruan
a5a849f615 cherry-pick [Bug] Set tenantDir permission 
#12486
 2022-10-27 14:02:57 +08:00
kezhenxu94
eb7b48596c cherry-pick Reorganize some classes in common module, remove duplicate classes 
#12321
 2022-10-27 13:53:41 +08:00
Wenjun Ruan
a17d0cc5d2 cherry-pick [Improvement][WorkerGroup]Remove workerGroup in registry #12217 2022-10-27 13:21:24 +08:00
Kerwin
58ef3cccd7
[3.1.1-prepare]cherry-pick [Improvement] remove log-server and server module #12206 (#12562) 2022-10-27 12:54:38 +08:00
Kerwin
f3277277f0
cherry-pick Refactor LogServiceClient Singleton to avoid repeat creation of NettyClient #11777 (#12542) 
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
 2022-10-26 16:20:13 +08:00
Kerwin
344293102d
[3.1.1-prepare][cherry-pick][Improvement] Add remote task model #11767 (#12541) 2022-10-26 15:44:13 +08:00
Kerwin
2402b8a6ef
Use temurin Java docker images instead of deprecated ones (#12334) (#12421) 
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
 2022-10-18 18:51:14 +08:00
Kerwin
d2e56af838
[Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql (#11721) (#12420) 
select different versions of spark-sql to execute different versions of spark-sql

Co-authored-by: limaiwang <limaiwang@163.com>
 2022-10-18 18:45:54 +08:00
sgw
703f9991b4 [DS-12154][worker] Optimize the log printing of the worker module (#12183) 
* [DS-12154][worker] Optimize the log printing of the worker module according to the log specification.
 2022-10-18 09:28:07 +08:00
caishunfeng
7e39396a76 [maven-release-plugin] prepare for next development iteration 2022-09-26 21:50:03 +08:00
caishunfeng
ae33ba5947 [maven-release-plugin] prepare release 3.1.0 2022-09-26 21:49:27 +08:00
Daniel Y
2a437607ae [Improvement-11834] Upgrade docker base image to support python3.9 (#11835) 
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9

* use openjdk:8-jre-slim-bullseye instead of buster across all components
 2022-09-19 15:50:25 +08:00
LiuBodong
0f3b42925f [Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly (#11870) 2022-09-19 15:30:49 +08:00
Kengo Seki
52b79b017e [Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810) 
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
 2022-09-19 14:48:17 +08:00
caishunfeng
f034a09d25
[Bug-11650][worker] #11650 fix SQL type task, stop task cause NPE (#11668) (#11958) 
Co-authored-by: 冯剑 <35831367+fengjian1129@users.noreply.github.com>
 2022-09-15 14:24:34 +08:00
kezhenxu94
277f137358
Add Kubernetes configmap reload to all components (#11730) 2022-09-02 12:03:19 +08:00
Wenjun Ruan
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize (#11702) 
* Refactor heart beat task, use json to serialize/deserialize
 2022-08-31 16:20:23 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701) 
* Fix kill job may cause worker oom
 2022-08-31 15:25:01 +08:00
Wenjun Ruan
1b120e3a59
Refactor worker execute task process (#11540) 
* Refactor worker execute task process
 2022-08-26 13:33:51 +08:00
JinYong Li
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug (#11232) 
* fix heartBeat bug

* modify class name

* fix conflict

Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
 2022-08-23 11:30:13 +08:00
Wenjun Ruan
3516533017
Remove logger header in task log file (#11555) 2022-08-19 14:01:52 +08:00
Eric Gao
9330d6cfcd
[Doc][Security] Update instructions on worker groups (#11483) 
* Update instructions on worker groups
 2022-08-15 17:44:00 +08:00
Wenjun Ruan
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost (#11368) 
* Add waiting strategy to support master/worker can recover from registry lost

* throw exception when zookeeper registry start failed due to interrupted
 2022-08-13 09:52:03 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task (#11350) 
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
 2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
Eric Gao
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server (#11240) 
* [Improvement][Metrics] Add metrics for alert server (#11131)

* Update related docs of metrics

* Add grafana demo dashboards for alert server metrics

* Refactor metric classes with UtilityClass annotation

* Refactor meter names in camelCase for checkstyle
 2022-08-03 15:42:06 +08:00
xuhhui
bfff3a7c5d
fix error (#11206) 2022-07-30 18:20:20 +08:00
zhuxt2015
3701a24d15
[Improvement][Task Log] Task status log print description instead of code (#11009) 
* use execution status instead of status code
 2022-07-22 13:34:31 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
zhuxt2015
a74d7ef665
[hotfix][Worker] Remove service dependency from worker module (#11008) 
* worker remove service dependency
 2022-07-17 22:16:35 +08:00
Wenjun Ruan
083ab2b5c9
Remove dao in worker (#10994) 2022-07-15 20:07:18 +08:00
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979) 
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed

* Move heartbeat error threshold to applicaiton.yml
 2022-07-15 20:06:53 +08:00
Wenjun Ruan
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master (#10886) 
* Fix network error cause worker cannot send message to master
 2022-07-12 14:08:42 +08:00
Eric Gao
2f7281c2d2
[Feature][Metrics] Add resource download related metrics for workers (#10749) 
* [Feature][Metrics] Add resource download related metrics for workers (#9324)

* [Feature][Metrics] Fix bugs and add grafana demos for worker resource download metrics (#9324)

* [Feature][Metrics] Add docs to resource related metrics (#9324)

* [Feature][Metrics] Use tags to indicate status in metrics (#9324)

* [Feature][Metrics] Fix demos, docs and remove redundant code (#9324)

* [Feature][Metrics] Remove .pnpm-debug.log (#9324)

* [Feature][Metrics] Fix style check (#9324)

* [Feature][Metrics] Replace KB with bytes for the unit of resource file size in metrics (#9324)

* [Feature][Metrics] Make code neat (#9324)
 2022-07-12 11:44:34 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866) 
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
 2022-07-11 09:57:00 +08:00
Wenjun Ruan
426567348e
Remove quartz in service (#10748) 
* Remove quartz in service
 2022-07-06 15:43:55 +08:00
Wenjun Ruan
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry (#10786) 
* Fix state event handle error will not retry

* Use state event handler to deal with the event
 2022-07-06 14:53:28 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. (#10704) 
* Global parameter and local parameter calculation external expansion.

* k8s task ut fix.

* TimePlaceholderUtils import DateUtils fix

* follow the review comments to fix.

* follow the review comments to fix.

* e2e rerun
 2022-06-30 22:45:25 +08:00
Wenjun Ruan
35b25da863
Validate master/worker config (#10649) 2022-06-28 20:17:43 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover (#10631) 
* fix worker failover may lose event
 2022-06-28 16:08:35 +08:00
pinkhello
719a9d4532
[Improvement][Worker] fixed naming of rpc package (#10614) 2022-06-26 10:30:09 +08:00
xiangzihao
1111371c9a
add datasource health check to the healthcheck endpoint (#10588) 2022-06-24 13:29:49 +08:00
xiangzihao
0f38217b12
fix_10514 (#10568) 2022-06-23 16:15:08 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516) 
* Optimize master log, add workflow instance id and task instance id in log

* Use MDC to set the workflow info in log4j

* Add workflowInstanceId and taskInstanceId in MDC
 2022-06-23 11:45:06 +08:00