test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-04 21:28:00 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
7,744 Commits 27 Branches 59 Tags 276 MiB
16b743c777
Commit Graph

256 Commits

Author SHA1 Message Date
Wenjun Ruan
9b269b3c53
Use common response for sync request (#13847) 2023-04-01 10:45:21 +08:00
Rick Cheng
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target (#13654) 2023-03-30 12:07:58 +08:00
Wenjun Ruan
68660ec96b
Refactor remote command (#13809) 
* Refactor remote command

* Rename Command to Message
 2023-03-29 17:54:57 +08:00
Wenjun Ruan
d1b6e6f02c
Take over task instance in master failover (#13798) 2023-03-28 16:02:57 +08:00
Eric Gao
394805b2c7
[Feature][Metrics] Tag workflow related metrics with process definition code (workflow id) (#13640) 
* Tag workflow related metrics with process definition code (workflow id)

* Clean up related metrics when deleting workflow definition

* Add license headers

* Update related UT cases

* Add an example in grafana-demo

* Add related docs
 2023-03-09 11:30:21 +08:00
Wenjun Ruan
c9066e8de9
Use MDC to filter task instance log (#13673) 
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
 2023-03-06 17:44:23 +08:00
fuchanghai
7bf3e3cdd6
[improve-#13201] update pid during running (#13206) 2023-02-22 14:22:49 +08:00
Rick Cheng
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS (#13332) 2023-02-13 16:38:26 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger (#13509) 2023-02-07 20:32:53 +08:00
Wenjun Ruan
de70421a1e
Remove the log path check (#13280) 2022-12-26 22:39:08 +08:00
Wenjun Ruan
14ec4a2398
Remove dao module in worker (#13242) 2022-12-22 12:25:29 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds (#12197) 
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.

Co-authored-by: gabrywu <gabrywu@apache.com>
 2022-10-31 16:52:53 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils (#12502) 
* improve JSONUtil

* merge spi.utils into common.utils

* split constants && remove some copy utils method

* remove StringUtils
 2022-10-23 21:23:54 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412) 
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
 2022-10-19 11:08:04 +08:00
Yann Ann
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules (#12362) 
* migrate junit4 -> junit5 in master, worker, and remote modules
 2022-10-15 21:58:31 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes (#12321) 2022-10-13 10:23:27 +08:00
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project (#12244) 
* Fully remove the usage of powermock from the whole project

* Upgrade org.reflections to 0.10.12
 2022-10-10 10:43:38 +08:00
sgw
165b9a58de
[DS-12131][master] Optimize the log printing of the master module acc… (#12152) 
* [DS-12131][master] Optimize the log printing of the master module according to the log specification.
 2022-10-09 13:56:50 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones (#12239) 
* fix DS support cross time zone, use timestamp replace date
 2022-10-01 12:50:42 +08:00
Wenjun Ruan
d3a77c68e6
Make LogServiceClient Singleton (#11777) 2022-09-14 16:18:08 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701) 
* Fix kill job may cause worker oom
 2022-08-31 15:25:01 +08:00
Wenjun Ruan
5fa3e7b1ed
Add pause to task instance status (#11390) 2022-08-11 09:25:53 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task (#11350) 
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
 2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
zhuxt2015
3701a24d15
[Improvement][Task Log] Task status log print description instead of code (#11009) 
* use execution status instead of status code
 2022-07-22 13:34:31 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
Wenjun Ruan
998e4d74dd
Kill task when we do master failover to avoid task lost (#10997) 2022-07-15 18:44:42 +08:00
Wenjun Ruan
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master (#10886) 
* Fix network error cause worker cannot send message to master
 2022-07-12 14:08:42 +08:00
caishunfeng
aa8b88a875
[Feature-10871] add workflow executing data query (#10875) 
* add workflow executing data query
* fix sonar check for interrupted
 2022-07-11 16:55:55 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866) 
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
 2022-07-11 09:57:00 +08:00
Wenjun Ruan
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry (#10786) 
* Fix state event handle error will not retry

* Use state event handler to deal with the event
 2022-07-06 14:53:28 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover (#10631) 
* fix worker failover may lose event
 2022-06-28 16:08:35 +08:00
WangJPLeo
f46faa02c3
[Fix-10452] Serial wait for policy recovery (#10453) 
* Serial wait for policy recovery

* processInstance state check null

* add sendresult(Host host, Command command) method and replace the original.
 2022-06-22 14:28:50 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479) 
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
 2022-06-16 21:46:18 +08:00
Wenjun Ruan
b0d9d3f9ab
Add some warning log in master (#10383) 
* Add some warn log in master

* fix may skip sleep
 2022-06-10 19:38:20 +08:00
Wenjun Ruan
e21d7b1551
[Feature][metrics] Add master, worker metrics (#10326) 
* Add mater metrics

* fix UT

* Add url to mysql profile

* Add worker metrics

* Update grafana config

* Add system metrics doc

* Add process failover counter

* Add metrics image

* Change jpg to png

* Add command insert metrics

* Fix UT

* Revert UT
 2022-06-09 10:55:39 +08:00
Wenjun Ruan
516757cc74
[Fix] [Worker] Fix worker will hang if fails to start (#10342) 
* Fix worker will hang if fails to start

* Add .run to ignore
 2022-06-06 10:49:23 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
JinYong Li
49979c658e
[Fix-8828] [Master] Assign tasks to worker optimization (#9919) 
* fix 9584

* master recall

* fix ut

* update logger

* update delay queue

* fix ut

* remove sleep

Co-authored-by: 进勇 <lijinyong@cai-inc.com>
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
 2022-05-31 11:49:54 +08:00
guoshupei
ca95d2f928
[Fix-9221] [alert-server] optimization and gracefully close (#9246) 
* [Fix-9221] [alert-server] optimization and gracefully close

This closes #9221

* [Fix-9221] [alert-server] remove unused mock data

This closes #9221

* [Fix-9221] [alert-server] remove unused mock data

This closes #9221

* [Fix-9221] [alert-server] remove unnecessary Mockito stubbings

* [Fix-9221] [alert-server] init AlertPluginManager in AlertServer

* [Fix-9221] [alert-server] AlertServerTest add AlertPluginManager installPlugin

* [Fix-9221] [alert-server] replace @Eventlistener with @PostConstruct

* [Fix-9221] [alert-server] sonar check solution

* [Improvement-9221] [alert] update constructor injection and replace IStoppable with Closeable

Co-authored-by: guoshupei <guoshupei@lixiang.com>
 2022-04-08 10:02:10 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
caishunfeng
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719) 
* rebase dev

* change task state to dispatch if dispatch success

* update task host when dispatch

* add dispatch task event

* test

* check tenant after enable auto create

* handle dispatch state

Co-authored-by: caishunfeng <534328519@qq.com>
 2022-03-30 17:35:05 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584) 
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
 2022-03-15 15:16:36 +08:00
wangyang
0d8079a816
[DS-7992][feat] Alert module judging strategy (#8636) 2022-03-14 14:25:27 +08:00
yimaixinchen
a72742b9de
remove unuse import jsonFormat (#8548) 2022-02-25 22:05:05 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477) 
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
 2022-02-24 18:20:34 +08:00
Wenjun Ruan
66dcf2376a
Optimize some code (#8324) 2022-02-10 19:07:53 +08:00
Wenjun Ruan
093815785b
Remove unused class (#8299) 2022-02-08 19:50:19 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
xuhhui
0cfb2043cf
Improvement: improve Pong.java mistake (#7822) 2022-01-12 01:20:13 +08:00