Eric Gao
e86630bb7b
[Improvement][Metrics] Add more worker related metrics and fix some previous ones ( #14254 )
...
* Add more worker related metrics and fix some previous ones
* update metrics docs
2023-06-01 16:54:48 +08:00
Rick Cheng
428f1559a3
[Improv-14083][Resource Center] Handle deleteSource in OSS / S3 / GCS ( #14084 )
2023-05-22 20:44:43 +08:00
Wenjun Ruan
05df0269f6
Fix task exception might not exist in task instance log ( #14085 )
2023-05-21 09:55:35 +08:00
Aaron Wang
74ed7856bc
[Bug-14105][Task] Replace default tenant when downloading resource ( #14106 )
2023-05-16 18:11:58 +08:00
Aaron Wang
1e3ae7b942
[improv][Resource Center] Implement getResourceFileName in StorageOperator ( #14097 )
2023-05-15 14:02:30 +08:00
Rick Cheng
3eb4c0ed1d
[Improvement-14054][Worker] Kill multiple yarn apps at the same time ( #14055 )
...
* [Improvement-14054][Worker] Kill multiple yarn apps at the same time
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/YarnApplicationManager.java
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-05-06 19:13:12 +08:00
Wenjun Ruan
cf91544b99
Refactor logic task ( #13948 )
2023-04-26 10:01:51 +08:00
klaokai
f6e0f2e471
[Bug] [worker] WorkerConfig output is wrong ( #14000 ) ( #14001 )
2023-04-24 19:04:34 +08:00
Aaron Wang
731123cebc
[Bug][Worker] get worker status bug ( #13992 )
...
* get worker status bug
* modify configuration name
2023-04-24 09:51:47 +08:00
Aaron Wang
45586f71d5
[BUG][Resource Center] Task can not select main JAR ( #13536 )
2023-04-18 18:38:16 +08:00
Wenjun Ruan
61a689aa5a
Use percentage to represent memory/cpu usage ( #13896 )
2023-04-13 15:48:20 +08:00
Rick Cheng
7ee66f2d02
[Feature-13429][Remote Logging] Add support for writing task logs to Google Cloud Storage ( #13777 )
2023-04-06 17:49:33 +08:00
Wenjun Ruan
41a8ba9aab
Support alert server HA ( #13865 )
...
* Support alert server cluster
* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2023-04-05 17:31:50 +08:00
Aaron Wang
d6fcf60534
[Improvement-13751][Worker] Support real-time pod log collection ( #13752 )
...
* Support real-time pod log collection
* datax task test have to mock ShellCommandExecutor with whole dependencies
* replace \n with EMPTY_STRING
* modify error log msg
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* throw exception in pod log collection thread
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-04-05 16:13:45 +08:00
Wenjun Ruan
64833952fc
Remove the extra default tenant in workflow save page ( #13867 )
2023-04-04 17:14:40 +08:00
Wenjun Ruan
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2023-03-30 14:11:23 +08:00
Wenjun Ruan
68660ec96b
Refactor remote command ( #13809 )
...
* Refactor remote command
* Rename Command to Message
2023-03-29 17:54:57 +08:00
Wenjun Ruan
d1b6e6f02c
Take over task instance in master failover ( #13798 )
2023-03-28 16:02:57 +08:00
Wenjun Ruan
1f365819a6
Ignore unknown VM options in start.sh ( #13719 )
2023-03-10 17:34:09 +08:00
Wenjun Ruan
aef5524ee7
Clearer task runnable. ( #13689 )
...
* Clean unused method in AbstractTask
* Kill task by process.destroy
* wait 5/s after destroy process
2023-03-09 20:46:29 +08:00
Wenjun Ruan
c9066e8de9
Use MDC to filter task instance log ( #13673 )
...
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2023-03-06 17:44:23 +08:00
Wenjun Ruan
ca9df868f1
Fix task instance log order might be wrong due to async print ( #13660 )
2023-03-02 22:37:05 +08:00
fuchanghai
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2023-02-22 14:22:49 +08:00
Aaron Wang
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes ( #13550 )
2023-02-21 23:26:21 +08:00
Rick Cheng
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2023-02-13 16:38:26 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
Wenjun Ruan
45694aa697
Fix MessageSender may be null due to class initialize late ( #13446 )
2023-01-28 20:53:32 +08:00
hokie-chan
3b980cb06a
[fix][worker][bug] master/worker crash when registry recover from SUSPENDED to RECONNECTED ( #13328 )
2023-01-03 19:24:11 +08:00
Aaron Wang
ccad56e88e
[Improvement][Master] Validate same content of input file when using task cache ( #13298 )
...
* support file content checksum
* fix inject null storageOperate bug
2023-01-03 11:38:13 +08:00
Wenjun Ruan
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
Wenjun Ruan
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2022-12-10 23:50:19 +08:00
Wenjun Ruan
ffc9fb280a
Add gc timestampt ( #13059 )
2022-12-01 10:00:27 +08:00
Wenjun Ruan
32a8241ef2
Don't source env file in task, this will override tenant's environment ( #13028 )
2022-11-28 21:09:50 +08:00
Yann Ann
3106054ea7
[Improvement-12907] Change heartbeat log level to debug ( #12980 )
2022-11-25 17:37:30 +08:00
rickchengx
38b876733c
[Feature-10498] Mask the password in the log of sqoop task ( #11589 )
2022-11-24 14:54:54 +08:00
John Bampton
27c37b8828
Fix grammar and spelling ( #12937 )
2022-11-18 23:03:34 +08:00
Kerwin
401fb4edd4
Fix execute shell task exception no dolphinscheduler_env.sh file execute permission ( #12909 )
2022-11-17 21:43:35 +08:00
Wenjun Ruan
403c6a6bb6
Fix memory leak in worker due to message retry map ( #12878 )
2022-11-13 20:08:38 +08:00
JieguangZhou
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2022-11-03 16:33:39 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
kezhenxu94
065d5caccc
Only expose necessary actuator endpoints ( #12571 )
2022-10-28 07:40:32 +08:00
HanayoZz
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
Wenjun Ruan
76a9e6d8a6
Add task executor threads full policy config in worker ( #12510 )
2022-10-26 10:29:22 +08:00
Wenjun Ruan
a0d37fc7ae
Set tenantDir permission ( #12486 )
2022-10-23 21:54:01 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2022-10-20 14:26:56 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
kezhenxu94
4ea7eec750
Use temurin Java docker images instead of deprecated ones ( #12334 )
2022-10-17 15:46:21 +08:00
Yann Ann
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules ( #12362 )
...
* migrate junit4 -> junit5 in master, worker, and remote modules
2022-10-15 21:58:31 +08:00
