Wenjun Ruan
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2022-12-10 23:50:19 +08:00
Wenjun Ruan
ffc9fb280a
Add gc timestampt ( #13059 )
2022-12-01 10:00:27 +08:00
Wenjun Ruan
32a8241ef2
Don't source env file in task, this will override tenant's environment ( #13028 )
2022-11-28 21:09:50 +08:00
Yann Ann
3106054ea7
[Improvement-12907] Change heartbeat log level to debug ( #12980 )
2022-11-25 17:37:30 +08:00
rickchengx
38b876733c
[Feature-10498] Mask the password in the log of sqoop task ( #11589 )
2022-11-24 14:54:54 +08:00
John Bampton
27c37b8828
Fix grammar and spelling ( #12937 )
2022-11-18 23:03:34 +08:00
Kerwin
401fb4edd4
Fix execute shell task exception no dolphinscheduler_env.sh file execute permission ( #12909 )
2022-11-17 21:43:35 +08:00
Wenjun Ruan
403c6a6bb6
Fix memory leak in worker due to message retry map ( #12878 )
2022-11-13 20:08:38 +08:00
JieguangZhou
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2022-11-03 16:33:39 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
kezhenxu94
065d5caccc
Only expose necessary actuator endpoints ( #12571 )
2022-10-28 07:40:32 +08:00
HanayoZz
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
Wenjun Ruan
76a9e6d8a6
Add task executor threads full policy config in worker ( #12510 )
2022-10-26 10:29:22 +08:00
Wenjun Ruan
a0d37fc7ae
Set tenantDir permission ( #12486 )
2022-10-23 21:54:01 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2022-10-20 14:26:56 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
kezhenxu94
4ea7eec750
Use temurin Java docker images instead of deprecated ones ( #12334 )
2022-10-17 15:46:21 +08:00
Yann Ann
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules ( #12362 )
...
* migrate junit4 -> junit5 in master, worker, and remote modules
2022-10-15 21:58:31 +08:00
Eric Gao
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project ( #12244 )
...
* Fully remove the usage of powermock from the whole project
* Upgrade org.reflections to 0.10.12
2022-10-10 10:43:38 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
caishunfeng
40090d459e
[Improvement] remove log-server and server module ( #12206 )
...
* remove log-server module
* remove dolphinscheduler-server module
2022-09-30 16:41:41 +08:00
Wenjun Ruan
aacabc6ec7
Remove workerGroup in registry ( #12217 )
2022-09-30 13:58:53 +08:00
sgw
962fa6dbd5
[DS-12154][worker] Optimize the log printing of the worker module ( #12183 )
...
* [DS-12154][worker] Optimize the log printing of the worker module according to the log specification.
2022-09-29 16:01:29 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
Wenjun Ruan
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2022-09-14 16:18:08 +08:00
Daniel Y
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 ( #11835 )
...
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9
* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-14 09:29:45 +08:00
limaiwang
25b78a8003
[Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql ( #11721 )
...
select different versions of spark-sql to execute different versions of spark-sql
2022-09-12 10:35:27 +08:00
LiuBodong
3ca9680b20
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly ( #11870 )
2022-09-11 19:11:03 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
Kengo Seki
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
冯剑
45ebd528d6
[Bug-11650][worker] #11650 fix SQL type task, stop task cause NPE ( #11668 )
2022-09-06 10:12:40 +08:00
kezhenxu94
277f137358
Add Kubernetes configmap reload to all components ( #11730 )
2022-09-02 12:03:19 +08:00
Wenjun Ruan
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2022-08-31 16:20:23 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2022-08-31 15:25:01 +08:00
Wenjun Ruan
1b120e3a59
Refactor worker execute task process ( #11540 )
...
* Refactor worker execute task process
2022-08-26 13:33:51 +08:00
JinYong Li
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug ( #11232 )
...
* fix heartBeat bug
* modify class name
* fix conflict
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2022-08-23 11:30:13 +08:00
Wenjun Ruan
3516533017
Remove logger header in task log file ( #11555 )
2022-08-19 14:01:52 +08:00
Eric Gao
9330d6cfcd
[Doc][Security] Update instructions on worker groups ( #11483 )
...
* Update instructions on worker groups
2022-08-15 17:44:00 +08:00
Wenjun Ruan
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2022-08-13 09:52:03 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
Eric Gao
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server ( #11240 )
...
* [Improvement][Metrics] Add metrics for alert server (#11131 )
* Update related docs of metrics
* Add grafana demo dashboards for alert server metrics
* Refactor metric classes with UtilityClass annotation
* Refactor meter names in camelCase for checkstyle
2022-08-03 15:42:06 +08:00
xuhhui
bfff3a7c5d
fix error ( #11206 )
2022-07-30 18:20:20 +08:00
zhuxt2015
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2022-07-22 13:34:31 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
zhuxt2015
a74d7ef665
[hotfix][Worker] Remove service dependency from worker module ( #11008 )
...
* worker remove service dependency
2022-07-17 22:16:35 +08:00
