Commit Graph

8375 Commits

Author SHA1 Message Date
wangxj3
7acb0f141f [maven-release-plugin] prepare release 3.2.2 2024-06-29 20:13:11 +08:00
wangxj3
b9b3648291 Merge remote-tracking branch 'upstream/3.2.2-prepare' into 3.2.2-release 2024-06-29 19:15:06 +08:00
Wenjun Ruan
1e3bbfce77
[Fix-16211] Remove ExecutePath in buildJarCommand to fix Java Task in Jar Mode (#16238) (#16241)
(cherry picked from commit d42ac96c55)

Co-authored-by: blink <lanxing2@users.noreply.github.com>
2024-06-29 18:00:33 +08:00
wangxj3
0c917dec1e [maven-release-plugin] prepare for next development iteration 2024-06-28 15:46:10 +08:00
wangxj3
463d687119 [maven-release-plugin] prepare release 3.2.2 2024-06-28 15:42:21 +08:00
wangxj3
2b56d3ca44 [maven-release-plugin] prepare for next development iteration 2024-06-27 19:12:54 +08:00
wangxj3
8f578f62ee [maven-release-plugin] prepare release 3.2.2 2024-06-27 18:54:40 +08:00
wangxj3
6aa73a1d95 Merge remote-tracking branch 'upstream/3.2.2-prepare' into 3.2.2-release 2024-06-27 11:19:34 +08:00
Wenjun Ruan
f13f070242
Fix worker overload will casue task failed (#16223)
(cherry picked from commit 4a41e4290ec25bf1165ebc4f043baffb13729a72)
2024-06-27 08:25:42 +08:00
wangxj3
eecef8a3e9 [maven-release-plugin] prepare for next development iteration 2024-06-26 18:11:14 +08:00
wangxj3
d7ad8437ac [maven-release-plugin] prepare release 3.2.2 2024-06-26 17:39:24 +08:00
xiangzihao
66b7d1274a
[Chore] [Cherry-Pick] cherry pick some pr to 3.2.2 (#16215)
* [Fix-16174] Incorrect cluster installation guide. (#16208)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

* [Improvement][Helm] Make configmap of api/master/worker/alert configuration (#16058)

Update deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml

* Update deploy/kubernetes/dolphinscheduler/values.yaml

* [helm] remove appversion from labels (#16066)
2024-06-26 14:45:37 +08:00
xiangzihao
fe96907fbb
[Chore] Revert some incompatible change for 3.2.2 (#16214)
* revert pr 14977

* fix spotless

* revert pr 14977

* revert pr 14977
2024-06-26 13:17:51 +08:00
Wenjun Ruan
e65208b34e
Fix task cannot use workflow's environment (#16199) (#16206)
(cherry picked from commit b34fe46044)
2024-06-24 23:00:18 +08:00
Wenjun Ruan
2f928b15c9
[Chore] Disable composite indicator (#16172) (#16205)
(cherry picked from commit 3f3f0cd4cb)
2024-06-24 22:02:13 +08:00
zhuxt2015
e524cfe744
[Improvement-16125][dolphinscheduler-dist] Reduce binary distribution tarball file size (#16192) 2024-06-24 21:50:18 +08:00
xiangzihao
16bc1e7bfb
Revert "3.2.2 release add task-pure delete some task module to reduce the si…" (#16184)
This reverts commit 3ad53c5200.
2024-06-20 13:44:55 +08:00
xiangzihao
3cdf754b1b
Fix admin login tenant unspecified issue after upgrade to v3.2.1 (#16064) (#16185) 2024-06-20 13:44:28 +08:00
Wenjun Ruan
82ec84f85c
Fix condition task will cause workflow instance failed (#16152) (#16177)
(cherry picked from commit 5df4b1cbc3)
2024-06-19 12:39:48 +08:00
wangxj3
3ad53c5200
3.2.2 release add task-pure delete some task module to reduce the size of the deployment package (#16094)
* add task-pure delete some task module to reduce the size of the deployment package

* add version of 3.2.2

* Update Chart.yaml Rollback APPversion

* remove task pure plugin

* fix ci

* add pure datasource plugin

* remove ck,sr from datasource pure

* remove task plugin chunjun ,dms ,datafactory from pure

* add api pom dependency

* spotless:apply

* add dependency license

* [maven-release-plugin] prepare release 3.2.2

* [maven-release-plugin] prepare for next development iteration

* remove datasource-pure:
dolphinscheduler-datasource-sqlserver
dolphinscheduler-datasource-oracle

remove task-pure:
dolphinscheduler-task-dinky
dolphinscheduler-task-pytorch
dolphinscheduler-task-datasync
dolphinscheduler-task-kubeflow
dolphinscheduler-task-emr
dolphinscheduler-task-pigeon
dolphinscheduler-task-flink-stream

* add register pure remove jdbc and etcd

* add dolphinscheduler-storage-pure remove task of
dolphinscheduler-storage-obs
dolphinscheduler-storage-abs
dolphinscheduler-storage-gcs
dolphinscheduler-storage-oss

* mvn spotless:apply

---------

Co-authored-by: wangxj959 <wangxj959@chinaunicom.cn>
Co-authored-by: wangxj <wangxj31>
Co-authored-by: wangxj3 <wangxj@apache.org>
2024-06-08 17:05:20 +08:00
Wenjun Ruan
e91a272965
Fix SqlServer split merge sql will miss ; (#16118) (#16120)
(cherry picked from commit d6714bb77a)
2024-06-06 17:11:51 +08:00
Wenjun Ruan
31e79d10ea
Fix task nodes under switch task will not be skipped (#16109)
(cherry picked from commit f687a7d851c206ae30377ac3f9dc025cec35ffb2)
2024-06-05 23:41:52 +08:00
Wenjun Ruan
50c434e6c4
Fix task instance which status is dispatch will not be failover (#16110)
(cherry picked from commit a59ecf2aa19f4acd8d9981d14abe4d0c0d8e7fec)
2024-06-05 13:31:29 +08:00
Wenjun Ruan
72b402422a
Revert "[Improvement] Abnormal characters check (#15824)" (#16102) (#16103)
This reverts commit e5e7749251.

(cherry picked from commit 7ae9ca5a33)
2024-05-31 21:21:14 +08:00
xiangzihao
fd864bf351
Revert "3.2.2 release cherry-pick cp: Reduce the size of tarball to continue …" (#16068)
This reverts commit 646bbf4881.
2024-05-25 10:50:37 +08:00
wangxj3
646bbf4881
3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release (#16042)
* add version of 3.2.2

* Fix rpc channel leak due to concurrent operation (#16021)

* Fix rpc channel leak due to concurrent operation

* Throw channel create failed exception

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027)

* cp: Reduce the size of tarball to continue ASF release (#15004) (#15540)

* Reduce the size of tarball to continue ASF release

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt
# Conflicts:
#	tools/dependencies/known-dependencies.txt

* Increase block-until-connected in ZookeeperRegistryTestCase (#16041)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033)

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967)

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* improvement project code

* remove useless imports

* remove method onClearSearchTaskType

* add parameter doc

* optimisation logic

* code conflict resolution

* code conflict resolution

* [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)

* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* [Improvement][Monitor] Add UT for montor (#15998)

* formatting Code

* add pom

* Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038)

* Removed unused StateEventHandleFailure (#16052)

* Remove OmitStackTraceInFastThrow in start.sh (#16054)

* Update Chart.yaml Rollback APPversion

* modify log msg (#16062)

Co-authored-by: xiangzihao <460888207@qq.com>

* [DS-16046][fix] Set PreparedStatement parameter type (#16050)

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time

Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Update pom.xml

* [DSIP-42] Add dolphinscheduler-aws-authentication module  (#16043)

---------

Co-authored-by: wangxj <wangxj31>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: TianXinCoord <tianxincoord@163.com>
Co-authored-by: 小可耐 <46134044+sdhzwc@users.noreply.github.com>
Co-authored-by: sleo <97011595+alei1206@users.noreply.github.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: yinxiaolog <43245392+yinxiaolog@users.noreply.github.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2024-05-25 10:16:18 +08:00
jiabei
7b7a33337f
docs: fix type in contribute.md (#16022)
Co-authored-by: jxbwhe <iswanghe@aliyun.com>
2024-05-18 21:34:08 +08:00
Wenjun Ruan
cb0b4fa52f
Deprecated 2.0.9/3.0.6 upgrade test (#16012) 2024-05-17 19:17:54 +08:00
Wenjun Ruan
4e73fc28b0
Use random port in registry IT (#16014) 2024-05-17 17:35:44 +08:00
JohnHuang
79f90a87f8
[Feature][Helm] make api/master/worker/alert application.yaml configurable (#15922) 2024-05-17 15:23:21 +08:00
Wenjun Ruan
61915a2d5d
Fix workflow instance restart failed due to duplicate key in varpool (#16001) 2024-05-16 11:32:27 +08:00
旺阳
68a190674f
[Improvement][Monitor] Add UT for montor (#15998) 2024-05-15 14:44:04 +08:00
旺阳
0e5cb664bb
[Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)
* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2024-05-15 13:58:59 +08:00
Gee
4f4a13da02
[Fix-15992][server] maintain slot index by iterate MasterPriorityQueue orderly (#15993) 2024-05-15 11:44:27 +08:00
Wenjun Ruan
1a4537ecbd
Fix database schema change check failed with pg in ci (#15995)
Co-authored-by: xiangzihao <460888207@qq.com>
2024-05-15 10:36:27 +08:00
John Bampton
8d6e9eecfc
docs: fix spelling (#15996) 2024-05-15 09:41:02 +08:00
Eric Gao
dc306bfa1d
[Improvement] Use safe constructor with snake yaml (#15758) 2024-05-14 18:04:04 +08:00
licc6
680d2e0b05
Repair resource configuration exist typo (#15988)
Co-authored-by: fuchanghai <changhaifu@apache.org>
2024-05-13 19:47:47 +08:00
Wenjun Ruan
7c8fa9b48c
Add IntegretionTest for registry module (#15981) 2024-05-13 16:35:37 +08:00
Wenjun Ruan
3446fd8ab1
EMR task support replace params placeholder (#15975)
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2024-05-10 17:34:26 +08:00
Gallardot
ace20f96c2
[Bug] [Helm] No DB Nodes Exist (#15970) 2024-05-10 15:43:00 +08:00
JohnHuang
ad1a6af4fb
Add link to ETCD/JDBC Registry Guideline (#15597) 2024-05-09 16:52:37 +08:00
cntiger
60b019b729
[Improvement] Fix the git url command injection in pytorch task(#15873) (#15950)
* fix the git url command injection danger(#15873)

* [Improvement] Fix the git url command injection in pytorch,format code style task(#15873)

---------

Co-authored-by: cntigers <Xiaohu4321@>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-09 14:50:27 +08:00
Zzih96
5c569b705c
[fix-15907] Fix get remote shell exit code is incorrect (#15911)
* ./mvnw spotless:apply

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-remoteshell/src/main/java/org/apache/dolphinscheduler/plugin/task/remoteshell/RemoteExecutor.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

---------

Co-authored-by: 詹子恒 <ziheng.zhan@longbridge.sg>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-09 14:15:06 +08:00
刘阳阳(liuyangyang)
ba5de75829
Add tenantCode propagation to DynamicCommandUtils.createCommand (#15956) 2024-05-09 13:06:22 +08:00
Wenjun Ruan
8d336def61
[DSIP-35][Alert] Refactor the alert thread model (#15932) 2024-05-09 12:23:01 +08:00
privking
bbca37d03e
[FIX] Completed tasks cannot be re-executed in a workflow instance (#15884)
* fix bug: Failed to resume stopped workflow instance

* Revert "fix bug: Failed to resume stopped workflow instance"

This reverts commit 1546e9d5a51178a94bedd18a718b15431355428b.

* fix bug : Completed tasks cannot be re-executed in a workflow instance

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-09 11:36:56 +08:00
xiangzihao
8426d2346c
[HotFix] [CI] Temporary skipping mergeable check (#15958)
* temporary skipping mergeable check
2024-05-09 11:01:05 +08:00
Wenjun Ruan
99d8276be7
Optimizing the scope of RPC base classes (#15946)
* Optimizing the scope of RPC base classes

* Fix UT
2024-05-08 11:46:09 +08:00
JohnHuang
d218b021e7
[DSIP-25][Remote Logging] Split remote logging configuration (#15826)
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-06 11:33:16 +08:00