Commit Graph

8355 Commits

Author SHA1 Message Date
Wenjun Ruan
e91a272965
Fix SqlServer split merge sql will miss ; (#16118) (#16120)
(cherry picked from commit d6714bb77a)
2024-06-06 17:11:51 +08:00
Wenjun Ruan
31e79d10ea
Fix task nodes under switch task will not be skipped (#16109)
(cherry picked from commit f687a7d851c206ae30377ac3f9dc025cec35ffb2)
2024-06-05 23:41:52 +08:00
Wenjun Ruan
50c434e6c4
Fix task instance which status is dispatch will not be failover (#16110)
(cherry picked from commit a59ecf2aa19f4acd8d9981d14abe4d0c0d8e7fec)
2024-06-05 13:31:29 +08:00
Wenjun Ruan
72b402422a
Revert "[Improvement] Abnormal characters check (#15824)" (#16102) (#16103)
This reverts commit e5e7749251.

(cherry picked from commit 7ae9ca5a33)
2024-05-31 21:21:14 +08:00
xiangzihao
fd864bf351
Revert "3.2.2 release cherry-pick cp: Reduce the size of tarball to continue …" (#16068)
This reverts commit 646bbf4881.
2024-05-25 10:50:37 +08:00
wangxj3
646bbf4881
3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release (#16042)
* add version of 3.2.2

* Fix rpc channel leak due to concurrent operation (#16021)

* Fix rpc channel leak due to concurrent operation

* Throw channel create failed exception

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027)

* cp: Reduce the size of tarball to continue ASF release (#15004) (#15540)

* Reduce the size of tarball to continue ASF release

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt
# Conflicts:
#	tools/dependencies/known-dependencies.txt

* Increase block-until-connected in ZookeeperRegistryTestCase (#16041)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033)

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967)

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* improvement project code

* remove useless imports

* remove method onClearSearchTaskType

* add parameter doc

* optimisation logic

* code conflict resolution

* code conflict resolution

* [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)

* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* [Improvement][Monitor] Add UT for montor (#15998)

* formatting Code

* add pom

* Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038)

* Removed unused StateEventHandleFailure (#16052)

* Remove OmitStackTraceInFastThrow in start.sh (#16054)

* Update Chart.yaml Rollback APPversion

* modify log msg (#16062)

Co-authored-by: xiangzihao <460888207@qq.com>

* [DS-16046][fix] Set PreparedStatement parameter type (#16050)

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time

Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Update pom.xml

* [DSIP-42] Add dolphinscheduler-aws-authentication module  (#16043)

---------

Co-authored-by: wangxj <wangxj31>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: TianXinCoord <tianxincoord@163.com>
Co-authored-by: 小可耐 <46134044+sdhzwc@users.noreply.github.com>
Co-authored-by: sleo <97011595+alei1206@users.noreply.github.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: yinxiaolog <43245392+yinxiaolog@users.noreply.github.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2024-05-25 10:16:18 +08:00
jiabei
7b7a33337f
docs: fix type in contribute.md (#16022)
Co-authored-by: jxbwhe <iswanghe@aliyun.com>
2024-05-18 21:34:08 +08:00
Wenjun Ruan
cb0b4fa52f
Deprecated 2.0.9/3.0.6 upgrade test (#16012) 2024-05-17 19:17:54 +08:00
Wenjun Ruan
4e73fc28b0
Use random port in registry IT (#16014) 2024-05-17 17:35:44 +08:00
JohnHuang
79f90a87f8
[Feature][Helm] make api/master/worker/alert application.yaml configurable (#15922) 2024-05-17 15:23:21 +08:00
Wenjun Ruan
61915a2d5d
Fix workflow instance restart failed due to duplicate key in varpool (#16001) 2024-05-16 11:32:27 +08:00
旺阳
68a190674f
[Improvement][Monitor] Add UT for montor (#15998) 2024-05-15 14:44:04 +08:00
旺阳
0e5cb664bb
[Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)
* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2024-05-15 13:58:59 +08:00
Gee
4f4a13da02
[Fix-15992][server] maintain slot index by iterate MasterPriorityQueue orderly (#15993) 2024-05-15 11:44:27 +08:00
Wenjun Ruan
1a4537ecbd
Fix database schema change check failed with pg in ci (#15995)
Co-authored-by: xiangzihao <460888207@qq.com>
2024-05-15 10:36:27 +08:00
John Bampton
8d6e9eecfc
docs: fix spelling (#15996) 2024-05-15 09:41:02 +08:00
Eric Gao
dc306bfa1d
[Improvement] Use safe constructor with snake yaml (#15758) 2024-05-14 18:04:04 +08:00
licc6
680d2e0b05
Repair resource configuration exist typo (#15988)
Co-authored-by: fuchanghai <changhaifu@apache.org>
2024-05-13 19:47:47 +08:00
Wenjun Ruan
7c8fa9b48c
Add IntegretionTest for registry module (#15981) 2024-05-13 16:35:37 +08:00
Wenjun Ruan
3446fd8ab1
EMR task support replace params placeholder (#15975)
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2024-05-10 17:34:26 +08:00
Gallardot
ace20f96c2
[Bug] [Helm] No DB Nodes Exist (#15970) 2024-05-10 15:43:00 +08:00
JohnHuang
ad1a6af4fb
Add link to ETCD/JDBC Registry Guideline (#15597) 2024-05-09 16:52:37 +08:00
cntiger
60b019b729
[Improvement] Fix the git url command injection in pytorch task(#15873) (#15950)
* fix the git url command injection danger(#15873)

* [Improvement] Fix the git url command injection in pytorch,format code style task(#15873)

---------

Co-authored-by: cntigers <Xiaohu4321@>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-09 14:50:27 +08:00
Zzih96
5c569b705c
[fix-15907] Fix get remote shell exit code is incorrect (#15911)
* ./mvnw spotless:apply

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-remoteshell/src/main/java/org/apache/dolphinscheduler/plugin/task/remoteshell/RemoteExecutor.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

---------

Co-authored-by: 詹子恒 <ziheng.zhan@longbridge.sg>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-09 14:15:06 +08:00
刘阳阳(liuyangyang)
ba5de75829
Add tenantCode propagation to DynamicCommandUtils.createCommand (#15956) 2024-05-09 13:06:22 +08:00
Wenjun Ruan
8d336def61
[DSIP-35][Alert] Refactor the alert thread model (#15932) 2024-05-09 12:23:01 +08:00
privking
bbca37d03e
[FIX] Completed tasks cannot be re-executed in a workflow instance (#15884)
* fix bug: Failed to resume stopped workflow instance

* Revert "fix bug: Failed to resume stopped workflow instance"

This reverts commit 1546e9d5a51178a94bedd18a718b15431355428b.

* fix bug : Completed tasks cannot be re-executed in a workflow instance

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-09 11:36:56 +08:00
xiangzihao
8426d2346c
[HotFix] [CI] Temporary skipping mergeable check (#15958)
* temporary skipping mergeable check
2024-05-09 11:01:05 +08:00
Wenjun Ruan
99d8276be7
Optimizing the scope of RPC base classes (#15946)
* Optimizing the scope of RPC base classes

* Fix UT
2024-05-08 11:46:09 +08:00
JohnHuang
d218b021e7
[DSIP-25][Remote Logging] Split remote logging configuration (#15826)
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-05-06 11:33:16 +08:00
Evan Sun
fa6ea8ba7b
[TEST] increase coverage of project workergroup relation service (#15944)
Co-authored-by: abzymeinsjtu <abzymeinsjtu@B-54Q8MD6R-0244.local>
2024-05-06 10:02:04 +08:00
Evan Sun
ebcdaeb9ac
[TEST] increase coverage of project preference service test (#15939) 2024-04-29 21:03:01 +08:00
Wenjun Ruan
0a11cd21bd
Fix jar path is not correct in java task (#15906) 2024-04-29 18:07:05 +08:00
Wenjun Ruan
647cbae400
[DSIP-32][Master] Add command fetcher strategy for master fetch command (#15900) 2024-04-29 16:14:23 +08:00
calvin
5a6c6c37f0
[Improvement-15910][UI] Supposed to provide a default value for the custom parallelism when using the mode of parallel execution. (#15912)
* worked out the issue

* imrpove the parallism strategy

* imrpove the parallism strategy

* merge from dev
2024-04-28 18:47:20 +08:00
小可耐
7a55adeae9
[Improvement-15919][datasource] Improvement datasource get name (#15920) 2024-04-28 11:02:44 +08:00
DaqianLiao
b29965bdce
[Fix] In updateWorkerNodes method, the workerNodeInfoWriteLock should be used. #15898 (#15903)
Co-authored-by: answerliao <answerliao@tencent.com>
2024-04-26 11:24:05 +08:00
Wenjun Ruan
d1135eabc7
[DSIP-34] Change required_approving_review_count to 2 (#15914) 2024-04-25 19:39:56 +08:00
Wenjun Ruan
446f6ba72b
Fix auto create tennat concurrently will cause the task failed (#15909) 2024-04-25 18:14:48 +08:00
Wenjun Ruan
b3b8c0784d
Fix kill dynamic task doesn't kill the wait to run workflow instances (#15896) 2024-04-25 11:05:51 +08:00
Evan Sun
1d13ef09c3
[TEST] increase coverage of project parameter service test (#15905)
Co-authored-by: abzymeinsjtu <abzymeinsjtu@B-54Q8MD6R-0244.local>
2024-04-24 23:31:56 +08:00
旺阳
59f060e278
[Improvement] Fix alert code smell 2024-04-23 15:10:08 +08:00
Wenjun Ruan
e9d85914d7
Fix queryByTypeAndJobId might error due to multiple result (#15883) 2024-04-22 17:41:17 +08:00
Wenjun Ruan
e2c8b080f9
[DSIP-31] Reduce the thread pool size of hikari (#15890) 2024-04-22 17:05:24 +08:00
John Huang
6c78c8ec9a
[Improvement][Spark] Support Local Spark Cluster (#15589)
* [Improvement][Spark] Support Local Spark Cluster

* remote default local from deploy mode

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-04-22 15:05:59 +08:00
privking
e66441a2c9
[FIX] Fix cannot recover a stopped workflow instance (#15880) 2024-04-20 07:32:18 +08:00
Wenjun Ruan
285c5a8eb5
[DSIP-28] Donnot scan whole bean under classpath (#15874) 2024-04-19 18:12:40 +08:00
Gallardot
a9decc911f
[Bug][Helm] fix image.registry (#15860)
Signed-off-by: Gallardot <gallardot@apache.org>
Co-authored-by: fuchanghai <changhaifu@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2024-04-19 16:52:17 +08:00
旺阳
d306f1d04b
Refactor record audit log logic (#15881) 2024-04-19 15:20:31 +08:00
Wenjun Ruan
32f9288974
Fix dsip name (#15876) 2024-04-19 10:31:34 +08:00