test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-02 04:08:31 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
8,002 Commits 27 Branches 59 Tags 276 MiB
a491e3a849
Commit Graph

320 Commits

Author SHA1 Message Date
Aaron Wang
a491e3a849
fix: null app-info-path cause NPE (#14752) 2023-08-19 13:28:45 +08:00
caishunfeng
59998f4c75
[Improvement] add dispatch fail times when compare (#14761) 
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
 2023-08-18 15:26:01 +08:00
KingsleyY
b3af374fe6
[Fix-14646][Worker] Fix match wrong pid on linux server (#14652) 
* [DS-14646][Worker] Fix match wrong pid on linux server
This closes #14646

---------

Co-authored-by: yangxin3 <yangxin3@newhope.cn>
 2023-08-17 19:17:16 +08:00
Edison Catto
25480ae9e7
[Feature][Resource Center] Add support for Huawei Cloud OBS (#14643) 
* [Feature][Resource Center] Add support for Huawei Cloud OBS as storage of resource center

* add license and add doc

* add 3-party dependency license

* Update LICENSE

* fix

* Update pom.xml

* fix

* fix

---------

Co-authored-by: sunkang <sunkang@hjfruit.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
 2023-08-14 13:25:48 +08:00
Aaron Wang
cb55476235
[Fix-14721] [K8S Task] Handle job delete event when user manually delete the job in k8s cluster (#14722) 
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
 2023-08-11 17:15:07 +08:00
Aaron Wang
5ec9085113
fix: do not override k8sConfigYaml in ShShellInterceptorBuilder (#14682) 
Co-authored-by: xiangzihao <460888207@qq.com>
 2023-08-02 18:01:34 +08:00
Wenjun Ruan
9ce8871158
Support get property from env/jvm/common.properties (#14645) 2023-07-28 14:41:58 +08:00
KingsleyY
512f4c7021
[Fix-10795][plugin] sqoop task query columns bug (#10776) 
* [Improvement][Api] add @Override on the Override method

* [FIX]sqoop task query columns bug

* merge two if

* update checkstyle problem

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopColumnType.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/generator/sources/MySQLSourceGenerator.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* code format

* code format

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: xiangzihao <460888207@qq.com>
 2023-07-26 10:42:29 +08:00
Wenjun Ruan
4aab0b234f
Use AdHoc datasource client in sqlTask (#14631) 
* Use AdHoc datasource client in sqlTask

* Add method in DataSourceChannel to create PooledDataSourceClient and AdHocDataSourceClient
 2023-07-26 08:58:50 +08:00
Eric Gao
d6a4e99e19
fix process utils comment typo (#14613) 2023-07-21 13:18:10 +08:00
Aaron Wang
04800a48d1
[Feature-13763][K8S Task] collect real-time log (#14379) 
* [Improvement-13763][K8S Task] collect real-time log

* fix codesmell

* get pod watcher until pod is ready

* fix codesmell

* specify container name & loop waiting pod creation

* sleep when pod is not initialized

---------

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
 2023-07-20 18:38:25 +08:00
Kerwin
9664bef933
Fix wrong judgment condition when building parameters of seatunnel task node. (#14594) 2023-07-20 12:26:10 +08:00
Jay Chung
de2d215aee
impv: Correct datax execute and python execute script name (#14523) 
* impv: Correct datax execute and python execute script name

we use PYTHON_LAUNCHER for python script execute and
DATAX_LAUNCHER for datax script name

* Add pr number

* fix ut

* style

---------

Co-authored-by: xiangzihao <460888207@qq.com>
 2023-07-20 09:13:55 +08:00
Wenjun Ruan
c30cca9d9a
Support execute shell in different interceptor (#14582) 2023-07-19 14:50:04 +08:00
Rick Cheng
9778652f17
[Improvement-14136][task] Support submitting spark-sql task with the sql file in resource center (#14527) 
Co-authored-by: xiangzihao <460888207@qq.com>
 2023-07-19 10:09:30 +08:00
FlechazoW
027d66d200
[Typo] standardized license header. (#14548) 
Co-authored-by: wentz <wentz@getui.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
 2023-07-17 13:56:23 +08:00
Wenjun Ruan
e6d94632b9
Add WorkflowExecuteContext (#14544) 2023-07-14 18:40:01 +08:00
Rick Cheng
09afa75a3d
[Improvement][Worker] No need to kill remote app when appid is empty (#14534) 2023-07-13 15:40:58 +08:00
Rick Cheng
c1cffba349
[Improvement] Remove the unused newDaemonSingleThreadExecutor (#14520) 2023-07-12 21:03:41 +08:00
zyfjrx
4111acb58e
[Bug] [Seatunnel-Task] Run Mode defaults to run, and you cannot select a null value (#14267) 
Co-authored-by: zhang <zhangyfcoder@163.com>
 2023-07-12 09:06:19 +08:00
旺阳
175d976cd2
[Feature][Data Quality] Data Quality Support Choose Database (#14406) 
* add dataquality database api

* change ui

* api change

* update

* fix spotless

* fix h2

* fix pg

* fix-dead-line

* update

* fix-spotless

* update pg sql

* add ut

* fix ut
 2023-07-12 09:00:28 +08:00
jegger
75d29f6101
[improvement] DataQuality module improve (#14463) 2023-07-10 13:38:09 +08:00
Aaron Wang
0b69236b04
[Feature-14251][Task] Support yarn queue definition in yarn task (#14310) 2023-07-07 14:24:47 +08:00
fuchanghai
742a2dc780
[Fix-14447][K8S Task] Fix IndexOutofBoundsException for pod name without dash (#14448) 2023-07-07 12:53:16 +08:00
Rick Cheng
cff53feaeb
[Improvement-14464][Task] Remove the useless taskResultString (#14465) 2023-07-06 17:31:33 +08:00
Rick Cheng
68a3a75211
[Improvement-14457][UT] Improve the expected and actual value of Assertions (#14459) 2023-07-06 13:12:48 +08:00
Aaron Wang
65b6a4b097
[Feature-14421][K8S Task] Configurable image pull policy (#14426) 2023-07-05 09:17:47 +08:00
Rick Cheng
13dfa859b5
[Improvement-14371][Hivecli] Add support for hivevar setting in hivecli task (#14423) 2023-07-04 14:02:23 +08:00
Jeremy
9c92b4b240
fix: default datax plugin (#14204) 2023-06-25 16:18:38 +08:00
Rick Cheng
b49b91c45d
[Improvement-14331][common] Remove the duplicate ParameterUtils (#14332) 2023-06-14 15:00:54 +08:00
Wenjun Ruan
a55612e4f7
Fix switch task use boolean type error (#14326) 2023-06-13 13:26:05 +08:00
Rick Cheng
1cd5070b5d
[Improvement-14329][common] Remove the duplicate BusinessTimeUtils (#14330) 2023-06-13 08:30:34 +08:00
Rick Cheng
4fa0ab4d7e
[feature][Resource] Add support for Azure Blob Storage in the resource center (#14082) 2023-06-12 12:05:49 +08:00
ORuteMa
de2cc0e235
[bug][plugin]Fix: Correct the way to determine the yarn queue in Flink CommandLine and SQL mode (#14237) 
* Fix: Correct the way to determine the yarn queue in Flink CommandLine

* fix the yarn queue in sql mode && refine the code

* refine code

* remove unnecessary comment

* fix yarn queue properties

* remove redundant variable
 2023-06-12 11:04:39 +08:00
Aaron Wang
b7fb393eed
fix: trim node selector requirement value (#14307) 2023-06-09 18:06:52 +08:00
Rick Cheng
66e09dc66e
[Improvement-14295][common] Remove the duplicate FileUtils (#14301) 2023-06-08 08:59:53 +08:00
JieguangZhou
e7cdc7c136
[Feature][Task Plugin] Add Dynamic task to generate dynamic worklfows with list parameters (#14127) 
* add dynamic task plugin

* fix ui

* update code

* add DOC

* ADD UT

* fix doc

* fix sanity check

* add upgrade ddl
* fix sql
 2023-06-07 16:01:06 +08:00
Rick Cheng
2c057d6a8f
[Improvement-14292][Common] Remove the duplicate OSUtils (#14293) 2023-06-07 14:59:25 +08:00
Rick Cheng
4d798e774e
[Improvement-14280][Task] update the task execution status check (#14281) 2023-06-06 15:25:57 +08:00
HQFGit
d587b5f03d
[Bug][Master][Fixes 14216]workflow state keep READY_STOP when sub_process state … (#14221) 
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes #14216

* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes  mvn spotless:apply#14216

* Update testFile.txt.crc

* Update testFile.txt.crc

* Delete testFile.txt.crc
 2023-06-06 11:04:32 +08:00
Aaron Wang
9e0991145e
[Improvement-14146][K8S Task] print complete error logs in task log file (#14147) 2023-05-27 17:39:03 +08:00
Rick Cheng
1011852ab2
[Improvement-14195][Parameter] remove misleading comment of combining local and global parameter (#14196) 2023-05-26 09:29:09 +08:00
Aaron Wang
4e78248238
[Feature][K8S Task] support node selector (#14126) 
* feat: support K8S node selector

* fix CI
 2023-05-26 09:21:23 +08:00
haibingtown
af7da0630e
[Feature] Add project name/workflow definition name/instance id/task name to the built-in variables (#14099) 
* [Feature] Add project name/workflow definition name/instance id/task name to the built-in variables
 2023-05-25 11:02:39 +08:00
Alex Ting
606b56403e
[Fix-13381][plugin] fix error while parsing hive load sql (#13378) 
* [fix][task plugin][task sql] fix parsing error while using variables sometimes parsing exception happened if task sql contains a variable followed by \" or \' .

for example:
input sql: load inpath '/tmp/test_table/dt=${dt}'
into table test_table partition(dt=${dt})

after replace: preparing : load inpath '/tmp/test_table/dt=?
into table test_table partition(dt=?)
 2023-05-22 11:19:30 +08:00
Kerwin
8315279620
[Feature-#14148][Task] Added status check interval and dependent failure policy parameters for dependent task nodes (#14150) 
* Added a dependent task failure policy in the dependent task node
 2023-05-19 14:49:50 +08:00
Jay Chung
56b0f91f6e
fix: Save task seatunnel error (#14129) 2023-05-19 10:47:16 +08:00
fuchanghai
67f50e3904
[fix-#13940] add status of `` stop`` in class of TaskExecutionStatus (#14016) 2023-05-18 13:31:02 +08:00
Aaron Wang
c3809fbecf
[Improvement-13868][K8S Task] Support the custom labels for k8s task (#14067) 
* [Improvement-13868] Support the custom labels for k8s task

* fix front-end type mismatched
 2023-05-16 12:04:33 +08:00
JieguangZhou
a3b2cc0275
[CI] Try fix api test error (#14094) 
* try fix api test error

* add workergroupapitest to ci and fix log output

* fix log output

* skip remote shell UT

---------

Co-authored-by: SbloodyS <460888207@qq.com>
 2023-05-12 21:39:00 +08:00