test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-03 04:39:00 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
7,284 Commits 27 Branches 59 Tags 276 MiB
9ab79e064c
Commit Graph

781 Commits

Author SHA1 Message Date
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project (#12244) 
* Fully remove the usage of powermock from the whole project

* Upgrade org.reflections to 0.10.12
 2022-10-10 10:43:38 +08:00
calvin
b4947b471c
fix this issue (#12281) 2022-10-10 09:34:32 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones (#12239) 
* fix DS support cross time zone, use timestamp replace date
 2022-10-01 12:50:42 +08:00
Tq
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts (#12126) 
* fix htrace dependency conflicts

* replace htrace-core with hbase-noop-htrace
 2022-09-26 13:49:08 +08:00
Yann Ann
289e1ecdc6
[Bug] [Common] update IPv4 Pattern (#11762) 
* [Bug] [Common] IPv4 Pattern is wrong. #11758
 2022-09-26 09:10:22 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
JieguangZhou
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use (#12071) 
optimize code
 2022-09-21 10:45:05 +08:00
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center (#11708) 
* Add support for Alibaba Cloud OSS as storage of resource center

* Add dependency licenses

* Refactor code, add unit tests and related docs
 2022-09-20 22:44:06 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task (#11670) 
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
 2022-09-17 13:40:56 +08:00
Wenjun Ruan
d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
jackfanwan
71b4087421
Example Change the regular expression of the data source password (#11459) 2022-09-16 13:34:22 +08:00
Wenjun Ruan
9813ded981
Add debug logs for properties read from common.properties (#11910) 2022-09-15 10:53:19 +08:00
LiuBodong
3ca9680b20
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly (#11870) 2022-09-11 19:11:03 +08:00
Kengo Seki
3664d85143
[Improve] Upgrade Hadoop to 3.2.4 (#11832) 2022-09-09 10:47:03 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model (#11767) 2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
xiangzihao
ebcffb04aa
fix hdfs defaultFs not working (#11823) 2022-09-07 16:40:43 +08:00
Kengo Seki
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810) 
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
 2022-09-07 13:48:09 +08:00
Tq
b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav (#11727) 
* feature: add task type list and enable user to add task type to fav

* make DependentExecute.java variable protected

* add standalone server fileSet
 2022-09-02 11:40:23 +08:00
Wenjun Ruan
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize (#11702) 
* Refactor heart beat task, use json to serialize/deserialize
 2022-08-31 16:20:23 +08:00
Wenjun Ruan
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701) 
* Fix kill job may cause worker oom
 2022-08-31 15:25:01 +08:00
Wenjun Ruan
c110f4acae
Refresh server startup time when recovery from waiting (#11595) 2022-08-22 22:36:49 +08:00
kezhenxu94
13e4720aa2
Upgrade spring boot and swagger (#11574) 2022-08-22 10:27:40 +08:00
juzimao
3b72c6efe7
[Improvement]Add a stop function when you are ready to do pause operation (#11543) 
* add can stop for execute state machine

* add execute type check for pause operation
 2022-08-18 21:32:53 +08:00
Wenjun Ruan
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost (#11368) 
* Add waiting strategy to support master/worker can recover from registry lost

* throw exception when zookeeper registry start failed due to interrupted
 2022-08-13 09:52:03 +08:00
fuchanghai
6e2bed4d6b
[fix-11378] fix API-Server release response message typo (#11400) 
* [fix-11295] remove duplicate classes

* [fix-11378] fix response message error

Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
 2022-08-11 09:21:32 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task (#11350) 
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
 2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
fuchanghai
05932bcf55
[fix-11295] remove duplicate classes (#11355) 2022-08-10 10:20:16 +08:00
zhuxt2015
c34503e3f6
[fix] Process definition length too long when copy or import (#10621) 2022-08-04 15:04:14 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db (#11221) 2022-08-01 13:04:27 +08:00
Sheldon
efd68f4805
[Fix-10665] [S3] Fix s3 download method (#10675) 
when overwrite param is true and dest file is exist, throw exception
make dirs for ancestor directories of the source file path if it is not exists
To describe the error more clearly, change "the file isn`t exists" to specific error message

Co-authored-by: sheldonliu <sheldonliu>
 2022-07-27 21:25:13 +08:00
liyangyang
6f0a609793
[Improvement 11072][common] Added HTTPS request support in kerberos environment (#11127) 
* [Improvement-11072][dolphinscheduler-common] Added HTTPS request support in kerberos environment

* [Improvement-11072][dolphinscheduler-common] supplement

Co-authored-by: liyangyang <liyangyang@bizseer.com>
 2022-07-25 23:21:42 +08:00
liyangyang
f13e7a95ae
[Improvement-11013][dolphinscheduler-common] YarnHAAdminUtils#getActiveRMName function Add HTTPS Hadoop environment support (#11017) 
* [Improvement-11013][dolphinscheduler-common] YarnHAAdminUtils#getActiveRMName function Add HTTPS Hadoop environment support

* [Improvement-11013][dolphinscheduler-common] add test code

* [Improvement-11013][dolphinscheduler-common] code format

* [Improvement-11013][dolphinscheduler-common] add hadooputils test code

* [Improvement-11013][dolphinscheduler-common] remove extra blank lines

* [Improvement-11013][dolphinscheduler-common] adjusts test code

* [Improvement-11013][dolphinscheduler-common] remove useless import

* [Improvement-11013][dolphinscheduler-common] add test code

* [Improvement-11013][dolphinscheduler-common] shortened string

* [Improvement-11013][dolphinscheduler-common]  format test code style

Co-authored-by: liyangyang <liyangyang@bizseer.com>
 2022-07-25 11:07:40 +08:00
旺阳
28b40da1b1
[Improve] Hide Change Pwd Page When Login With LDAP (#11041) 
* hide-change-pwd-page-when-ldap-login

* add unit test
 2022-07-20 18:48:16 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979) 
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed

* Move heartbeat error threshold to applicaiton.yml
 2022-07-15 20:06:53 +08:00
Eric Gao
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#10925) 
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814)
 2022-07-15 09:26:28 +08:00
Jiajie Zhong
0db9bbd538
[fix] delete workflow or task used by other tasks (#10873) 
currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
 2022-07-12 21:22:06 +08:00
陈家名
59cd86157f
[Python] Supports creating or editing resources. (#10823) 2022-07-12 20:44:59 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866) 
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
 2022-07-11 09:57:00 +08:00
pinkhello
030fb89d6e
[Fixed-10833] [Bug] [Quartz] timezone display doesn't match the next_fire_time in ds 3.0.0-beta1 version (#10865) 
* closed [10619]  [Improvement][Master] batch remove TaskInstaceId and workflowInstanceId

* fixed # 10833 timezone display error

* checkstyle
 2022-07-09 22:28:08 +08:00
Wenjun Ruan
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect (#10839) 
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
 2022-07-09 11:54:59 +08:00
qianli2022
1e48bbc5c6
[Feature-10629][Improvement] Support multi cluster environments - namespace use cluster (#10650) 
* server code

* namespace ui

* fix dao test

* fix test

* fix

* docs and image update

Co-authored-by: qianl4 <qianl4@cicso.com>
 2022-07-07 16:05:52 +08:00
Wenjun Ruan
426567348e
Remove quartz in service (#10748) 
* Remove quartz in service
 2022-07-06 15:43:55 +08:00
Wenjun Ruan
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry (#10786) 
* Fix state event handle error will not retry

* Use state event handler to deal with the event
 2022-07-06 14:53:28 +08:00
WangJPLeo
d2fe16d252
[Optimization] Config module resource relation fix. (#10718) 
* Config module resource relation fix.

* Boolean judge fix

* e2e rerun

* e2e rerun

* e2e rerun

* ArrayList fix

* change variable name.

* Add transaction processing on create data resource method
 2022-07-05 15:51:23 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. (#10704) 
* Global parameter and local parameter calculation external expansion.

* k8s task ut fix.

* TimePlaceholderUtils import DateUtils fix

* follow the review comments to fix.

* follow the review comments to fix.

* e2e rerun
 2022-06-30 22:45:25 +08:00
juzimao
e0b0f986f7
[Fix-10641] [Common] Fix the hdfs NPE (#10642) 
Fix the hdfs NPE (#10642)
 2022-06-28 22:51:09 +08:00
Wenjun Ruan
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover (#10631) 
* fix worker failover may lose event
 2022-06-28 16:08:35 +08:00