Commit Graph

608 Commits

Author SHA1 Message Date
Kirs
a288e6c160
[Plugin][Task]Task Spi (#6118) 2021-09-07 17:05:02 +08:00
Kerwin
36d60a564a
[Feature][Task]Task plugin spi (#6116) 2021-09-07 15:04:18 +08:00
Kirs
4715de11fc
[Feature][Plugin]Task (#6105)
* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952)

* [Bug][WorkerServer] SqlTask NullPointerException #5549

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428

* fix the readme typing issue (#5998)

* Fix unchecked type conversions

* Use indentation level reported by checkstyle

* Reorganize CI workflows to fasten the wasted time and resources (#6011)

* Add standalone server module to make it easier to develop (#6022)

* Task node of SWITCH (#5939)

* [Feature-#5273][server-master] Task node of SWITCH (#5922)



Co-authored-by: wangxj <wangxj31>

* remove description of bonecp (#6030)

Co-authored-by: shaojwu <shaojwu@ebay.com>

* [Improvement][Api Module]split alert group list-paging interface (#5941)

* [Improvement][Api Module]split alert group list-paging interface

* [FIX-#6007]Wrong complement date (#6026)

* [FIX-#6007]Wrong complement date

* [style]Wrong complement date

* [Improvement-6024][dist] Remove useless packaging commands (#6029)

·Remove useless packaging commands in dolphinscheduler-bin.xml

This closes #6024

Co-authored-by: mask <liuhu@zhiyoutec.com>

* [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909)

* fix the npe in MasterExec

* fix the compile error

* Add `.asf.yaml` to easily set the GitHub metadata (#6035)

* fix dead server cannot stop (#6046)

* Enhancement Translation (#6042)

* replaced Loading... with i18n

* modified Edit zh_CN translation

* Delete zh_CN.js

Co-authored-by: David <dailidong66@gmail.com>

* fix bug #6053 zh_CN.js is lost

* [Feature][Plugin]Task

* fix license head

* fix code style

* fix code style

* fix build err

Co-authored-by: didiaode18 <563646039@qq.com>
Co-authored-by: Roy <yongjuncao1213@gmail.com>
Co-authored-by: lyxell <alyxell@kth.se>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: myangle1120 <942542838@qq.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: shaojwu <shaojwu@ebay.com>
Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com>
Co-authored-by: linquan <1175687813@qq.com>
Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com>
Co-authored-by: mask <liuhu@zhiyoutec.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com>
Co-authored-by: David <dailidong66@gmail.com>
Co-authored-by: lenboo <baoliang.leon@gmail.com>
2021-09-06 09:38:29 +08:00
Kerwin
5a12e00cbe
[Feature][Task]Task plugin spi (#6084) 2021-09-02 10:08:38 +08:00
Kerwin
22c5292b93
[Feature][SPI] Task (#6044) 2021-08-27 15:02:24 +08:00
Kerwin
79952d926f
[Feature][SPI] Task (#5996) 2021-08-18 10:32:19 +08:00
Tandoy
846d408e66
[Improvement][Master]fix typo (#5934)
·fix typo in MasterBaseTaskExecThread
2021-08-02 22:34:46 +08:00
kyoty
bca92157a0
[Fix-5808][Server] When we try to transfer data using datax between different types of data sources, the worker will exit with ClassCastException (#5809)
* bug fix

* fix bug

* simplify the code format

* add a new parameter to make it easier to understand.
2021-07-27 15:46:15 +08:00
深刻
4c0993cdf4
[Fix-5781][UT] Fix test coverage in sonar (#5817)
* build(UT): make jacoco running in offline-instrumentation

issue: #5781

* build(UT): remove the jacoco agent dependency in microbench

issue: #5781
2021-07-26 22:04:36 +08:00
Hua Jiang
4a68bfbe1c
[Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867)
* provide two system parameters to support the rest of type of tasks

* provide two system parameters to support the rest of type of tasks

* improve test conversion
2021-07-21 18:36:20 +08:00
wen-hemin
9ae2266cd4
fix: Remove unused class (#5833)
Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-07-17 16:23:39 +08:00
wen-hemin
596821a5b6
[Improvement][Api Module] refactor registry client, remove spring annotation (#5814)
* fix: refactor registry client, remove spring annotation

* fix UT

* fix UT

* fix checkstyle

* fix UT

* fix UT

* fix UT

* fix: Rename RegistryCenterUtils method name

Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-07-14 11:54:41 +08:00
wen-hemin
e78344df2e
fix: Remove duplicate "registryClient.close" method calls (#5805)
Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-07-13 01:14:07 +08:00
傅双波
1e3ff41928
fix (#5803)
Co-authored-by: shuangbofu <fusb@tuya.com>
2021-07-12 18:04:17 +08:00
kyoty
7bffe0ac85
[Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. (#5796)
* improve timestamp format

make the startime in the log of httptask to be easier to read.


* fix bad code smell and update the note.
2021-07-12 14:31:48 +08:00
Wenjun Ruan
30af55b82a
[Improvement][Worker] Task log may be lost #5775 (#5783) 2021-07-10 01:14:59 +08:00
kyoty
626c47399a
[FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE 2021-07-10 01:08:16 +08:00
Hua Jiang
9fd5145b66
[Improvement-5773][server] need to support two parameters related to task (#5774)
* add some new parameter for task

* restore official properties

* improve imports

* modify a variable's name

Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
2021-07-09 17:00:32 +08:00
Tanvi Moharir
ab527a5e5a
[Improvement#5741][Worker] Improve task process status log (#5776) 2021-07-09 00:28:00 +08:00
wangxj3
7a18adae52
[FIX-#5721][master-server] Global params parameter missing (#5757)
Co-authored-by: wangxj <wangxj31>
2021-07-07 17:34:38 +08:00
wangxj3
6df58e8d78
[Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)
* add globalParams new plan with varPool

* add unit test

* add python task varPoolParams


Co-authored-by: wangxj <wangxj31>
2021-06-25 11:37:23 +08:00
ji04xiaogang
3b80760c42
Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)
Co-authored-by: shenglm <shenglm840722@126.com>
2021-06-25 09:13:26 +08:00
Kirs
f508788689
[BUG-#5678][Registry]fix registry init node miss (#5686) 2021-06-24 15:23:29 +08:00
Shiwen Cheng
93f1d4df7d
[Improvement-5405][Docker/K8s] Adapt to registry SPI (#5681) 2021-06-23 11:30:15 +08:00
Squidyu
664206cb81
wrong word: 'WAITTING' ->'WAITING' (#5682)
* Add:Name verification remove the first and last spaces.

* Update: wrong word: 'WAITTING' ->'WAITING'

Co-authored-by: Squid <2824638304@qq.com>
2021-06-22 21:56:21 +08:00
kyoty
75813e7866
[bug fix] Some password attributes can be viewd directly in the log (#5638)
* fix the password display incorrectly

* update

* update

* fic code style

fic code style

* Update SensitiveDataConverterTest.java

* Update SensitiveDataConverter.java

* Update SensitiveDataConverter.java

* Update SensitiveDataConverterTest.java

* Update pom.xml
2021-06-18 19:10:57 +08:00
justoy
0d7c32a1e8
Update TaskInstanceCacheManagerImpl.java (#5656) 2021-06-17 18:45:34 +08:00
zhuangchong
3026f04d85
[Improvement-5452][Task] ds flink task support submit a PyFlink job via the CLI (#5453)
* flink task support submit a PyFlink job via the CLI.

* optimize attribute name.

* Modify pyflink parameter judgment logic
2021-06-17 15:19:25 +08:00
LOUKHNATI Mohamed Khalil
86ffc1f3f2
[Bug][WorkerServer] SqlTask NullPointerException (#5556)
* [Bug][WorkerServer] add null checks to sqlParamsMap and sqlParamsMap.get(i), Add test to verify it

* Change Test Name

* Fix Code style issues, Modify checking null for Sql params in sqlParamsMap with clearer logging.
2021-06-16 18:21:41 +08:00
kyoty
67711442d5
[Improvement][SQL] Query return number In SQL should be configurable (#5632)
* [Improvement][SQL] Query return number should be configurable #5580

* Update SqlTaskTest.java

* Update sql.vue
2021-06-16 09:40:21 +08:00
zhuangchong
e2d6265e26
[Improvement][registry-plugin] Optimize registry plugin loading and initial installation #5635 2021-06-15 20:51:52 +08:00
Shiwen Cheng
8bf042ae6e
[Fix-5596][Python] Fix conflict between python_home and datax_home configuration in dolphinscheduler_env.sh (#5612) 2021-06-12 01:23:18 +08:00
blackberrier
79eb2e85d7
[Improvement-5539][Master] Check status of taskInstance from cache (#5572)
* improvement:check status of taskInstance from cache

* issue5572 use timer instead of while&sleep; consider concurrent modification

* use computeifpresent instead of lock

* simplify getByTaskInstanceId function

* add ut for TaskInstanceCacheManagerImpl; fix bug in TaskInstanceCacheManagerImpl

* add Apache license header;add test class in root pom
2021-06-10 09:39:12 +08:00
Kirs
e2243d63be
[Feature-#3961][Registry]Registry-SPI (#5562)
[Feature#3961]Registry SPI

All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
2021-06-09 15:21:02 +08:00
Chouc
75be09735a
[DS-5559][fix][Master Server] Master Server was shutdown but the process still in system (#5588)
* Close spring context to destory beans that has running thread
2021-06-04 13:27:18 +08:00
Shiwen Cheng
a5a0c7c5f8
[Fix-5570][Worker] Fix worker.groups in worker.properties is still commented and improve install scripts and configurations (#5571) 2021-06-02 11:21:01 +08:00
geosmart
56273319a2
[Bug][Master] remove check with executePath when kill yarn job (#5552)
* [Bug][Master] remove check with executePath when kill yarn job

* [Bug][Master] init execute path if not exist

* [Bug][Master] move createWorkDirIfAbsent out side of executePath calculate  part

Co-authored-by: wanggang <wanggy01@servyou.com.cn>
2021-05-29 12:14:46 +08:00
ruanwenjun
842c5400e6
[Improvement][Server] Must restart master if Zk reconnect (#5210) (#5211) 2021-05-25 05:03:29 +08:00
ruanwenjun
9d0c816cee
[Bug][worker] task throw ConcurrentModifiedException #5528 (#5530)
* [Bug][worker] task throw ConcurrentModifiedException #5528

* fix code smell
2021-05-25 04:53:56 +08:00
Shiwen Cheng
b537f4eb26
[Improvement][Docker/K8s] Support more configurations and service accesses, reduce the duplication configurations, improve image for python and the compatibility of python home (#5393)
* [Improvement][Docker] Support more configurations and improve image for python

* [Improvement][K8s] Support more configurations, more service accesses, and reduce the duplication configurations

* [Improvement][Python] Improve the compatibility of python home

* [Improvement][Install] Fix install config and install.sh

* [Improvement][Install] Fix workflow ut

* [Improvement][Docker] Optimize docker-swarm/check script

* [Improvement][DB] Update default username and password of database

* Update comments in master/worker.properties

* Specify version for all images

* [Improvement][Docker] Optimize PS1 and WORKDIR

* [Improvement][SQL] Reduce heap size to 64m in create-dolphinscheduler.sh and upgrade-dolphinscheduler.sh

* [Fix-5431][K8s] Fix master and worker cannot get the right address with custom DNS
2021-05-24 00:03:35 +08:00
zhuangchong
1b7dcea610
[Improvement-5440][Maven] Project module maven dependency optimization (#5441)
* Optimize dependencies between modules.

* update module-alert dependency.

* add repository in the pom xml.

* update Central Repository in the pom.xml.
2021-05-19 10:05:00 +08:00
Yao WANG
bc22ae7c91
[Improvement][Task] Remove deprecated TaskRecordDao and simply the after() method in the AbstractTask class (#5492)
* remove the depreated data quality module

* fix lint

Co-authored-by: Yao <ywang46@paypal.com>
2021-05-18 17:00:03 +08:00
baoliang
dd80f98196 update sleep exception 2021-05-12 16:50:39 +08:00
baoliang
7aa8b518ac Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 13:55:27 +08:00
baoliang
2dd0d936fb Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 11:45:40 +08:00
baoliang
156dadac69 Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 10:08:28 +08:00
JinyLeeChina
515fa4912e
[Feature][JsonSplit] Fix ProcessService ut (#5435)
* Fix ProcessService ut

* codeStyle

* Fix AlertPlugin ut

* Fix AlertPlugin ut

* Fix server module ut

* Fix server module ut

* Fix api ut

* codeStyle

Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-05-10 23:17:56 +08:00
zhuangchong
9c77faa8ac
[Improvement-5058][Master] The master-server creates the DB data source twice (#5139)
* reduce the number of data sources created twice by master server.

* update ProcessAlertManagerTest test class to increase coverage.

* add DolphinSchedulerPluginLoaderTest test ignore.

Co-authored-by: zhuang chong
2021-05-10 09:56:17 +08:00
baoliang
c850b85484 Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/procedure/ProcedureTask.java
2021-05-07 01:38:53 +08:00
baoliang
b594afc94c Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/procedure/ProcedureTask.java
2021-05-07 01:38:31 +08:00