Commit Graph

331 Commits

Author SHA1 Message Date
Kerwin
36d60a564a
[Feature][Task]Task plugin spi (#6116) 2021-09-07 15:04:18 +08:00
Kirs
4715de11fc
[Feature][Plugin]Task (#6105)
* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952)

* [Bug][WorkerServer] SqlTask NullPointerException #5549

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428

* fix the readme typing issue (#5998)

* Fix unchecked type conversions

* Use indentation level reported by checkstyle

* Reorganize CI workflows to fasten the wasted time and resources (#6011)

* Add standalone server module to make it easier to develop (#6022)

* Task node of SWITCH (#5939)

* [Feature-#5273][server-master] Task node of SWITCH (#5922)



Co-authored-by: wangxj <wangxj31>

* remove description of bonecp (#6030)

Co-authored-by: shaojwu <shaojwu@ebay.com>

* [Improvement][Api Module]split alert group list-paging interface (#5941)

* [Improvement][Api Module]split alert group list-paging interface

* [FIX-#6007]Wrong complement date (#6026)

* [FIX-#6007]Wrong complement date

* [style]Wrong complement date

* [Improvement-6024][dist] Remove useless packaging commands (#6029)

·Remove useless packaging commands in dolphinscheduler-bin.xml

This closes #6024

Co-authored-by: mask <liuhu@zhiyoutec.com>

* [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909)

* fix the npe in MasterExec

* fix the compile error

* Add `.asf.yaml` to easily set the GitHub metadata (#6035)

* fix dead server cannot stop (#6046)

* Enhancement Translation (#6042)

* replaced Loading... with i18n

* modified Edit zh_CN translation

* Delete zh_CN.js

Co-authored-by: David <dailidong66@gmail.com>

* fix bug #6053 zh_CN.js is lost

* [Feature][Plugin]Task

* fix license head

* fix code style

* fix code style

* fix build err

Co-authored-by: didiaode18 <563646039@qq.com>
Co-authored-by: Roy <yongjuncao1213@gmail.com>
Co-authored-by: lyxell <alyxell@kth.se>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: myangle1120 <942542838@qq.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: shaojwu <shaojwu@ebay.com>
Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com>
Co-authored-by: linquan <1175687813@qq.com>
Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com>
Co-authored-by: mask <liuhu@zhiyoutec.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com>
Co-authored-by: David <dailidong66@gmail.com>
Co-authored-by: lenboo <baoliang.leon@gmail.com>
2021-09-06 09:38:29 +08:00
百岁
b04bceb03c
[Feature-5992] Enhance using experience of DataX in DS by integeration with TIS (#6015) 2021-08-27 21:07:51 +08:00
Kerwin
22c5292b93
[Feature][SPI] Task (#6044) 2021-08-27 15:02:24 +08:00
深刻
4c0993cdf4
[Fix-5781][UT] Fix test coverage in sonar (#5817)
* build(UT): make jacoco running in offline-instrumentation

issue: #5781

* build(UT): remove the jacoco agent dependency in microbench

issue: #5781
2021-07-26 22:04:36 +08:00
wangxj3
6df58e8d78
[Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)
* add globalParams new plan with varPool

* add unit test

* add python task varPoolParams


Co-authored-by: wangxj <wangxj31>
2021-06-25 11:37:23 +08:00
kyoty
75813e7866
[bug fix] Some password attributes can be viewd directly in the log (#5638)
* fix the password display incorrectly

* update

* update

* fic code style

fic code style

* Update SensitiveDataConverterTest.java

* Update SensitiveDataConverter.java

* Update SensitiveDataConverter.java

* Update SensitiveDataConverterTest.java

* Update pom.xml
2021-06-18 19:10:57 +08:00
blackberrier
79eb2e85d7
[Improvement-5539][Master] Check status of taskInstance from cache (#5572)
* improvement:check status of taskInstance from cache

* issue5572 use timer instead of while&sleep; consider concurrent modification

* use computeifpresent instead of lock

* simplify getByTaskInstanceId function

* add ut for TaskInstanceCacheManagerImpl; fix bug in TaskInstanceCacheManagerImpl

* add Apache license header;add test class in root pom
2021-06-10 09:39:12 +08:00
Kirs
e2243d63be
[Feature-#3961][Registry]Registry-SPI (#5562)
[Feature#3961]Registry SPI

All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
2021-06-09 15:21:02 +08:00
Shiwen Cheng
b537f4eb26
[Improvement][Docker/K8s] Support more configurations and service accesses, reduce the duplication configurations, improve image for python and the compatibility of python home (#5393)
* [Improvement][Docker] Support more configurations and improve image for python

* [Improvement][K8s] Support more configurations, more service accesses, and reduce the duplication configurations

* [Improvement][Python] Improve the compatibility of python home

* [Improvement][Install] Fix install config and install.sh

* [Improvement][Install] Fix workflow ut

* [Improvement][Docker] Optimize docker-swarm/check script

* [Improvement][DB] Update default username and password of database

* Update comments in master/worker.properties

* Specify version for all images

* [Improvement][Docker] Optimize PS1 and WORKDIR

* [Improvement][SQL] Reduce heap size to 64m in create-dolphinscheduler.sh and upgrade-dolphinscheduler.sh

* [Fix-5431][K8s] Fix master and worker cannot get the right address with custom DNS
2021-05-24 00:03:35 +08:00
QuakeWang
46660b58ed
[Test] Add SparkParameters test (#5478)
* Create SparkparametersTest.java

Add a test case

* Update and rename SparkparametersTest.java to SparkParametersTest.java

* Update pom.xml

add SparkParametersTest.java in pom file
2021-05-22 23:56:39 +08:00
zhuangchong
1b7dcea610
[Improvement-5440][Maven] Project module maven dependency optimization (#5441)
* Optimize dependencies between modules.

* update module-alert dependency.

* add repository in the pom xml.

* update Central Repository in the pom.xml.
2021-05-19 10:05:00 +08:00
baoliang
156dadac69 Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 10:08:28 +08:00
ruanwenjun
25727d2d95
[Improvement][Alert] Replace HSSFWorkbook (#5400)
* [Improvement][Alert] Replace HSSFWorkbook

* add license
2021-05-10 09:57:49 +08:00
zhuangchong
9c77faa8ac
[Improvement-5058][Master] The master-server creates the DB data source twice (#5139)
* reduce the number of data sources created twice by master server.

* update ProcessAlertManagerTest test class to increase coverage.

* add DolphinSchedulerPluginLoaderTest test ignore.

Co-authored-by: zhuang chong
2021-05-10 09:56:17 +08:00
QuakeWang
f88bbbd637
[Improvement] Create RollViewLogRequestCommandTest.java (#5417)
* Create RollViewLogRequestCommandTest.java

Add a test case about RollViewLogRequestCommand

* Update pom.xml

add the test files in the parent pom
2021-05-07 20:58:23 +08:00
baoliang
b594afc94c Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/procedure/ProcedureTask.java
2021-05-07 01:38:31 +08:00
ruanwenjun
ee95d07a90
[Feature][Alert plugin] Support slack alert (#5382) (#5384)
* [Feature][Alert plugin] Support slack alert (#5382)

* fix code smell

* add markdown table
2021-04-30 10:11:32 +08:00
zhuangchong
88a07f7b55
[Feature-4093][server] Support for stored procedures and stored function calls and data source supports DB2 (#4094)
* Support for stored procedures and stored function calls and data source supports DB2.


Co-authored-by: zhuangchong <zhuangchong8@163.com>
2021-04-29 18:29:44 +08:00
baoliang
746d9eca1d Merge remote-tracking branch 'upstream/dev' into json_split
# Conflicts:
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionVersionServiceImpl.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectServiceImpl.java
#	dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionControllerTest.java
#	dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProcessDefinitionVersionServiceTest.java
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/model/TaskNode.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/processdure/ProcedureTask.java
#	dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
2021-04-26 14:14:05 +08:00
Shiwen Cheng
6b8461e901
Revert "[Improvement][DAO] CreateDolphinScheduler (#5357) (#5358)" (#5383)
This reverts commit b108ac43b8.
2021-04-25 23:29:00 +08:00
ruanwenjun
e6d8da484f
[Improvement][API] Create Datasource #5083 (#5116)
* [Improvement][API] Create Datasource #5083

  * Refact create/ update datasource API

* fix Code Smells

* fix code smell

* resolve code smell

* use ConnectionParam to replace BaseDatasource

* solve license

* add ut

* code check

* add ut

* fix ut coverage

* fix ut
2021-04-25 13:51:59 +08:00
Shiwen Cheng
2e1d6975fd
[Improvement][Build] Optimize long release name and remove unused nginx release and upgrade spring boot version (#5308)
* [Build] Optimize long release name and remove unused nginx release

* [Build] Upgrade spring boot version

* [Improvement] Improve code style
2021-04-25 09:55:38 +08:00
ruanwenjun
b108ac43b8
[Improvement][DAO] CreateDolphinScheduler (#5357) (#5358)
* [Improvement][DAO] Create CreateDolphinScheduler (#5357)

* add ut

* fix code smell
2021-04-23 22:03:47 +08:00
Wenjun
abdd2337b1
[Bug][worker] Fix sql task query result is empty (#5322) (#5324)
* [Bug][worker] Fix sql task query result is empty (#5322)

* add ut

* remove duplicate

* add ut
2021-04-22 12:02:32 +08:00
didiaode18
f82da57277
There is a vulnerability in postgresql 42.1.4 ,upgrade recommended #5292 (#5318)
* There is a vulnerability in postgresql 42.1.4 ,upgrade recommended #5292
2021-04-20 15:04:29 +08:00
zhuangchong
fe144e46a3
[Improvement-4898][API] the paging query interface validates page params (#4899)
* fix api list-paging method check page params.

* update process definition version class check page params.

* add ProcessInstanceControllerTest in the pom xml file.

* solve sonar exception.

* add project controller test class.

* update create project return result

* update project controller test class code style.

* update sonar problem.

* update QueueController test class.
2021-04-19 11:01:52 +08:00
Shiwen Cheng
e92e29ef9a
[Fix-5309][Common] Fix the wrong memory (#5312)
* [Fix-5309][Common] Fix the wrong memory

* [Fix-5309][Common] Fix unit test and remove unused code
2021-04-19 10:24:54 +08:00
Shiwen Cheng
94a08c8f32
[Improvement] Remove incubating (#5305) 2021-04-16 22:56:51 +08:00
wenjun
7843ed4087
[Improvement][ApiServer] Traffic limit #5235 (#5307)
* [Improvement][ApiServer] Traffic limit #5235

* code style check
2021-04-16 22:41:02 +08:00
Kirs
ac15f45b0b
Update the main repository link (#5286) 2021-04-16 10:56:48 +08:00
Shiwen Cheng
e39409a412
[Improvement-5275][Server] Fix wrong excludeFilters and move zk/ZKMasterClient and registry/ServerNodeManager into master (#5278)
* [Improvement-5275][Server] Move zk/ZKMasterClient and registry/ServerNodeManager into master

* [Improvement-5275][Server] Fix code smells

* [Improvement-5275][Server] Add ZKMasterClientTest
2021-04-16 00:17:21 +08:00
wenjun
f3dfaacca2
[Improvement][Service] Optimize query log (#5205)
* [Improvement][Service] Optimize query log

* remove IPUtils
2021-04-15 20:25:01 +08:00
Shiwen Cheng
1c1f6663a6
[Improvement][WorkerGroup] Optimize and reduce database and zookeeper queries and replace worker address input with multiple-select (#5182)
* [Improvement][Server] Optimize and reduce database and ZooKeeper query

* [Improvement][WorkerGroup] Replace worker address input with multiple-select

* [Improvement][WorkerGroup] Fix unit test

* [Improvement][API&UI] Optimize long host to short host and long host display in k8s
2021-04-14 21:39:39 +08:00
wenjun
cc7a4446f5
[Improvement][Task] DataX Node doesn't Support Clickhouse as datasource (#5180) (#5243)
* [Improvement][Module Name] DataX Node doesn't Support Clickhouse as the datasource (#5180)

* add ut

* add ut

* resolve code smell
2021-04-14 11:31:40 +08:00
lenboo
c8ac6f99fe Merge remote-tracking branch 'upstream/dev' into spilit
# Conflicts:
#	dolphinscheduler-common/src/main/resources/common.properties
2021-03-30 11:31:42 +08:00
wenjun
a8b47e1d4f
[Bug][Master] Typo in MasterConfig #5141 (#5142) 2021-03-26 09:47:21 +08:00
lenboo
010b49c933 merge from dev 2021-03-22 15:18:16 +08:00
Shiwen Cheng
8ac72e80e6
[Feature-5087][SqlTask] Add a switch to send mail and print head logs in SqlTask (#5088)
* [Feature-5087][SqlTask] Add the switch to send mail and print head logs in SqlTask

* [Feature-5087][SqlTask] Add SqlParametersTest
2021-03-19 15:07:39 +08:00
Shiwen Cheng
5856a12855
[Improvement-4984][Worker] Refactor and Improve worker load balance (#4996)
* [Improvement][Remote] Improve unit tests

* [Improvement][Worker] Improve worker load balance

* [Improvement][Worker] Fix code smells

* [Improvement][Worker] Rename weight to hostWeight
2021-03-18 10:03:58 +08:00
lenboo
cf63fc1fee Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-16 15:50:06 +08:00
Kirs
6bf5031f45
[FIX] fix MySQL datasource jdbc connect parameters (#4851)
* [FIX-CVE-2020-11974] fix MySQLDataSource Security

* remove useless code

* add ut

* A more reasonable way

* code style

* fix test error

* code style

* update exception msg
2021-03-16 14:40:07 +08:00
lenboo
a574d2e38c Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-12 10:44:43 +08:00
wenjun
ea6b1de120
[Improvement][worker] Optimize KillYarnJob (4939) (#4943) 2021-03-11 22:13:44 +08:00
lenboo
6ef54ebe9b Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-11 17:01:29 +08:00
Shiwen Cheng
beb5adb985
[Feature][MR] Support mapreduce name (#5029) 2021-03-11 12:35:08 +08:00
Simon
bd4b2f85a9
[Feature][jsonsplit]add ProcessTaskRelationServiceImplTest/TaskDefinitionServiceImplTest (#4949)
* Modify Project and ProjectUser Mapper

* Modify Project and ProjectUser Mapper

* project_code is bigint(20)

* modify ERROR name

* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation

* modify import/export processdefinition, add genProcessData

* fix ut and bug

* code style

* repalce project_id with code

* conflicts solve

* conflicts solve

* conflicts solve

* bugfix

* modify listResources mothod and remove getResourceIds mothod

* 1

* conflicts solve

* modify listResources mothod and remove getResourceIds mothod

* modify listResources mothod and remove getResourceIds mothod

* replace processDefinitionVersion with processDefinitionLog

* codestyle

* codestyle

* add ProcessTaskRelationServiceImplTest/TaskDefinitionServiceImplTest

* conflicts solve

* conflicts solve

* conflicts solve

* add ProcessTaskRelationServiceImplTest/TaskDefinitionServiceImplTest

* codestyle
2021-03-06 10:56:49 +08:00
zhuangchong
5bfe3fca67
[Fix][SqlTask] fix hive sqltask invalid connection parameters. (#4917)
* fix hive sqltask invalid connection parameters.

* add setConnParams test method.

* update base datasource test class code style.

* add BaseDataSourceTest in the pom xml.

* update hive conf list jdbc url stitching.

* update hive datasource test class.
2021-03-03 09:15:37 +08:00
lgcareer
43ba29a2dd
[Improvement-4624] When the server exist in the dead server list of zk,need stop service byself (#4626)
* [Improvement-4624] When the server exist in the dead server list of zk,need stop service byself

* [Improvement-4624]fix check style and add MaterRegistryTest

* [Improvement-4624]fix check style and add ZookeeperRegistryCenterTest

* [Improvement-4624]fix check style and add ZookeeperRegistryCenterTest

* [Improvement-4624]add RegisterOperatorTest

* [Improvement-4624]update RegisterOperatorTest

* [Improvement-4624]resolve code smell

* [Improvement-4624]revert LICENSE-@form-create-element-ui
2021-03-02 14:05:00 +08:00
CalvinKirs
b4c7e35340 Merge branch 'dev' into rpc
# Conflicts:
#	tools/dependencies/known-dependencies.txt
2021-02-26 18:10:49 +08:00