Commit Graph

82 Commits

Author SHA1 Message Date
Kerwin
116eff6663
[3.1.2-prepare][version] Modify the version to 3.1.2 (#13121) 2022-12-07 17:09:48 +08:00
simsicon
fc67fab006 cherry-pick Upgrade clickhouse jdbc driver #12639 2022-11-16 12:33:39 +08:00
Tq
3efe4bc308 [BUG][BE-12113]fix htrace dependency conflicts (#12126)
* fix htrace dependency conflicts

* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:11 +08:00
Eric Gao
e7b12bf205
[Cherry-Pick][3.1.0-prepare] Cherry-Pick OssOperator for Resource Center to 3.1.0 (#12094)
* Cherry OssOperator for Resource Center to 3.1.0

* Fix doc dead link
2022-09-22 15:00:25 +08:00
kezhenxu94
89b4192d3d Remove htrace dependencies (#11881)
* Remove htrace dependencies

* Update pom.xml
2022-09-19 15:27:41 +08:00
Kengo Seki
216ceea641 [Improve] Upgrade Hadoop to 3.2.4 (#11832) 2022-09-19 15:26:58 +08:00
kezhenxu94
fcc75ef1c6 Bump up dependencies to fix cves (#11765) 2022-09-19 15:10:56 +08:00
Wenjun Ruan
1ea2e848cd
[Feature][Registry] Support etcd as registry (#10981) (#11997)
(cherry picked from commit 2eb8b9f7f0)

Co-authored-by: wjf <jffwang@qq.com>
2022-09-17 09:01:24 +08:00
fuchanghai
554562bfa9
[fix-#11691] Add lz4-java in clickhouse datasource. (#11697) 2022-09-05 09:15:58 +08:00
kezhenxu94
277f137358
Add Kubernetes configmap reload to all components (#11730) 2022-09-02 12:03:19 +08:00
kezhenxu94
17a9dd25fa
Fix vulnerability in LDAP login (#11586) 2022-08-22 13:49:14 +08:00
kezhenxu94
13e4720aa2
Upgrade spring boot and swagger (#11574) 2022-08-22 10:27:40 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map (#11402) 2022-08-16 20:25:17 +08:00
Wenjun Ruan
c7789bf049
Add base loop task execotor and http template parser (#11137)
* Add base loop task execotor and http template parser

* Add JsonPathUtils
2022-07-26 14:51:50 +08:00
JieguangZhou
af7b359333
[Feature][Task Plugin] Add SageMaker Pipeline task plugin for MLOps scenario (#10826) (#10935)
* add sagemaker pipeline task plugin

[DOC] add Sagemaker task plugin document

change license and fix dependencies

* [fix] Optimize the code

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerParameters.java

Optimization parameter judgment

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

* [fix] nips

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-07-21 18:16:58 +08:00
Chris Ho
0cf31232de
[Feature][dependencies] Bump spring-core from 5.3.12 to 5.3.19 (#10857) 2022-07-09 15:19:46 +08:00
PJ Fanning
d044e0479d
issue-10356: upgrade logback to fix cve (#10357) 2022-06-03 19:21:40 +08:00
PJ Fanning
b6350280e6
issues-10354: upgrade commons-io to fix CVE (#10355) 2022-06-03 17:17:52 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
Tq
14144e8d2e
[Bug][dep] Bump httpclient version to 4.5.13 (#9631) 2022-04-27 12:11:11 +08:00
PJ Fanning
f45fe85703
upgrade commons-compress to 1.21 (#9540) 2022-04-20 17:42:14 +08:00
PJ Fanning
7fecb92fc2
use secure version of postgresql (#9573)
* use secure version of postgresql

* Update known-dependencies.txt
2022-04-20 17:14:47 +08:00
Eric Gao
a5bbf7852d
[Feature][Task-Plugin]Add zeppelin task-plugin to support Apache Zeppelin (#9327) 2022-04-17 22:10:10 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
BaoLiang
44bb654c5e
Revert "upgrade mybatis-plus to 3.4.3 (#8515)" (#8659)
This reverts commit ca6d148a1b.
2022-03-02 16:10:39 +08:00
ronyang1985
20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR (#8503)
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-03-01 11:43:37 +08:00
BaoLiang
ca6d148a1b
upgrade mybatis-plus to 3.4.3 (#8515) 2022-02-25 08:31:12 +08:00
Kirs
0ef49b4a98
[Improve]Use hibernate validation framework (#4) (#8467)
* [Improve]Use hibernate validation framework
Improve email verification method
Support domain name verification
* Email is at least a second-level domain name
2022-02-21 16:54:46 +08:00
ronyang1985
83e88c4bc6
[Fix-8389][Common] Fix oshi compatibility issues cause it fail to boot on m1 mac (#8390) 2022-02-21 11:14:39 +08:00
qianli2022
151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 (#8252)
* add db operator for namespace

* add k8s client

* add k8s controller

* add namespace ui

* add licenses

* add kubernetes model known-dependencies

* add new licenses

* fix SonarCloud Code Analysis bug

* remove license for standard protocol

* fix variable MemoryStr to memoryStr

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-02-15 10:21:15 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time (#7054) 2021-12-01 13:55:26 +08:00
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user (#6974)
* cache manager for tenant and user

* add test

* fix checkstyle

* add dependencies

* fix sonar check

* use spring boot cache configuration

* add license

* check style

* check style

* add license

* change proxy name to processor

* add license header

* check style

* add license file

* add license file

* add license

Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-30 08:29:25 +08:00
KyoYang
80dcf9b03e
[Improvement-#5293]upgrade cron-utils version 5.0.5 -> 9.1.3 (#6857)
* fix complement data retry bug; issues: [Bug] [Master] A bug on task retry mechanism #6613

Co-authored-by: yangqiyu <yangqiyu@deepexi.com>
2021-11-18 14:32:32 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage (#6767) 2021-11-10 18:29:17 +08:00
kezhenxu94
05bae0b226
Refactor alert plugins and simplify its usage (#6723) 2021-11-09 18:59:34 +08:00
Jiajie Zhong
dd6ed36f65
Add Python API implementation of workflows-as-code (#6269)
* Init DS python SDK pydolphinscheduler: python code definition

* Doc first

* Add quick start and developer doc

* Java documentation change

* Add LICENSE-py4j.txt

* Add py4j to release-docs/LICENSE

* Move dependency version to parent pom

* Remove outdated code

* Add tenant parameter to tutorial
2021-10-31 20:35:46 +08:00
wind
c8ee2df216
[DS-6483][MasterServer] upgrade the druid and mysql connector version (#6484)
* [DS-6483][MasterServer] upgrade the druid and mysql connector version

* remove mysql connector in known-dependencies.txt

* change mysql connector version to 8.0.16

Co-authored-by: caishunfeng <534328519@qq.com>
2021-10-11 00:40:49 +08:00
百岁
34352c1bf8
Enhance using experience of DataX by introduce TIS (#6229)
* Enhance using experience of DataX by introduce TIS [Feature-5992]

* Enhance using experience of DataX by introduce TIS [Feature-5992]

* fix stylecheck error

* make testCase pass

* make dev testCase pass

* add new java dependency Java-WebSocket

* modfiy TISParameters.java avoid to name confliction

* add InterruptedException checking

* reAdd async-http-client annotation

* in order to staisfy the coverage degree add test which has been removed

* make testCase pass

* make testCase pass

* add jacoco dependency

* make code duplications be more lower

* add Java-WebSocket dependency in root pom

* remove useless code comment

* make tis http apply path and post body configurable ,the params save in config.properites

* Remove the dangerous instance of double-checked locking
2021-09-26 18:40:41 +08:00
Kirs
4715de11fc
[Feature][Plugin]Task (#6105)
* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952)

* [Bug][WorkerServer] SqlTask NullPointerException #5549

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428

* fix the readme typing issue (#5998)

* Fix unchecked type conversions

* Use indentation level reported by checkstyle

* Reorganize CI workflows to fasten the wasted time and resources (#6011)

* Add standalone server module to make it easier to develop (#6022)

* Task node of SWITCH (#5939)

* [Feature-#5273][server-master] Task node of SWITCH (#5922)



Co-authored-by: wangxj <wangxj31>

* remove description of bonecp (#6030)

Co-authored-by: shaojwu <shaojwu@ebay.com>

* [Improvement][Api Module]split alert group list-paging interface (#5941)

* [Improvement][Api Module]split alert group list-paging interface

* [FIX-#6007]Wrong complement date (#6026)

* [FIX-#6007]Wrong complement date

* [style]Wrong complement date

* [Improvement-6024][dist] Remove useless packaging commands (#6029)

·Remove useless packaging commands in dolphinscheduler-bin.xml

This closes #6024

Co-authored-by: mask <liuhu@zhiyoutec.com>

* [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909)

* fix the npe in MasterExec

* fix the compile error

* Add `.asf.yaml` to easily set the GitHub metadata (#6035)

* fix dead server cannot stop (#6046)

* Enhancement Translation (#6042)

* replaced Loading... with i18n

* modified Edit zh_CN translation

* Delete zh_CN.js

Co-authored-by: David <dailidong66@gmail.com>

* fix bug #6053 zh_CN.js is lost

* [Feature][Plugin]Task

* fix license head

* fix code style

* fix code style

* fix build err

Co-authored-by: didiaode18 <563646039@qq.com>
Co-authored-by: Roy <yongjuncao1213@gmail.com>
Co-authored-by: lyxell <alyxell@kth.se>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: myangle1120 <942542838@qq.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: shaojwu <shaojwu@ebay.com>
Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com>
Co-authored-by: linquan <1175687813@qq.com>
Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com>
Co-authored-by: mask <liuhu@zhiyoutec.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com>
Co-authored-by: David <dailidong66@gmail.com>
Co-authored-by: lenboo <baoliang.leon@gmail.com>
2021-09-06 09:38:29 +08:00
百岁
b04bceb03c
[Feature-5992] Enhance using experience of DataX in DS by integeration with TIS (#6015) 2021-08-27 21:07:51 +08:00
ruanwenjun
25727d2d95
[Improvement][Alert] Replace HSSFWorkbook (#5400)
* [Improvement][Alert] Replace HSSFWorkbook

* add license
2021-05-10 09:57:49 +08:00
Shiwen Cheng
2e1d6975fd
[Improvement][Build] Optimize long release name and remove unused nginx release and upgrade spring boot version (#5308)
* [Build] Optimize long release name and remove unused nginx release

* [Build] Upgrade spring boot version

* [Improvement] Improve code style
2021-04-25 09:55:38 +08:00
didiaode18
f82da57277
There is a vulnerability in postgresql 42.1.4 ,upgrade recommended #5292 (#5318)
* There is a vulnerability in postgresql 42.1.4 ,upgrade recommended #5292
2021-04-20 15:04:29 +08:00
Shiwen Cheng
e92e29ef9a
[Fix-5309][Common] Fix the wrong memory (#5312)
* [Fix-5309][Common] Fix the wrong memory

* [Fix-5309][Common] Fix unit test and remove unused code
2021-04-19 10:24:54 +08:00
CalvinKirs
b4c7e35340 Merge branch 'dev' into rpc
# Conflicts:
#	tools/dependencies/known-dependencies.txt
2021-02-26 18:10:49 +08:00
Kirs
a9e1d316f5
[Improvement][SPI]Migrate related classes loaded by SPI classes to public modules (#4586)
* Migrate related classes loaded by SPI classes to public modules

* Migrate related classes loaded by SPI classes to public modules

* remove guava

* update guava

* update guava

* move to common model

* code style

* code style

* resolved conflicting file

* remove ignore test

* add jar

* fix ut error

* fix test

* fix error

* test

* exclude asm、resolver、aether-api

* code style

* remove default plugin jar config

* add LICENSE file

* add version

* add version

* remove ignore ut

* fix ut error

* add ut

* code style

* add license
2021-02-25 11:39:46 +08:00
Kirs
c9972936d6
[SPI][License]Add missing license (#4749) 2021-02-09 19:48:54 +08:00