Commit Graph

42 Commits

Author SHA1 Message Date
kezhenxu94
eb7b48596c cherry-pick Reorganize some classes in common module, remove duplicate classes
#12321
2022-10-27 13:53:41 +08:00
Kerwin
344293102d
[3.1.1-prepare][cherry-pick][Improvement] Add remote task model #11767 (#12541) 2022-10-26 15:44:13 +08:00
caishunfeng
7e39396a76 [maven-release-plugin] prepare for next development iteration 2022-09-26 21:50:03 +08:00
caishunfeng
ae33ba5947 [maven-release-plugin] prepare release 3.1.0 2022-09-26 21:49:27 +08:00
Tq
3efe4bc308 [BUG][BE-12113]fix htrace dependency conflicts (#12126)
* fix htrace dependency conflicts

* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:11 +08:00
kezhenxu94
775ef98b64 Add validations of possible malicious keys (#11966) 2022-09-19 16:32:15 +08:00
Kengo Seki
52b79b017e [Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810)
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-19 14:48:17 +08:00
caishunfeng
113dc4b5c5
[Feature-10273][Datasource] Add dataSource for AWS Athena (#11157) (#12039)
Co-authored-by: ZhaoGuodong <guodongym@163.com>
2022-09-19 13:45:19 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory (#11614)
* Add spi priority factory

* Add doc

* Add override log

* Use lombok

* Add comment
2022-08-24 19:51:47 +08:00
Wenjun Ruan
71cf7e1baa
[Fix-10918] Close datasource when expire from guava cache (#11120)
* Close datasource when expire from guava cache
* Remove duplicate datasource in HiveDataSourceClient
2022-07-23 20:43:37 +08:00
Wenjun Ruan
cd62de968d
Add JDK11 on unit-test (#10364) 2022-06-28 14:21:51 +08:00
wjf
9345c16f3f
[Improve]Remove the code logger.isDebugEnabled() #5918 (#10542) 2022-06-22 17:31:20 +08:00
Wenjun Ruan
0bf4043d5f
Move datasource params to datasouce plugin module (#10353)
* Move datasource params to datasouce plugin module
2022-06-03 12:15:53 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
Tq
592e433b1e
[Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO (#10276)
* replace DataSourceController API with string JSON and cast to DTO

* add DataSourceProcessorManager and DataSourceProcessorProvider to datasource processor functions
2022-05-30 14:12:19 +08:00
zhuxt2015
206b7c1c51
[Fix-9352][datasource-api]data source client uses a cache with an expiration time (#9353)
* datasource client cache with expiration time

* remove unused import

* fixed unit test

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2022-04-27 16:56:24 +08:00
wqxs
32f76e487f
[fix-9428] [DataSource] delete the password and test again, it is successful without password bug (#9428) (#9531)
* fix bug: After the test data source is successful, delete the password, re-test, or success

* Modify the method of generating DatasourceUniqueId
2022-04-18 20:46:37 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
worry
13cc84eb8a
[Fix-8281][DataSource]Mysql jdbc connect failed. (#9199)
* [Fix-8281][DataSource]Mysql jdbc connect failed.This bug occurs when MySQL has no password.

* [Fix-8281][DataSource]Mysql jdbc connect failed.This bug occurs when MySQL has no password.
2022-03-28 12:32:49 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
ronyang1985
7b66b66e8f
[Feature-8527][Datasource] Add dataSource for Amazon Redshift (#8564)
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2022-03-01 19:29:52 +08:00
zhuxt2015
2a1406073a
fix jdbc connect parameters can not input '@' and '.' character (#8293)
* fix jdbc connect parameters can not input '@' and '.' character

This closes #8135

* add AbstractDataSourceProcessor class unit test

* add license header

* remove class comment

* optimize unit tests

* modify the variable name

Co-authored-by: 朱学通 <zhuxuetong@gridsum.com>
2022-02-18 21:58:43 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 (#6718)
* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA

* fix Constants bug

* remove the unused class

* add unit test

* fix code style error

* add unit test

* refactor data quality common entity

* fix code style error

* add unit test

* close e2e test

* fix code smell bug

* modify dataquality enum value to 14 in TaskType

* add data qualtiy task

* update

* add getDatasourceOptions interface

* fix checkstyle

* close e2e test

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* add unit test and alter dataquality task result

* fix checkstyle

* fix process service test error

* add unit test and fix code smells

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* change execute sql type name

* revert ui pom.xml

* fix data quality task error

* fix checkstyle error

* fix dq task src_connector_type ui select bug

* fix spark rw postgresql bug

* change mysql driver scope

* fix form-create json bug

* fix code smell

* fix DolphinException Bug

* fix ui validate rule and Alert title

* fix target connection param bug

* fix threshold validate change

* add rule input entry index

* change statistic_comparison_check logic

* remove check type change

* add DateExpressionReplaceUtil

* fix null point expetion

* fix null point expetion

* fix test error

* add more sql driver

* fix test error and remove DateExprReplaceUtil

* add get datasource tables and columns

* add get datasource tables and columns

* remove hive-jdbc in pom.xml

* fix code smells

* update sql

* change the pom.xml

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

* update sql

* [Feature][DataQuality] Add data quality task ui (#5054)

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* fix dq task src_connector_type ui select bug

* fix threshold validate change

* remove check type change

* add get datasource tables and columns

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality module  (#4830)

* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* update

* close e2e test

* fix spark rw postgresql bug

* change mysql driver scope

* add more sql driver

* remove hive-jdbc in pom.xml

* change the pom.xml

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality task backend (#4883)

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA


Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* refactor data_quality_module

* add header license

* data quality module refactor

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix code smell

* fix check style

* fix unit test error

* task statistics value add unique code

* fix unit test error

* fix checkstyle error

* fix checkstyle

* fix security hotspot

* fix unit test error

* fix security hotspot

* fix check

* add data quality task error handling

* fix unit test error

* add unit test

* add unit test

* optimize data quality result alert

* fix unit test

* fix sql script error

* fix bug

* update sql script

* fix checkstyle

* add license

* fix checkstyle

* fix checkstyle

* fix unit test

* add jacoco dependencies

* fix unit test

* fix unit test

* add jacoco dependencies

* add unit test

* add unit test

* add license

* fix checkstyle

* fix pom

* fix checkstyle

* fix checkstyle

* merge dev

* fix ui error

* fix pom error

* fix pom error

* fix test error

* fix test error

* mssql-jdbc exclude azure-keyvault

* fix test error

* merge dev and add unit test

* add notes

* rollback the CollectionUtils

* fix

* update sql

* fix

* fix

* fix query rule page error

* change dq.jar path

* fix sql error

* fix ui error

* fix(dq): jar path&task enum description

* add notes on DataQualityApplication

* fix dq result jump error

* fix(ui): page condition

* feat(ui): add show error output path

* change version

* remove all chinese word in sql

* merge

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
mask
898727709f
[fix-7671][plugin] Supports whether SQL is placed in the same session for configuration(fix-7671) (#7675)
-Supports whether SQL is placed in the same session for configuration
2022-01-13 18:06:40 +08:00
X&Z
57f000ed1f
[BUG] Repair JDBC connection of Oracle (#7883) 2022-01-08 20:45:27 +08:00
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue (#7491)
* add task group

* modify task group

* pull dev

* add license header

* modify code style

* fix code style

* fix sql error

* fix error

* fix test

* fix test

* fix test

* fix test

* fix code style

* fix ut

* code style

* fix unit test

* test ut

* ut

* add unittest

* test ut

* modify back ut

* majorization code

* fix conflict

* fix ut

* add task group api

* reset file

* fix ut

* fix lost column

* fix ut

* fix ut

* fix ut

* fix ut

* delete duplicate code

* fix code style 、name

* fix ut

* fix mapper

Co-authored-by: wangxj <wangxj31>
2021-12-24 16:37:59 +08:00
wangbowen
85beb50f03
[Fix-7485][datasource-plugin] Fix create hive datasource using ZooKeeper way in 2.0.1 (#7489) 2021-12-21 14:37:33 +08:00
Zonglei Dong
7e0010de35
[Refactor] Rename class file, unified word spelling. (#7261) 2021-12-15 01:04:50 +08:00
mask
8e62dd8246
[Fix-7291][datasource] Compatible with different versions of MySQL data source loading (#7291) (#7305) 2021-12-13 22:49:46 +08:00
mask
8c3fa47903
[Fix-7375][datasource] remove excess hive jdbc principal (#7375) (#7381)
* [Fix-7375][datasource] remove excess hive jdbc principal (#7375)

* fix ut
2021-12-13 21:38:50 +08:00
mask
deefc8e98a
[Fix-7235][datasource] unable to confige the multi-hosts (#7235) (#7253) 2021-12-08 09:21:54 +08:00
Zonglei Dong
0b9792db76
[Refactor]Rename class file, DatasourceUtil to DataSourceUtils. (#7181)
* Rename class file, DatasourceUtil  to  DataSourceUtils.
2021-12-07 18:49:05 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time (#7054) 2021-12-01 13:55:26 +08:00
sparklezzz
42e20cbf60
fix(kerboros): set correct classloader for kerboros conf load in plugin tasks (#7027)
Co-authored-by: xudong.zhang <xudong.zhang@nio.com>
2021-11-28 09:25:51 +08:00
yihong
861aaaf971
[FIX-#6505][Dao] upgrade the MySQL driver package for building MySQL jdbcUrl (#6708)
* replace com.mysql.jdbc.Driver everywhere with com.mysql.cj.jdbc.Driver
2021-11-18 08:39:11 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
mask
a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490) (#6828)
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490)

* fix pom dep scope

* filter alert

* filter alert

* filter alert

* fix checkstyle

* remove hikaricp version

* update pom

* add properties

* add properties

* rename properties

* fix e2e fail

* rename properties
2021-11-14 13:02:58 +08:00
Jiajie Zhong
6045728428
[ci] Add checkstyle prohibit rule System.out.println (#6778)
* [ci] Add checkstyle prohibit rule System.out.println

* Correct RegexpSingleline place
2021-11-13 16:47:29 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage (#6767) 2021-11-10 18:29:17 +08:00
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. (#6071) (#6361)
* [feature-6071][plugin] add datasource plugin. (#6071)

* remove the dependency of datasource in common and task-plugin module.

* the new datasource-api module provides data source connection acquisition.

* add sqlserver data source plug-in.

* add ds license

* fix check style

* update spi pom

* frx conflict resolution

* fix check style

* fix datasource ut

* fix code check style

* fix code check style

* fix datasource ut

* add pg hive mysql oracle clickhouse db2  datasource

* fix datasource package

* update datasource pom

* update datasource pom

* update datasource pom

* update datasource pom

* update pom

* update pom dep

* update pom dep

* fix license

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* add oracle package

* modify comments

* conflict resolution

* rename log comment

* rename log comment
2021-11-09 20:20:07 +08:00