Commit Graph

48 Commits

Author SHA1 Message Date
devosend
7ddaa2f47d [maven-release-plugin] prepare for next development iteration 2022-06-28 16:41:31 +08:00
devosend
0a1b9bdd52 [maven-release-plugin] prepare release 3.0.0-beta-2 2022-06-28 16:41:30 +08:00
devosend
d68dcda2bb [chore] pre-release change pom.xml 2022-06-28 08:24:21 +08:00
Jiajie Zhong
57ade38939 [maven-release-plugin] prepare release 3.0.0-beta-1 2022-05-26 20:26:32 +08:00
旺阳
e08b08efdd [improve] Change Mysql Driver (#10220)
(cherry picked from commit aba5f8a40e)
2022-05-25 23:42:28 +08:00
旺阳
5d8679fba9 [Fix-10103][k8s]Fix k8s Change DataSource Error (#10128)
(cherry picked from commit de5507fb19)
2022-05-24 23:01:42 +08:00
Jiajie Zhong
a9fa6b33a4
[chore] Change release version to 3.0.0-beta-1 (#9957) 2022-05-12 21:49:21 +08:00
Paul Zhang
8562f6a878
[Feature][Log]Add timezone information in log output (#9913) 2022-05-06 17:31:44 +08:00
LongJGun
778018dcfb
[Bug] [worker] fix CommandType TASK_EXECUTE_RUNNING_ACK don't consumed (#9849) (#9850) 2022-05-03 14:28:19 +08:00
Jiajie Zhong
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server (#9726)
* [common] Make dolphinscheduler_env.sh work

* Change dist tarball `dolphinscheduler_env.sh` location
  from `bin/` to `conf/`, which users could finish their
  change configuration operation in one single directory.
  and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
  when we start our sever instead of adding both
  `$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
caishunfeng
5657cb9aec
[Bug-9719][Master] fix failover fail because task plugins has not been loaded (#9720) 2022-04-24 20:34:21 +08:00
WangJPLeo
996790ce9e
[Improvement-9609][Worker]The resource download method is selected according to the configurati… (#9636)
* The resource download method is selected according to the configuration and the service startup verification is added.

* common check CI fix

* Startup check changed to running check

* code smell

* Coordinate resources to increase test coverage.

* Split resource download method.

* Unit Test Coverage

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-22 11:45:49 +08:00
caishunfeng
239be31ab7
[Bug] cancel application when kill task (#9624)
* cancel application when kill task

* add warn log

* add cancel application
2022-04-20 22:46:15 +08:00
WangJPLeo
9964c4c1e1
[Fix-9593] Storage Management StorageOperate No Instance (#9594)
* Storage Management StorageOperate No Instance

* Add StorageOperateManager unit test

* Add license header

* Fix issues in SonarCloud code analysis

Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-20 09:58:37 +08:00
caishunfeng
b4017d0afd
fix task kill (#9578) 2022-04-19 15:26:12 +08:00
caishunfeng
66d148872d
[Bug-9501][Worker] fix kill task error before running (#9509) 2022-04-14 21:34:02 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
caishunfeng
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719)
* rebase dev

* change task state to dispatch if dispatch success

* update task host when dispatch

* add dispatch task event

* test

* check tenant after enable auto create

* handle dispatch state

Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 17:35:05 +08:00
worry
b4b5241797
[DS-9266][feat][server]Implement show dolphinscheduler banner at launch (#9273)
- add banner.txt
- modify application.yaml

This closes #9266
2022-03-30 16:45:24 +08:00
caishunfeng
263791a63e
[Bug-9137][Server] fix timezone init (#9219)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
gaojun2048
3955d84ce3
[Bug][Security] Fix dir permission (#8938)
* only config dir need 700

* delete -R

* all config dir change

* yaml file in config dir change

* config dir permission change
2022-03-17 14:29:18 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
wangyang
0d8079a816
[DS-7992][feat] Alert module judging strategy (#8636) 2022-03-14 14:25:27 +08:00
guoshupei
6a5b3bb8f9
[Improvement-8758] [worker] Delete unused object preTaskCache (#8795) 2022-03-14 14:14:21 +08:00
gaojun2048
6514ed2da2
change dir permission (#8861) 2022-03-14 13:44:48 +08:00
gaojun2048
3b30f3bb7e
fix log read bug (#8852)
* fix log

* get DOLPHINSCHEDULER_HOME

* add DOLPHINSCHEDULER_WORKER_HOME to env
2022-03-14 13:15:14 +08:00
ronyang1985
20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR (#8503)
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-03-01 11:43:37 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477)
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
kezhenxu94
fa19006a18
Add a property to enable/disable metrics module (#8484) 2022-02-24 10:21:47 +08:00
caishunfeng
f2b9796bc2
[Bug-8110][WorkerServer] kill all running task before worker stop (#8490)
* kill all running task before worker stop

* kill local process

* remove kill yarn job

* adjust the orders of close

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-23 18:22:42 +08:00
yimaixinchen
da09c00cba
optimize worker taskProcessor logger. (#8242) 2022-02-11 09:48:04 +08:00
yimaixinchen
896dec21e7
change to autowired annotation (#8240) 2022-02-07 23:24:42 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 (#6718)
* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA

* fix Constants bug

* remove the unused class

* add unit test

* fix code style error

* add unit test

* refactor data quality common entity

* fix code style error

* add unit test

* close e2e test

* fix code smell bug

* modify dataquality enum value to 14 in TaskType

* add data qualtiy task

* update

* add getDatasourceOptions interface

* fix checkstyle

* close e2e test

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* add unit test and alter dataquality task result

* fix checkstyle

* fix process service test error

* add unit test and fix code smells

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* change execute sql type name

* revert ui pom.xml

* fix data quality task error

* fix checkstyle error

* fix dq task src_connector_type ui select bug

* fix spark rw postgresql bug

* change mysql driver scope

* fix form-create json bug

* fix code smell

* fix DolphinException Bug

* fix ui validate rule and Alert title

* fix target connection param bug

* fix threshold validate change

* add rule input entry index

* change statistic_comparison_check logic

* remove check type change

* add DateExpressionReplaceUtil

* fix null point expetion

* fix null point expetion

* fix test error

* add more sql driver

* fix test error and remove DateExprReplaceUtil

* add get datasource tables and columns

* add get datasource tables and columns

* remove hive-jdbc in pom.xml

* fix code smells

* update sql

* change the pom.xml

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

* update sql

* [Feature][DataQuality] Add data quality task ui (#5054)

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* fix dq task src_connector_type ui select bug

* fix threshold validate change

* remove check type change

* add get datasource tables and columns

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality module  (#4830)

* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* update

* close e2e test

* fix spark rw postgresql bug

* change mysql driver scope

* add more sql driver

* remove hive-jdbc in pom.xml

* change the pom.xml

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality task backend (#4883)

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA


Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* refactor data_quality_module

* add header license

* data quality module refactor

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix code smell

* fix check style

* fix unit test error

* task statistics value add unique code

* fix unit test error

* fix checkstyle error

* fix checkstyle

* fix security hotspot

* fix unit test error

* fix security hotspot

* fix check

* add data quality task error handling

* fix unit test error

* add unit test

* add unit test

* optimize data quality result alert

* fix unit test

* fix sql script error

* fix bug

* update sql script

* fix checkstyle

* add license

* fix checkstyle

* fix checkstyle

* fix unit test

* add jacoco dependencies

* fix unit test

* fix unit test

* add jacoco dependencies

* add unit test

* add unit test

* add license

* fix checkstyle

* fix pom

* fix checkstyle

* fix checkstyle

* merge dev

* fix ui error

* fix pom error

* fix pom error

* fix test error

* fix test error

* mssql-jdbc exclude azure-keyvault

* fix test error

* merge dev and add unit test

* add notes

* rollback the CollectionUtils

* fix

* update sql

* fix

* fix

* fix query rule page error

* change dq.jar path

* fix sql error

* fix ui error

* fix(dq): jar path&task enum description

* add notes on DataQualityApplication

* fix dq result jump error

* fix(ui): page condition

* feat(ui): add show error output path

* change version

* remove all chinese word in sql

* merge

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
wind
b05ba4d1b9
fix host update processor init (#8205)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-26 17:51:20 +08:00
Wenjun Ruan
f88def8ef7
[IMPROVEMENT-8178] Add Netty processor in Spring container (#8179)
* Add Netty processor in Spring container
2022-01-25 21:33:09 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
yimaixinchen
08e303f867
Fix ResponseCache Class Name ERROR (#8015)
* fix ResponseCache name error
2022-01-14 10:47:05 +08:00
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log (#7831)
* task logger

* update

* log format

* remove master config task-logger

* udpate log level

Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
Kerwin
5172862a0a
Add seatunnel task plugin (#7131) 2022-01-07 20:51:11 +08:00
springmonster
0911fd711d
[Bug-7832, Improvement] Add unit test for worker module, fix apache#7832. Optimize HttpUtils and HttpUtilsTest 1. Checking input parameters of getResponseContentString 2. Add unit test (#7798) 2022-01-06 09:56:30 +08:00
aaronlinv
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker (#6894) 2021-12-30 16:29:22 +08:00
wind
d3bd7309fb
[Bug-7686][Server]fix restart server after kill force (#7688)
* [DS-7686][Server]fix restart server after kill force

* update registry logic

Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-29 15:35:03 +08:00
zwZjut
0d861fe46a
to #7609 (#7611)
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
2021-12-25 12:26:43 +08:00
wind
f450b7ef28
[BUG-7554] Fix database lose data in standalone (#7620)
* Fix the wrong lifecycle phase of initialization

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2021-12-25 01:05:08 +08:00
SbloodyS
92bbb9e4af
remove date+1 when the data is supplemented (#7452) 2021-12-18 11:04:45 +08:00
kezhenxu94
2ff76c7e73
Fix some SQL issues and make docker-compose runnable (#7329) 2021-12-10 23:51:42 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00