Commit Graph

636 Commits

Author SHA1 Message Date
JinYong Li
85e56b5e2a
fix 9243 (#9244) 2022-03-28 20:45:01 +08:00
worry
d3251c9bcc
[Improvement-9227][master]implement use the slot to scan the database (#9228)
when the master assigns tasks by slot,implement use the slot to scan the database.

This closes #9227
2022-03-28 09:52:12 +08:00
shuai hou
5bd8ba4268
[fix-8929][DAO]Description Failed to upload the file because the full name was too long (#9020)
* [fix-8929][DAO]Description Failed to upload the file because the full name was too long

* [DS-#8929][fix]Description Failed to upload the file because the full name was too long

* [DS-#8929][fix]Description Failed to upload the file because the full name was too long

* [DS-#8929][fix]Description Failed to upload the file because the full name was too long

Co-authored-by: houshuai <houshuai@jiguang.cn>
2022-03-25 13:40:20 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
hjli
242b5dd5b6
[Bug-8760]Resource file deletion conditions (#8765) 2022-03-15 17:26:41 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
wangyang
0d8079a816
[DS-7992][feat] Alert module judging strategy (#8636) 2022-03-14 14:25:27 +08:00
ouyangyewei
b0ffec5a4c
to issue/#8826 (#8827)
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
2022-03-13 08:39:05 +08:00
Martin Huang
b0fc6e7a69
[Feature-5752][server]Add blocking task (#6272)
* fix python task execution error

* modify code style

* delete AbstractTask#setCommand

* migrate blocking task alert feature and common attribute

* blocking task migrate(complete base features)

* migrate blocking task test

* resolve TaskNode.java conflict

* fix package problem in BlockingParameters

* resolve duplicate code

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* resolve code style

* optimise code based on pr#6283 in ds master repo

* add log

* modify log

* resolve conflicts

* change hardcode to variable

* Add blocking task UI

* refactor old code to compatable with the newest branch

* add override annotation

* remove old code

* remove never used packages

* add missing packages declaration

* Refine the i18n description

* add UT

* resolve some code smell problems

* add fetch-depth

* enable unit test in sonar analysis

* disable unit test

* remove unused packages

* resolve duplicate codes

* add logs when task status changed and rename some variables

* add BLOCK & READY_BLOCK status and corresponding processing code

* revoke UI

* revert unit-test

Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2022-03-07 10:50:40 +08:00
xiangzihao
2ab8c1ca7d
[Fix-8616][WorkFlowLineage] work flow lineage search result missing data (#8684)
* fix bug_8616

* remove meaningless query column

* fix code smell

* fix remaining problems
2022-03-04 14:05:59 +08:00
BaoLiang
44bb654c5e
Revert "upgrade mybatis-plus to 3.4.3 (#8515)" (#8659)
This reverts commit ca6d148a1b.
2022-03-02 16:10:39 +08:00
yimaixinchen
a72742b9de
remove unuse import jsonFormat (#8548) 2022-02-25 22:05:05 +08:00
xiangzihao
891a002bea
[Feature-#8373][MasterServer] Dependent tasks can re-run automatically in the case of complement (#8496)
* first add feature_8373

* fix code smell

* add blank line

* fix some problems

* fix unit test error
2022-02-25 10:34:02 +08:00
BaoLiang
ca6d148a1b
upgrade mybatis-plus to 3.4.3 (#8515) 2022-02-25 08:31:12 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477)
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
jegger
6ece0283df
feat: Unify mysql insert value queto (#8502) 2022-02-24 11:09:54 +08:00
xiangzihao
512f8cb5dd
add index (#8500) 2022-02-23 16:53:36 +08:00
天仇
95c119226b
[Feature-8386][Server] Custom timezone - Add timezone into user info (#8457)
* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Add system default timezone like `GMT+08:00` when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Add system default timezone like `GMT+08:00` when create user
  3. Add timeZone property when update user info
  4. add ddl sql in update sql script

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

 1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386
2022-02-21 23:28:31 +08:00
zixi0825
df6eef84cd
fix(dq) change comparison_type from fluctuation to avg (#8382) 2022-02-20 20:59:08 +08:00
xiangzihao
f6e2a2cf23
[Fix-8296][API] Update process definition error (#8408)
* fix bug_8296

* fix bug_8296

* fix bug_8296

* add postgres sql

* add postgres sql

* add postgres sql
2022-02-18 21:05:43 +08:00
calvin
f18f28cc4a
fix this issue (#8399) 2022-02-16 15:10:19 +08:00
qianli2022
151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 (#8252)
* add db operator for namespace

* add k8s client

* add k8s controller

* add namespace ui

* add licenses

* add kubernetes model known-dependencies

* add new licenses

* fix SonarCloud Code Analysis bug

* remove license for standard protocol

* fix variable MemoryStr to memoryStr

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-02-15 10:21:15 +08:00
calvin
5f25d31a26
fix this issue (#8306) 2022-02-08 15:32:58 +08:00
calvin
6ae3efb5b6
fix this issue (#8274) 2022-02-04 13:23:36 +08:00
calvin
7333348632
fix this issue (#8273) 2022-01-31 17:41:48 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 (#6718)
* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA

* fix Constants bug

* remove the unused class

* add unit test

* fix code style error

* add unit test

* refactor data quality common entity

* fix code style error

* add unit test

* close e2e test

* fix code smell bug

* modify dataquality enum value to 14 in TaskType

* add data qualtiy task

* update

* add getDatasourceOptions interface

* fix checkstyle

* close e2e test

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* add unit test and alter dataquality task result

* fix checkstyle

* fix process service test error

* add unit test and fix code smells

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* change execute sql type name

* revert ui pom.xml

* fix data quality task error

* fix checkstyle error

* fix dq task src_connector_type ui select bug

* fix spark rw postgresql bug

* change mysql driver scope

* fix form-create json bug

* fix code smell

* fix DolphinException Bug

* fix ui validate rule and Alert title

* fix target connection param bug

* fix threshold validate change

* add rule input entry index

* change statistic_comparison_check logic

* remove check type change

* add DateExpressionReplaceUtil

* fix null point expetion

* fix null point expetion

* fix test error

* add more sql driver

* fix test error and remove DateExprReplaceUtil

* add get datasource tables and columns

* add get datasource tables and columns

* remove hive-jdbc in pom.xml

* fix code smells

* update sql

* change the pom.xml

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

* update sql

* [Feature][DataQuality] Add data quality task ui (#5054)

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* fix dq task src_connector_type ui select bug

* fix threshold validate change

* remove check type change

* add get datasource tables and columns

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality module  (#4830)

* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* update

* close e2e test

* fix spark rw postgresql bug

* change mysql driver scope

* add more sql driver

* remove hive-jdbc in pom.xml

* change the pom.xml

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality task backend (#4883)

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA


Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* refactor data_quality_module

* add header license

* data quality module refactor

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix code smell

* fix check style

* fix unit test error

* task statistics value add unique code

* fix unit test error

* fix checkstyle error

* fix checkstyle

* fix security hotspot

* fix unit test error

* fix security hotspot

* fix check

* add data quality task error handling

* fix unit test error

* add unit test

* add unit test

* optimize data quality result alert

* fix unit test

* fix sql script error

* fix bug

* update sql script

* fix checkstyle

* add license

* fix checkstyle

* fix checkstyle

* fix unit test

* add jacoco dependencies

* fix unit test

* fix unit test

* add jacoco dependencies

* add unit test

* add unit test

* add license

* fix checkstyle

* fix pom

* fix checkstyle

* fix checkstyle

* merge dev

* fix ui error

* fix pom error

* fix pom error

* fix test error

* fix test error

* mssql-jdbc exclude azure-keyvault

* fix test error

* merge dev and add unit test

* add notes

* rollback the CollectionUtils

* fix

* update sql

* fix

* fix

* fix query rule page error

* change dq.jar path

* fix sql error

* fix ui error

* fix(dq): jar path&task enum description

* add notes on DataQualityApplication

* fix dq result jump error

* fix(ui): page condition

* feat(ui): add show error output path

* change version

* remove all chinese word in sql

* merge

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
ouyangyewei
aeb7d89c50
to improvement #8192 (#8193)
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
2022-01-26 13:32:17 +08:00
yc322
fb9f4a1339
[Feature-5822][api] Add audit log (#6322)
* rebase

* update audit log type name

* fix the quoted property error

* fix bug

* update audit global switch default value

* fix wrong ddl comment

* fix wrong code style

* update audit log search sql

* add licenses

* add unit test

* fix wrong code style

* fix bugs

* fix wrong code style

* add test case

* add test case

* add license

* Add unit test in pom.xml

* resolve code smell problem

* mysql and postgresql ddl update

* update audit log path and audit page query

* update audit configuration

* update audit log schema, add resource type and resource id

* update audit query and list page

* resolve audit log front page bugs and log query error

* update audit configuration

* resolve wrong configuration

* update AuditLogControllerTest

* update application.yaml

* resolve code smell

* update standalone configuration

* resolve bugs

Co-authored-by: yc322 <1393242780@qq.com>
2022-01-26 09:53:18 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
wind
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization (#8156)
* submit task optimization

* cloneAndReset

* update

* tolerant task restart when init

* fix snoar check

* fix test

* delete unuse file

* taskInstance key

* code style

* skip complete when retry

Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-25 09:37:57 +08:00
springmonster
5e975dad34
[Enhancement-8006] Improve the performance of SQL query (#8007)
Co-authored-by: kuanghc1 <kuanghc1@lenovo.com>
2022-01-14 13:53:31 +08:00
JinYong Li
24a97fda75
[Bug] [dolphinscheduler-api]support workflow instance to definition (#7930)
* add task save and  binds workflow

* add task update with upstream

* support workflow instance to definition

* fix ut
2022-01-11 14:41:45 +08:00
mazhong
5ea06e7eb8
Update ProcessDefinitionMapper.xml (#7931)
Resolves  process definition list issue(data duplication )
2022-01-11 14:20:46 +08:00
天仇
361d68db26
[FIX-7732][fix] fix column 'is_directory' of table t_ds_resources type error in PG database (#7898)
Fix column 'is_directory' of table t_ds_resources type error in PG database
This closes #7732
2022-01-09 11:36:14 +08:00
JinYong Li
acf9c84c50
[Feature][workflow list edit] add task update with upstream (#7829)
* add task save and  binds workflow

* add task update with upstream
2022-01-05 22:08:42 +08:00
ououtt
4c2f77ee9c
[DS-7016][feat] Auto create workflow while import sql script with specific hint (#7214)
* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* datasource : datasource name
* name: task name
* upstream: pre task names

* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* remove excess blank lines

* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* datasource : datasource name
* name: task name
* upstream: pre task names

* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* datasource : datasource name
* name: task name
* upstream: pre task names

* Code optimization

* for Zip Bomb Attack

Co-authored-by: eye <eye.gu@aloudata.com>
2022-01-05 17:55:08 +08:00
天仇
4662e6d929
[BUG-7779][SQL] fix syntax error in init sql of postgre (#7782)
fix syntax error of table `t_ds_task_definition` and `t_ds_task_definition_log` in postgresql with column `task_group_priority int(4)`
2022-01-04 13:06:06 +08:00
JinYong Li
c97170853e
add task save and binds workflow (#7772) 2022-01-04 10:48:17 +08:00
uh001
3e34e69cfb
The sql has a syntax error, and because it is a function, the exception information will be hidden (#7761) 2021-12-31 18:04:59 +08:00
JinYong Li
d7d13f7f51
[Feature][workflow list edit] add task list page api (#7743)
* fix 7597

* fix copy bug

* add task list page api

* fix total

* fix total
2021-12-31 13:39:20 +08:00
calvin
71ccb42697
fix a few bugs (#7703) 2021-12-29 12:57:36 +08:00
calvin
5ab0135d18
[Fix-7345] Support the whole operations for the task group and the task group queue. (#7678)
* add task group

* modify task group

* add task group queue

* add task group queue

* add task group queue

* add task group queue

* add task group priority

* fix a few issue with api

* add task group queue

* modify task group queue

* modify task group

* fix a little bit issue

* fix a little bug in the page of managing task group
2021-12-29 11:40:44 +08:00
wind
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null (#7475)
* fix failover when node host is null

* add failover execute thread

* worker handle dead server

* fix task instance failover time check

* fix upgrade sql

* failover logic update

Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-28 19:32:10 +08:00
xiangzihao
8b292132c8
[Feature-7569]Optimize Dependent node loading times (#7626)
* feature 7569

* feature_7569

Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-27 18:05:56 +08:00
xiangzihao
c7d7eec679
[Fix-7520][Master] Data too long for column 'task_params' at row 1 (#7521)
* fix bug_7520

* fix bug_7520

* fix bug_7520

* bug_7520

Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-24 22:51:18 +08:00
kezhenxu94
7a888c544c
Try (#7603) 2021-12-24 22:19:59 +08:00
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue (#7491)
* add task group

* modify task group

* pull dev

* add license header

* modify code style

* fix code style

* fix sql error

* fix error

* fix test

* fix test

* fix test

* fix test

* fix code style

* fix ut

* code style

* fix unit test

* test ut

* ut

* add unittest

* test ut

* modify back ut

* majorization code

* fix conflict

* fix ut

* add task group api

* reset file

* fix ut

* fix lost column

* fix ut

* fix ut

* fix ut

* fix ut

* delete duplicate code

* fix code style 、name

* fix ut

* fix mapper

Co-authored-by: wangxj <wangxj31>
2021-12-24 16:37:59 +08:00
时光
a00c4f9709
finished issue#7527 (#7561)
Co-authored-by: 时光 <caizhedong@cai-inc.com>
2021-12-23 10:24:32 +08:00
lgcareer
cdd4e7bf03
[Improvement-#7529][tools] Init DB schema from the full sql file. (#7530) 2021-12-23 00:19:58 +08:00
Tom
bb140fbc3b
[DS-7227][refactor] optimize alert plugin instance list order (#7506)
[DS-7227][refactor] optimize alert plugin instance list order
query all alert plugin instance list order by update time desc
optimize `select *` sql in the AlterPluginInstanceMapper.xml
2021-12-21 09:22:16 +08:00