Martin Huang
b0fc6e7a69
[Feature-5752][server]Add blocking task ( #6272 )
...
* fix python task execution error
* modify code style
* delete AbstractTask#setCommand
* migrate blocking task alert feature and common attribute
* blocking task migrate(complete base features)
* migrate blocking task test
* resolve TaskNode.java conflict
* fix package problem in BlockingParameters
* resolve duplicate code
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* optimise code based on pr#6283 in ds master repo
* add log
* modify log
* resolve conflicts
* change hardcode to variable
* Add blocking task UI
* refactor old code to compatable with the newest branch
* add override annotation
* remove old code
* remove never used packages
* add missing packages declaration
* Refine the i18n description
* add UT
* resolve some code smell problems
* add fetch-depth
* enable unit test in sonar analysis
* disable unit test
* remove unused packages
* resolve duplicate codes
* add logs when task status changed and rename some variables
* add BLOCK & READY_BLOCK status and corresponding processing code
* revoke UI
* revert unit-test
Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2022-03-07 10:50:40 +08:00
Yao WANG
67a6813e9c
remove no useful code ( #8552 )
...
Co-authored-by: ywang46 <ywang46@paypal.com>
2022-03-03 19:09:35 +08:00
xiangzihao
74afdcf8f3
[Fix-8506][Master] process instance stop result wrong ( #8529 )
...
* fix bug_8506
* recover success state
2022-03-01 15:20:02 +08:00
yimaixinchen
a72742b9de
remove unuse import jsonFormat ( #8548 )
2022-02-25 22:05:05 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
kezhenxu94
fa19006a18
Add a property to enable/disable metrics module ( #8484 )
2022-02-24 10:21:47 +08:00
xiangzihao
234f399488
remove worker group lowercase ( #8448 )
2022-02-19 21:38:58 +08:00
xiangzihao
8200a3f15a
add try catch ( #8433 )
2022-02-18 21:46:52 +08:00
xiangzihao
864ceb164f
[Fix-Typo] Fix typo in dev branch. ( #8395 )
...
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* test
* fix dev_bug_8337
* fix dev_bug_8337
* fix dev_bug_8337
* fix bug_8337
* fix typo
2022-02-16 17:38:43 +08:00
xiangzihao
08f642c756
[Fix-8337][Master] Process instance can not be kill when task is failure and can be retry ( #8387 )
...
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* test
* fix dev_bug_8337
* fix dev_bug_8337
* fix dev_bug_8337
* fix bug_8337
2022-02-15 19:04:19 +08:00
Wenjun Ruan
66dcf2376a
Optimize some code ( #8324 )
2022-02-10 19:07:53 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
Wenjun Ruan
f88def8ef7
[IMPROVEMENT-8178] Add Netty processor in Spring container ( #8179 )
...
* Add Netty processor in Spring container
2022-01-25 21:33:09 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
wind
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-25 09:37:57 +08:00
天仇
90ea6ebb6f
[Improvement-7907][refactor] Optimization for query task instances list when build dag flow ( #7915 )
...
Optimization for query task instances list when build dag flow
This closes #7907
2022-01-16 22:30:09 +08:00
Assert
12d7e6261e
[Improvement][MasterServer] process complement data optimization #7925 ( #7976 )
...
* [Improvement][MasterServer] process complement data optimization #7925
* [Improvement][MasterServer] process complement data optimization #7925
Co-authored-by: shangeyao <sgy960921>
2022-01-13 10:42:28 +08:00
wind
a8c592bd93
[Bug-7788][MasterServer] fix submit duplicate tasks sometimes when retry ( #7809 )
...
* [Bug-7788] fix submit duplicate tasks sometimes when retry
* [Bug-7788][MasterServer] fix submit duplicate tasks sometimes when retry
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-11 22:54:42 +08:00
JinYong Li
24a97fda75
[Bug] [dolphinscheduler-api]support workflow instance to definition ( #7930 )
...
* add task save and binds workflow
* add task update with upstream
* support workflow instance to definition
* fix ut
2022-01-11 14:41:45 +08:00
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
wind
4b22ad6cf6
[Implement][MasterServer]TaskProcessor code optimization ( #7754 )
...
* task processor optimization
* fix test
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-04 13:38:31 +08:00
aaronlinv
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
2021-12-30 16:29:22 +08:00
wind
d3bd7309fb
[Bug-7686][Server]fix restart server after kill force ( #7688 )
...
* [DS-7686][Server]fix restart server after kill force
* update registry logic
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-29 15:35:03 +08:00
aaronlinv
9a1cea402e
[Fix-7062][MasterServer] NPE when run switch conditions sub-process ( #7687 )
2021-12-29 10:58:23 +08:00
wind
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-28 19:32:10 +08:00
lhjzmn
3af4d765c2
[Fix-7538] [server] Fix when there is a forbidden node in dag, the execution flow is abnormal ( #7613 )
...
* when there is a forbidden node in dag, the execution flow is abnormal
Co-authored-by: hongjie.li <hongjie.li@dmall.com>
2021-12-28 18:37:59 +08:00
Assert
e172015cd3
fix conflict pr 6997 ( #7571 )
...
Co-authored-by: shangeyao <sgy960921>
2021-12-27 12:03:27 +08:00
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue ( #7491 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
2021-12-24 16:37:59 +08:00
xiangzihao
1efa85ca27
[Fix-7534][Master] zookeeper failover error ( #7562 )
...
* fix bug_7534
* fix bug_7534
* fix bug_7534
* fix bug_7534
* fix bug_7534
* fix bug_7534
Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-24 11:40:21 +08:00
SbloodyS
9f56123a26
fix bug_7537 ( #7551 )
...
Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-22 20:21:52 +08:00
Kerwin
b23f756c22
[Fix-7146][server]Fix the task processor thread is not safe. ( #7488 )
...
* Fix the task processor thread is not safe.
* fix code style
2021-12-20 17:58:57 +08:00
wind
41bf1a955e
[Improvement-7213][MasterServer] execute thread pool code optimization ( #7258 )
...
* threadpool optimization
* threadpool params
* rebase dev
* ut check fix
* add return
* rebase dev
* event loop
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-16 23:45:32 +08:00
wind
1cbc86b9e6
add task-logger config ( #7423 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-16 16:14:46 +08:00
wind
99b8ec6492
[Bug-7319][MasterServer] fix taskNode NPE when switch else branch is empty ( #7320 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-13 22:46:48 +08:00
Kerwin
939fa0c3ba
fix TaskResponseProcessor process NullPointerException ( #7318 )
2021-12-10 16:55:56 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
