zixi0825
6a02b15e40
[Fix][data-quality] change data-quality task main jar path ( #8752 )
...
* feat(data-quality) change data-quality task main jar path
* feat(data-quality) change data-quality task main jar path
Co-authored-by: zixi0825 <sunzhaohe@linklogis.com>
2022-03-09 21:49:19 +08:00
Martin Huang
b0fc6e7a69
[Feature-5752][server]Add blocking task ( #6272 )
...
* fix python task execution error
* modify code style
* delete AbstractTask#setCommand
* migrate blocking task alert feature and common attribute
* blocking task migrate(complete base features)
* migrate blocking task test
* resolve TaskNode.java conflict
* fix package problem in BlockingParameters
* resolve duplicate code
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* optimise code based on pr#6283 in ds master repo
* add log
* modify log
* resolve conflicts
* change hardcode to variable
* Add blocking task UI
* refactor old code to compatable with the newest branch
* add override annotation
* remove old code
* remove never used packages
* add missing packages declaration
* Refine the i18n description
* add UT
* resolve some code smell problems
* add fetch-depth
* enable unit test in sonar analysis
* disable unit test
* remove unused packages
* resolve duplicate codes
* add logs when task status changed and rename some variables
* add BLOCK & READY_BLOCK status and corresponding processing code
* revoke UI
* revert unit-test
Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2022-03-07 10:50:40 +08:00
ronyang1985
20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR ( #8503 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-03-01 11:43:37 +08:00
yimaixinchen
a72742b9de
remove unuse import jsonFormat ( #8548 )
2022-02-25 22:05:05 +08:00
xiangzihao
891a002bea
[Feature-#8373][MasterServer] Dependent tasks can re-run automatically in the case of complement ( #8496 )
...
* first add feature_8373
* fix code smell
* add blank line
* fix some problems
* fix unit test error
2022-02-25 10:34:02 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
wangbowen
c1592ff38e
[Fix-8394][Server]fix when the data is supplemented business time still date+1 ( #8511 )
2022-02-24 10:07:59 +08:00
Kirs
0ef49b4a98
[Improve]Use hibernate validation framework ( #4 ) ( #8467 )
...
* [Improve]Use hibernate validation framework
Improve email verification method
Support domain name verification
* Email is at least a second-level domain name
2022-02-21 16:54:46 +08:00
ronyang1985
83e88c4bc6
[Fix-8389][Common] Fix oshi compatibility issues cause it fail to boot on m1 mac ( #8390 )
2022-02-21 11:14:39 +08:00
qianli2022
151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 ( #8252 )
...
* add db operator for namespace
* add k8s client
* add k8s controller
* add namespace ui
* add licenses
* add kubernetes model known-dependencies
* add new licenses
* fix SonarCloud Code Analysis bug
* remove license for standard protocol
* fix variable MemoryStr to memoryStr
Co-authored-by: qianl4 <qianl4@cicso.com>
2022-02-15 10:21:15 +08:00
wind
fc9a31f813
[E2E] add switch task case ( #8180 )
...
* fix switch bug
* add switch e2e test
* add switch workflow
* pre task option
* selectPreTask id
* update
* switch task e2e
* naming optimization
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-10 10:00:50 +08:00
ouyangyewei
294882f5d7
[Improvement-8218][dolphinscheduler-api] QueryTaskInstanceListPaging API performance improvement ( #8223 )
2022-02-09 12:58:15 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
yc322
fb9f4a1339
[Feature-5822][api] Add audit log ( #6322 )
...
* rebase
* update audit log type name
* fix the quoted property error
* fix bug
* update audit global switch default value
* fix wrong ddl comment
* fix wrong code style
* update audit log search sql
* add licenses
* add unit test
* fix wrong code style
* fix bugs
* fix wrong code style
* add test case
* add test case
* add license
* Add unit test in pom.xml
* resolve code smell problem
* mysql and postgresql ddl update
* update audit log path and audit page query
* update audit configuration
* update audit log schema, add resource type and resource id
* update audit query and list page
* resolve audit log front page bugs and log query error
* update audit configuration
* resolve wrong configuration
* update AuditLogControllerTest
* update application.yaml
* resolve code smell
* update standalone configuration
* resolve bugs
Co-authored-by: yc322 <1393242780@qq.com>
2022-01-26 09:53:18 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
wind
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-25 09:37:57 +08:00
mask
898727709f
[fix-7671][plugin] Supports whether SQL is placed in the same session for configuration(fix-7671) ( #7675 )
...
-Supports whether SQL is placed in the same session for configuration
2022-01-13 18:06:40 +08:00
Ivan0626
b807c677ab
Close the FileOutputStream in FileUtils.java file ( #7937 )
...
Close the FileOutputStream in FileUtils.java file
2022-01-12 09:24:50 +08:00
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
Kerwin
5172862a0a
Add seatunnel task plugin ( #7131 )
2022-01-07 20:51:11 +08:00
springmonster
0911fd711d
[Bug-7832, Improvement] Add unit test for worker module, fix apache#7832. Optimize HttpUtils and HttpUtilsTest 1. Checking input parameters of getResponseContentString 2. Add unit test ( #7798 )
2022-01-06 09:56:30 +08:00
aaronlinv
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
2021-12-30 16:29:22 +08:00
wind
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-28 19:32:10 +08:00
BaoLiang
605767e47b
[Fix-7654]Dependent node onChangeProjectCode error ( #7672 )
...
* fix #7654 Dependent node onChangeProjectCode error
* fix #7654 Dependent node onChangeProjectCode error
* fix #7654 Dependent node onChangeProjectCode error
* update ut
2021-12-28 18:09:42 +08:00
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue ( #7491 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
2021-12-24 16:37:59 +08:00
wind
41bf1a955e
[Improvement-7213][MasterServer] execute thread pool code optimization ( #7258 )
...
* threadpool optimization
* threadpool params
* rebase dev
* ut check fix
* add return
* rebase dev
* event loop
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-16 23:45:32 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
Kerwin
4e6265df11
fix Property or field 'all' cannot be found ( #7246 )
2021-12-08 09:00:00 +08:00
Manhua
2dc09c627f
[Fix-5551][common] Fix ExecutionStatus for hive-on-Tez application ( #6289 )
...
* add ENDED state
* Update HadoopUtils.java
2021-12-05 14:03:29 +08:00
zwZjut
af9ed35295
[Feature][MasterServer] add cache for workGroup and schedule #6987 ( #7161 )
...
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987 : @Param -> @Name
* to #6987 : fix Sonar
* to #6987 : fix Sonar
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
2021-12-04 16:59:05 +08:00
wind
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow ( #7090 )
...
* add cache manager for workflow
* [DS-6988][MasterServer] add cache manager for workflow
* cache evict code optimization
* test
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-03 23:13:11 +08:00
wangxj3
2b519f31e9
[Feature-#6422] task group queue ( #6722 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
2021-12-03 14:02:31 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time ( #7054 )
2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config ( #6947 )
...
* [DS-5229][fix] server port custom config
This closes #5229
* [DS-5229][feat] Implement server port custom config
This closes #5229
* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862 )
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md
* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852 )
Co-authored-by: caishunfeng <534328519@qq.com>
* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229
Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-01 00:21:03 +08:00
Kerwin
f0630bc9a0
Fix view task status module on project homepage cannot read property label of undefined ( #6956 )
2021-11-30 10:00:54 +08:00
Hua Jiang
a0955a078a
[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. ( #7046 )
2021-11-30 09:59:48 +08:00
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user ( #6974 )
...
* cache manager for tenant and user
* add test
* fix checkstyle
* add dependencies
* fix sonar check
* use spring boot cache configuration
* add license
* check style
* check style
* add license
* change proxy name to processor
* add license header
* check style
* add license file
* add license file
* add license
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-30 08:29:25 +08:00
lvshaokang
1a88b1863b
[Improvement-5785][common] Fix code format ( #6864 )
...
* [DS-5785][style] Fix common module code format
* [DS-5785][style] Fix common module code format
2021-11-26 14:43:23 +08:00
kezhenxu94
ef0cb46661
Fix shell tools for database schema cannot run ( #7003 )
2021-11-26 11:10:38 +08:00
wind
e6239e808b
[DS-6829][WorkerServer] skip create log dir and print log in dryRun model ( #6852 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-21 17:51:56 +08:00
JinYong Li
4bec792160
[Fix][Common] rewrite code generate,fix bit shift ( #6914 )
...
* rewrite code generate,fix bit shift
* fix ut
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* fix ut
2021-11-19 16:04:05 +08:00
Kerwin
802fc498b5
[Fix-#6605][Common] Fix the situation where the obtained string property is empty. ( #6661 )
...
* Fix the situation where the obtained string property is empty.
* recovery test
* update code style.
* megre dev.
2021-11-19 10:58:17 +08:00
chouc
f564687a89
[Improvement][Log] optimize task log path to reduce task running time ( #6731 )
...
* [Improvement][Log] optimize task log path to reduce task running time
* Reduce the creation of log directories
Co-authored-by: Kirs <acm_master@163.com>
2021-11-17 17:08:44 +08:00
OS
880c1d28ff
[Feature][master] change sub process id to code. ( #6846 )
...
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* code style
* fix ut process
* fix ut process
2021-11-15 01:00:44 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
2021-11-14 20:22:13 +08:00
mask
a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP ( #6490 ) ( #6828 )
...
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490 )
* fix pom dep scope
* filter alert
* filter alert
* filter alert
* fix checkstyle
* remove hikaricp version
* update pom
* add properties
* add properties
* rename properties
* fix e2e fail
* rename properties
2021-11-14 13:02:58 +08:00
Assert
089f73ebe4
[Fix-6707][common] Fix complement data error because of global variables ( #6759 )
...
* [Fix-6707][common] Fix complement data error because of global variables
* [Fix-6707][common] Fix complement data error because of global variables
* [Fix-6707][common] Fix complement data error because of global variables
* [Fix-6707][common] Fix complement data error because of global variables
Co-authored-by: shangeyao <sgy960921>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
2021-11-13 17:06:15 +08:00
Jiajie Zhong
6045728428
[ci] Add checkstyle prohibit rule System.out.println ( #6778 )
...
* [ci] Add checkstyle prohibit rule System.out.println
* Correct RegexpSingleline place
2021-11-13 16:47:29 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
2021-11-10 18:29:17 +08:00
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. ( #6071 ) ( #6361 )
...
* [feature-6071][plugin] add datasource plugin. (#6071 )
* remove the dependency of datasource in common and task-plugin module.
* the new datasource-api module provides data source connection acquisition.
* add sqlserver data source plug-in.
* add ds license
* fix check style
* update spi pom
* frx conflict resolution
* fix check style
* fix datasource ut
* fix code check style
* fix code check style
* fix datasource ut
* add pg hive mysql oracle clickhouse db2 datasource
* fix datasource package
* update datasource pom
* update datasource pom
* update datasource pom
* update datasource pom
* update pom
* update pom dep
* update pom dep
* fix license
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* add oracle package
* modify comments
* conflict resolution
* rename log comment
* rename log comment
2021-11-09 20:20:07 +08:00
