caishunfeng
852597c9bf
[Bug-9070][Server]fix taskExecutionContext json convert error ( #9072 )
...
* [Bug-9070][Server]fix taskExecutionContext json convert error
* test
* add license header
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-22 14:02:07 +08:00
zixi0825
63a10037a6
fix package bug ( #9051 )
2022-03-21 19:57:03 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
zixi0825
6a02b15e40
[Fix][data-quality] change data-quality task main jar path ( #8752 )
...
* feat(data-quality) change data-quality task main jar path
* feat(data-quality) change data-quality task main jar path
Co-authored-by: zixi0825 <sunzhaohe@linklogis.com>
2022-03-09 21:49:19 +08:00
ronyang1985
20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR ( #8503 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-03-01 11:43:37 +08:00
yimaixinchen
a72742b9de
remove unuse import jsonFormat ( #8548 )
2022-02-25 22:05:05 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
Yao WANG
dd1397aaed
[Bug] [Task-plugin] ShellTask add the parentfolder exist check and create if not exit ( #8526 )
...
* check the parent dir exist and create it
* fix lint
Co-authored-by: ywang46 <ywang46@paypal.com>
2022-02-24 17:36:54 +08:00
Ivan0626
60ddede6a6
fixbug: varPool value is error ( #7724 )
2022-02-23 14:53:20 +08:00
ShuiMuNianHuaLP
806333a11c
fix sudo.enable=false Is invalid ( #8388 )
2022-02-17 15:23:21 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
Wenjun Ruan
a1cc2fd11a
[task] Python task support PYTHON_HOME ( #8196 )
2022-01-26 11:09:57 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
Shiwen Cheng
0057cf3fe7
[Improvement][task-plugin] Remove unreasonable log ( #8115 )
2022-01-24 13:44:43 +08:00
seagle
f1a0215978
[bug] Fix datax task type error parse clickhouse reader sql ( #8091 )
2022-01-18 16:53:25 +08:00
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
Kerwin
5172862a0a
Add seatunnel task plugin ( #7131 )
2022-01-07 20:51:11 +08:00
J·Y
a4168ae24e
[Improvement] The spark version is associated with the spark command ( #7544 )
...
* [Improvement] The spark version is associated with the spark command
2022-01-04 14:21:57 +08:00
xiangzihao
c58dbefaa5
[Fix-7459][Sqoop] running error in parallel ( #7752 )
...
* fix bug_7459
* fix bug_7459
* fix bug_7459
* fix bug_7459
* fix bug_7459
Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-31 16:26:07 +08:00
wangxj3
cc77963522
[FIX-#6727][worker-server] Fix procedure params bug ( #7680 )
...
* fix Procedure param error
* code style
Co-authored-by: wangxj <wangxj31>
2021-12-28 19:19:30 +08:00
J·Y
6ae46f2c1b
[Improvement] Merge redundant codes ( #7550 )
2021-12-23 09:53:33 +08:00
Zonglei Dong
7e0010de35
[Refactor] Rename class file, unified word spelling. ( #7261 )
2021-12-15 01:04:50 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
Zonglei Dong
0b9792db76
[Refactor]Rename class file, DatasourceUtil to DataSourceUtils. ( #7181 )
...
* Rename class file, DatasourceUtil to DataSourceUtils.
2021-12-07 18:49:05 +08:00
张俊杰
463fc76bb7
[Fix-6809][dolphinscheduler-task-sql] Fix sql task can't send alert ( #7089 )
2021-12-01 14:08:40 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time ( #7054 )
2021-12-01 13:55:26 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
2021-11-14 20:22:13 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
2021-11-10 18:29:17 +08:00
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. ( #6071 ) ( #6361 )
...
* [feature-6071][plugin] add datasource plugin. (#6071 )
* remove the dependency of datasource in common and task-plugin module.
* the new datasource-api module provides data source connection acquisition.
* add sqlserver data source plug-in.
* add ds license
* fix check style
* update spi pom
* frx conflict resolution
* fix check style
* fix datasource ut
* fix code check style
* fix code check style
* fix datasource ut
* add pg hive mysql oracle clickhouse db2 datasource
* fix datasource package
* update datasource pom
* update datasource pom
* update datasource pom
* update datasource pom
* update pom
* update pom dep
* update pom dep
* fix license
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* add oracle package
* modify comments
* conflict resolution
* rename log comment
* rename log comment
2021-11-09 20:20:07 +08:00
mask
9b1984c244
[fix-6570][task] add sql task udf resource load ( #6570 ) ( #6571 )
...
* [fix-6570][task] add sql task udf resource load (#6570 )
-add sql task udf resource load
-Fix UdfFuncRequest serialization
* fix checkstyle
* add defaultFS in sqlTaskContext
* fix checkstyle
2021-10-21 15:23:49 +08:00
Kirs
09657d8576
[ #6337 ]When the SQL result reaches the limit value, increase the log result prompt ( #6528 )
2021-10-20 14:58:46 +08:00
Hua Jiang
8200bc1452
[Fix-6529] Improve the python task to solve the issue about getting the environment variables. ( #6531 )
...
* modify config
* improve python task
* improve python task
2021-10-18 17:33:16 +08:00
Kirs
bddf3e6f2c
[FIX-#6514]Task Class Load Err ( #6513 )
...
* [FIX-#6412]Task Class Load Err
* Task Class Load Err
2021-10-13 10:30:49 +08:00
Hua Jiang
a80fca70f7
[Fix-6497][Server] Fix the missing code about the feature of supporting multiple environments. ( #6509 )
2021-10-12 17:47:04 +08:00
百岁
d284e44d52
Rename tis to pigeon from backend layer ( #6437 )
...
* add mock getTaskLogName() for UT test pass
* make task-tis rename to task-pigeon
* rename module name 'tis' to pigeon
* rename module name 'tis' to pigeon,make sanity check pass
* remove execute console.log()
2021-10-01 14:54:04 +08:00
Kirs
1843e986f5
[BUG][Task-API]load codec class fail ( #6411 )
2021-09-29 16:03:06 +08:00
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT ( #6383 )
2021-09-28 12:00:43 +08:00
mask
a62a7a7216
[Fix][Task] sql limit param no default value ( #6337 ) ( #6387 )
...
* add default query sql limit
2021-09-28 11:38:22 +08:00
百岁
545173faa3
add mock getTaskLogName() for UT test pass ( #6369 )
2021-09-27 16:08:49 +08:00
百岁
34352c1bf8
Enhance using experience of DataX by introduce TIS ( #6229 )
...
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* fix stylecheck error
* make testCase pass
* make dev testCase pass
* add new java dependency Java-WebSocket
* modfiy TISParameters.java avoid to name confliction
* add InterruptedException checking
* reAdd async-http-client annotation
* in order to staisfy the coverage degree add test which has been removed
* make testCase pass
* make testCase pass
* add jacoco dependency
* make code duplications be more lower
* add Java-WebSocket dependency in root pom
* remove useless code comment
* make tis http apply path and post body configurable ,the params save in config.properites
* Remove the dangerous instance of double-checked locking
2021-09-26 18:40:41 +08:00
Kirs
be158328fb
[Bug #6306 ][Worker] task log path generated error. ( #6338 )
2021-09-24 19:55:11 +08:00
caishunfeng
61c6461994
[BUG-6331][WorkerServer] fix preparedStatement auto close by try with resource ( #6334 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-09-24 18:39:04 +08:00
Kirs
163a0696a4
[Bug #6306 ][Worker] task log path generated error. ( #6311 )
...
* [Bug #6306 ][Worker] task log path generated error.
* [Bug #6306 ][Worker] task log path generated error.
2021-09-23 16:51:58 +08:00
Kirs
9115062838
[Plugin][Task]Fix Task log may be lost ( #6294 )
2021-09-23 09:07:12 +08:00
Kirs
552fd17878
[Plugin][Task]Fix Yarn Task Kill Not valid ( #6293 )
2021-09-22 21:59:12 +08:00
Kirs
7e8febed4d
[Plugin][Task]Fix Sql Task NPE ( #6290 )
2021-09-22 18:07:29 +08:00
Kirs
fab4eae923
[SPI][Task]fix Task parse Err ( #6244 )
...
* [SPI][Task]fix Task parse Err
2021-09-17 18:17:15 +08:00
Kirs
fced3892ee
[Fix][Task] Wrong complement date ( #6009 ) ( #6186 )
...
* [Fix][Task] Wrong complement date (#6009 )
2021-09-17 17:04:51 +08:00
Kirs
b90152fc2e
[SPI][Task]add SQL Task ( #6237 )
...
Add Sql Task And Fix Task Parse Err
2021-09-17 13:10:44 +08:00
mask
259ec909d1
[Improvement-6198][task] remove Sparktaskchannelfactory duplicate declaration ( #6199 )
...
* remove Sparktaskchannelfactory duplicate declaration
2021-09-15 12:14:50 +08:00