xxsc0529
1199a916b3
[Feature-16132][Task Plugin] support for oceanbase datax task ( #16281 )
...
* Add support for oceanbase's datax task --feat
* fix code style
---------
Co-authored-by: xiangzihao <460888207@qq.com>
2024-07-16 09:27:05 +08:00
Wenjun Ruan
ec6a3e0727
[DSIP-46] Unify the logic task params data structure between front-end and back-end ( #16193 )
2024-06-25 13:32:13 +08:00
Wenjun Ruan
5b6b0ceb31
Fix resource file usage(Delete Resource/ResourceUser which is deprecated) ( #15355 )
2023-12-23 22:47:59 +08:00
Wenjun Ruan
71ee1f0c3d
Support parse task output params under multiple log ( #15244 )
2023-12-01 10:16:36 +08:00
Wenjun Ruan
4aab0b234f
Use AdHoc datasource client in sqlTask ( #14631 )
...
* Use AdHoc datasource client in sqlTask
* Add method in DataSourceChannel to create PooledDataSourceClient and AdHocDataSourceClient
2023-07-26 08:58:50 +08:00
Jay Chung
de2d215aee
impv: Correct datax execute and python execute script name ( #14523 )
...
* impv: Correct datax execute and python execute script name
we use PYTHON_LAUNCHER for python script execute and
DATAX_LAUNCHER for datax script name
* Add pr number
* fix ut
* style
---------
Co-authored-by: xiangzihao <460888207@qq.com>
2023-07-20 09:13:55 +08:00
Wenjun Ruan
c30cca9d9a
Support execute shell in different interceptor ( #14582 )
2023-07-19 14:50:04 +08:00
Jeremy
9c92b4b240
fix: default datax plugin ( #14204 )
2023-06-25 16:18:38 +08:00
Rick Cheng
b49b91c45d
[Improvement-14331][common] Remove the duplicate ParameterUtils ( #14332 )
2023-06-14 15:00:54 +08:00
Jeremy
db6c061c8b
[Feature][plugin] Add databend datasource in datasource plugin ( #13866 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-04-24 21:28:14 +08:00
fuchanghai
ec29220977
[improve-#13966] mask datax log ( #13969 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-04-21 14:51:37 +08:00
Aaron Wang
d6fcf60534
[Improvement-13751][Worker] Support real-time pod log collection ( #13752 )
...
* Support real-time pod log collection
* datax task test have to mock ShellCommandExecutor with whole dependencies
* replace \n with EMPTY_STRING
* modify error log msg
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* throw exception in pod log collection thread
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-04-05 16:13:45 +08:00
Wenjun Ruan
a1861bc2df
Use DATAX_PYTHON to specify a datax python version ( #13849 )
...
(cherry picked from commit 752452ecbd2f8094b819eae219f3eec0f6a6427b)
2023-04-03 08:47:32 +08:00
fuchanghai
6218f818c0
[improve-#13665] improve datax parser if sql contain * ( #13680 )
...
* [improve-#13665] improve datax parser if sql contain *
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-03-07 11:16:53 +08:00
fuchanghai
79d3ed0744
Update DataxUtils.java ( #13604 )
2023-02-22 19:01:43 +08:00
fuchanghai
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2023-02-22 14:22:49 +08:00
fuchanghai
37939add0c
[fix-13492]add single quotes ( #13534 )
...
* [fix-13492]add single quotes
* [improve-#13201] fix test error
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-02-09 19:02:27 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
Wenjun Ruan
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
Wenjun Ruan
92a9b7e243
Use System#lineSeparator to replace /n ( #12889 )
2022-11-15 13:57:45 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
Yann Ann
3bef85f546
fix datax NPE issue ( #12388 )
2022-10-18 09:03:16 +08:00
fuchanghai
b7bd8d7803
[Improvement] add presto in datax ( #12371 )
2022-10-16 17:19:29 +08:00
Eric Gao
875682d267
[Improvement][Test] Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example ( #12299 )
...
* Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example
2022-10-11 11:54:38 +08:00
fuchanghai
e101a5cb2c
[fix-#11815] fix ck column names contain special characters ( #11818 )
...
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-09-08 15:09:17 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
fuchanghai
f8b9aad239
[feature-#11689] change replaceAll to replace in method of tryExecuteSqlResolveColumnNames ( #11696 )
2022-09-05 09:54:18 +08:00
Wenjun Ruan
1b120e3a59
Refactor worker execute task process ( #11540 )
...
* Refactor worker execute task process
2022-08-26 13:33:51 +08:00
fuchanghai
d8d5d392a7
[feature-11408] add hive option ( #11493 )
2022-08-20 08:25:53 +08:00
lugela
e5d06c40a5
[Feature-10182][task-datax] ( #11387 )
...
In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase
2022-08-17 10:01:20 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
zhuxt2015
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters ( #10950 )
...
* datax task plugin support custom parameters
2022-07-15 09:27:23 +08:00
WangJPLeo
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2022-06-30 22:45:25 +08:00
zhuxt2015
7e7df1d8ff
[Bug] [Task] SQL task no exception stack trace in task log ( #10656 )
2022-06-29 09:57:05 +08:00
xiangzihao
bbba20d39a
change to common lang3 and fix checkstyle ( #10380 )
2022-06-09 15:56:42 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2022-05-31 15:22:41 +08:00
zhuxt2015
206b7c1c51
[Fix-9352][datasource-api]data source client uses a cache with an expiration time ( #9353 )
...
* datasource client cache with expiration time
* remove unused import
* fixed unit test
* fix
* fix
* fix
* fix
* fix
* fix
* fix
2022-04-27 16:56:24 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
2022-04-07 18:15:19 +08:00
Kerwin
d7d756e7b0
Fix DataX and Sqoop Task execute NullPointerException ( #9192 )
2022-03-26 20:15:41 +08:00
Kerwin
25fc1dcb5f
Fix the exception of task getting external resource data during execution ( #9134 )
2022-03-23 19:00:09 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
seagle
f1a0215978
[bug] Fix datax task type error parse clickhouse reader sql ( #8091 )
2022-01-18 16:53:25 +08:00
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
Zonglei Dong
0b9792db76
[Refactor]Rename class file, DatasourceUtil to DataSourceUtils. ( #7181 )
...
* Rename class file, DatasourceUtil to DataSourceUtils.
2021-12-07 18:49:05 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
2021-11-10 18:29:17 +08:00
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. ( #6071 ) ( #6361 )
...
* [feature-6071][plugin] add datasource plugin. (#6071 )
* remove the dependency of datasource in common and task-plugin module.
* the new datasource-api module provides data source connection acquisition.
* add sqlserver data source plug-in.
* add ds license
* fix check style
* update spi pom
* frx conflict resolution
* fix check style
* fix datasource ut
* fix code check style
* fix code check style
* fix datasource ut
* add pg hive mysql oracle clickhouse db2 datasource
* fix datasource package
* update datasource pom
* update datasource pom
* update datasource pom
* update datasource pom
* update pom
* update pom dep
* update pom dep
* fix license
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* add oracle package
* modify comments
* conflict resolution
* rename log comment
* rename log comment
2021-11-09 20:20:07 +08:00
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT ( #6383 )
2021-09-28 12:00:43 +08:00
Kirs
fab4eae923
[SPI][Task]fix Task parse Err ( #6244 )
...
* [SPI][Task]fix Task parse Err
2021-09-17 18:17:15 +08:00
