zhuangchong
5f85235462
[maven-release-plugin] prepare for next development iteration
2022-09-22 16:10:18 +08:00
zhuangchong
cb03ca8166
[maven-release-plugin] prepare release 3.0.1
2022-09-22 16:10:17 +08:00
zhuangchong
364cafe138
[maven-release-plugin] prepare for next development iteration
2022-09-19 13:36:46 +08:00
zhuangchong
17e36b7060
[maven-release-plugin] prepare release 3.0.1
2022-09-19 13:36:45 +08:00
Kerwin
c4c943277c
[3.0.1-prepare]cherry-pick [Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11994 )
...
* cherry-pick [Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2022-09-18 14:23:35 +08:00
JinYong Li
ab8f878631
fix heartBeatTaskCount bug ( #12024 )
2022-09-18 11:49:21 +08:00
Jiajie Zhong
889fd580e3
[maven-release-plugin] prepare for next development iteration
2022-08-05 18:16:57 +08:00
Jiajie Zhong
9efd1ace78
[maven-release-plugin] prepare release 3.0.0
2022-08-05 18:16:57 +08:00
Wenjun Ruan
f3250bf5fa
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
(cherry picked from commit 2be1d4bf0a
2022-07-19 12:23:20 +08:00
Wenjun Ruan
db31deb54f
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
(cherry picked from commit 66624c5c86
2022-07-19 11:41:25 +08:00
Wenjun Ruan
81cadd15d2
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
(cherry picked from commit 3ecbee3885
2022-07-19 11:10:08 +08:00
devosend
7ddaa2f47d
[maven-release-plugin] prepare for next development iteration
2022-06-28 16:41:31 +08:00
devosend
0a1b9bdd52
[maven-release-plugin] prepare release 3.0.0-beta-2
2022-06-28 16:41:30 +08:00
devosend
d68dcda2bb
[chore] pre-release change pom.xml
2022-06-28 08:24:21 +08:00
Jiajie Zhong
57ade38939
[maven-release-plugin] prepare release 3.0.0-beta-1
2022-05-26 20:26:32 +08:00
Jiajie Zhong
a9fa6b33a4
[chore] Change release version to 3.0.0-beta-1 ( #9957 )
2022-05-12 21:49:21 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
2022-04-07 18:15:19 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
aaronlinv
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
2021-12-30 16:29:22 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
wind
3a351da752
fix failover logic ( #7275 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-08 21:35:18 +08:00
zwZjut
f577b5bd30
to #7228 ( #7249 )
...
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
2021-12-07 20:05:57 +08:00
wind
a80a7b3fcf
[Feature-6963][MasterServer] unified cache manager ( #7187 )
...
* unified cache
* reduce db select
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-07 15:55:22 +08:00
wind
ff9bc806ac
[Bug-7206] [MasterServer] fix process isntance always running when task timeout ( #7207 )
...
* fix timeout
* add task timeout map to avoid repeated timeout event
* split task check list for retry and timeout
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-06 18:20:56 +08:00
Kerwin
36ebfe0984
[Implement][server]TaskProcessor code optimization ( #7146 )
...
* TaskProcessor code optimization
* fix code style
* update Collectors.toMap
2021-12-06 14:15:09 +08:00
SbloodyS
d83735ab51
[Fix-7037][dolphincheduler-server] process instance can not be killed when complement data ( #7140 )
...
* fix bug 7037
* update processComplementData
Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-05 15:03:52 +08:00
SbloodyS
8d39bf14b1
[Fix-7093][dolphincheduler-server] fix dependent node stuck in running ( #7116 )
...
* fix bug 7093
* remove redundant else
Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-05 13:52:48 +08:00
zwZjut
af9ed35295
[Feature][MasterServer] add cache for workGroup and schedule #6987 ( #7161 )
...
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987 : @Param -> @Name
* to #6987 : fix Sonar
* to #6987 : fix Sonar
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
2021-12-04 16:59:05 +08:00
wind
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow ( #7090 )
...
* add cache manager for workflow
* [DS-6988][MasterServer] add cache manager for workflow
* cache evict code optimization
* test
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-03 23:13:11 +08:00
wangxj3
2b519f31e9
[Feature-#6422] task group queue ( #6722 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
2021-12-03 14:02:31 +08:00
Kerwin
9fffbdb316
add TaskProcessorFactoryTest Ignore. ( #7130 )
2021-12-02 19:15:57 +08:00
zwZjut
c42531e39c
to #7108 ( #7109 )
2021-12-02 16:34:38 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time ( #7054 )
2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config ( #6947 )
...
* [DS-5229][fix] server port custom config
This closes #5229
* [DS-5229][feat] Implement server port custom config
This closes #5229
* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862 )
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md
* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852 )
Co-authored-by: caishunfeng <534328519@qq.com>
* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229
Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-01 00:21:03 +08:00
zwZjut
100a9ba3fc
to #7055 : tenant-auto-create: true ( #7056 )
...
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
2021-11-30 16:44:13 +08:00
zhang
c2585a1170
[BUG] master task log can't output repair ( #7029 )
...
* update task_definition_log field typ
update task_definition_log field task_params type,modify text to longtext to be consistent with the main table
* update t_ds_task_definition_log field task_params
update t_ds_task_definition_log field task_params
* [FIX:211128] master task log Can't output repair
Co-authored-by: crane <zhangbq3306@163.com>
2021-11-30 11:04:32 +08:00
wind
be3bd4c83d
[DS-7004][MasterServer]fix master still work when it lose zk connection ( #7045 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-30 11:00:17 +08:00
Hua Jiang
a0955a078a
[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. ( #7046 )
2021-11-30 09:59:48 +08:00
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user ( #6974 )
...
* cache manager for tenant and user
* add test
* fix checkstyle
* add dependencies
* fix sonar check
* use spring boot cache configuration
* add license
* check style
* check style
* add license
* change proxy name to processor
* add license header
* check style
* add license file
* add license file
* add license
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-30 08:29:25 +08:00
wind
e6fe39ea47
[DS-6961][MasterServer] batch dispatch ( #6962 )
...
* [DS-6961][MasterServer] batch dispatch
* fix test
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-25 12:55:37 +08:00
kezhenxu94
2de4a3cbb5
Add a demo docker-compose for testing the meter module ( #6938 )
2021-11-24 17:22:30 +08:00
wind
e6239e808b
[DS-6829][WorkerServer] skip create log dir and print log in dryRun model ( #6852 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-21 17:51:56 +08:00
wind
28872d8706
[DS-6891][MasterServer] reduce db operation when process instance running ( #6904 )
...
* remove taskInstanceCacheManager
* remove taskInstanceCacheManager
* [DS-6891][MasterServer] reduce db operation when process instance running
* [DS-6891][MasterServer] reduce db operation when process instance running
* fix test
* fix Transactional method call
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-19 18:31:24 +08:00
wind
595e4843d0
[DS-6849][MasterServer] fetch more commands and handle in parallel ( #6850 )
...
* [DS-6849][MasterServer] fetch more commands and handle in parallel
* add return
* handle command with check
* add test
* delete master prefix
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-19 12:03:49 +08:00
OS
653eae2419
[BUG][Master] process cannot finish and its status is always running ( #6886 )
...
* fix 6882: process cannot finish and its status is always running
* fix 6882: process cannot finish and its status is always running
2021-11-17 17:39:22 +08:00
chouc
f564687a89
[Improvement][Log] optimize task log path to reduce task running time ( #6731 )
...
* [Improvement][Log] optimize task log path to reduce task running time
* Reduce the creation of log directories
Co-authored-by: Kirs <acm_master@163.com>
2021-11-17 17:08:44 +08:00
