JinYong Li
65744d4624
[Cherry-pick-dev][API] fix relation api bug ( #8207 )
...
* pick 8111
* fix relation delete (#8190 )
* pick 8190
2022-01-28 13:32:22 +08:00
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
Wenjun Ruan
f88def8ef7
[IMPROVEMENT-8178] Add Netty processor in Spring container ( #8179 )
...
* Add Netty processor in Spring container
2022-01-25 21:33:09 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
wind
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-25 09:37:57 +08:00
springmonster
e0a99dd909
[Bug-8053] Fix CronUtils.getMaxCycle return null ( #8086 )
2022-01-18 10:44:30 +08:00
天仇
90ea6ebb6f
[Improvement-7907][refactor] Optimization for query task instances list when build dag flow ( #7915 )
...
Optimization for query task instances list when build dag flow
This closes #7907
2022-01-16 22:30:09 +08:00
Wenjun Ruan
ecaeadea33
Fix global params cannot transport into sub_process ( #7969 ) ( #7973 )
2022-01-12 21:53:22 +08:00
JinYong Li
24a97fda75
[Bug] [dolphinscheduler-api]support workflow instance to definition ( #7930 )
...
* add task save and binds workflow
* add task update with upstream
* support workflow instance to definition
* fix ut
2022-01-11 14:41:45 +08:00
xiangzihao
7395b980d3
[Fix-7767]Task instance is not deleted while deleting the process instance ( #7768 )
...
* fix bug_7767
* fix bug_7767
* fix bug_7767
* fix bug_7767
* fix bug_7767
2022-01-03 15:52:50 +08:00
xiangzihao
fa906e7d01
[Fix-7753][Master] Failed to pass global parameters between parent and sub process ( #7755 )
...
* fix bug_7753
* fix bug_7753
* fix bug_7753
Co-authored-by: SbloodyS <sbloodys@qq.com>
2022-01-01 15:23:50 +08:00
aaronlinv
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
2021-12-30 16:29:22 +08:00
calvin
6b5db0ac5b
fix this issue ( #7683 )
2021-12-28 20:01:46 +08:00
wind
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-28 19:32:10 +08:00
wind
d75f22781b
fix workflow copy ( #7659 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-28 11:25:18 +08:00
xiangzihao
3d9d91ccc3
[Feature-7576][Master] Optimize complement task's date ( #7585 )
...
* feature_7576
* feature_7576
* feature 7576
* feature 7576
2021-12-25 12:22:22 +08:00
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue ( #7491 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
2021-12-24 16:37:59 +08:00
wind
d3533851a0
[Bug-7333][MasterServer] fix duplicate key task definition ( #7335 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-22 22:44:08 +08:00
Kerwin
f1b85cf892
Fix the missing project information in the process instance timeout alarm ( #7324 )
2021-12-10 18:55:10 +08:00
Kerwin
3fadfca085
fix in the process alarm message, projectid is 0 ( #7298 )
2021-12-10 12:25:35 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
wind
a80a7b3fcf
[Feature-6963][MasterServer] unified cache manager ( #7187 )
...
* unified cache
* reduce db select
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-07 15:55:22 +08:00
JinYong Li
2792c35d68
[Bug] [API] queryProcessDefinitionByCode bug ( #7221 )
2021-12-07 14:41:26 +08:00
wind
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow ( #7090 )
...
* add cache manager for workflow
* [DS-6988][MasterServer] add cache manager for workflow
* cache evict code optimization
* test
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-03 23:13:11 +08:00
wangxj3
2b519f31e9
[Feature-#6422] task group queue ( #6722 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
2021-12-03 14:02:31 +08:00
JinYong Li
b265297fc8
[Improvement][API] fix 6772 new api bug ( #7098 )
...
* add processDefinition releaseWorkflowAndSchedule
* add ProcessTaskRelationServiceImpl.moveTaskProcessRelation
* add dependency and subprocess judgement
* fix api bug
* fix codestyle
2021-12-02 14:06:39 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time ( #7054 )
2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config ( #6947 )
...
* [DS-5229][fix] server port custom config
This closes #5229
* [DS-5229][feat] Implement server port custom config
This closes #5229
* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862 )
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md
* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852 )
Co-authored-by: caishunfeng <534328519@qq.com>
* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229
Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-01 00:21:03 +08:00
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user ( #6974 )
...
* cache manager for tenant and user
* add test
* fix checkstyle
* add dependencies
* fix sonar check
* use spring boot cache configuration
* add license
* check style
* check style
* add license
* change proxy name to processor
* add license header
* check style
* add license file
* add license file
* add license
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-30 08:29:25 +08:00
wind
28872d8706
[DS-6891][MasterServer] reduce db operation when process instance running ( #6904 )
...
* remove taskInstanceCacheManager
* remove taskInstanceCacheManager
* [DS-6891][MasterServer] reduce db operation when process instance running
* [DS-6891][MasterServer] reduce db operation when process instance running
* fix test
* fix Transactional method call
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-19 18:31:24 +08:00
JinYong Li
4bec792160
[Fix][Common] rewrite code generate,fix bit shift ( #6914 )
...
* rewrite code generate,fix bit shift
* fix ut
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* fix ut
2021-11-19 16:04:05 +08:00
wind
595e4843d0
[DS-6849][MasterServer] fetch more commands and handle in parallel ( #6850 )
...
* [DS-6849][MasterServer] fetch more commands and handle in parallel
* add return
* handle command with check
* add test
* delete master prefix
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-19 12:03:49 +08:00
OS
1160b53940
[Feature][Master] change sub process id to code ( #6858 )
...
* feature 6597 change sub process id to code
* feature 6597 change sub process id to code
2021-11-15 20:27:19 +08:00
OS
880c1d28ff
[Feature][master] change sub process id to code. ( #6846 )
...
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* code style
* fix ut process
* fix ut process
2021-11-15 01:00:44 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
2021-11-14 20:22:13 +08:00
mask
a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP ( #6490 ) ( #6828 )
...
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490 )
* fix pom dep scope
* filter alert
* filter alert
* filter alert
* fix checkstyle
* remove hikaricp version
* update pom
* add properties
* add properties
* rename properties
* fix e2e fail
* rename properties
2021-11-14 13:02:58 +08:00
Jiajie Zhong
6045728428
[ci] Add checkstyle prohibit rule System.out.println ( #6778 )
...
* [ci] Add checkstyle prohibit rule System.out.println
* Correct RegexpSingleline place
2021-11-13 16:47:29 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
2021-11-10 18:29:17 +08:00
kezhenxu94
63602bf77c
Clean up the utility codes ( #6732 )
2021-11-08 23:22:07 +08:00
OS
24bd8a16e1
[Bug][Master] The task of dependent should use taskCode ( #6730 )
...
* fix #6682 The task of dependent should use taskCode
* fix #6682 The task of dependent should use taskCode
2021-11-07 23:20:35 +08:00
Wenjun Ruan
0afb780718
[Improvement-5807][Dao] Use h2 in UT ( #6500 )
...
* Use H2 in UT
* change scope
* Change profile name to h2
* Add datasource-h2.properties, datasource-postgresql.properties
* Fix UT
* Resolve conflict
* Remove startDatabase
2021-11-07 23:19:19 +08:00
kezhenxu94
5400abc74f
Refactor registry plugin and simplify its usage ( #6712 )
2021-11-07 15:18:10 +08:00
wangxj3
dc8b87e473
[Feature-#6268][server-master] Serial execte process ( #6185 )
...
* add serial processInstance
* fix bug
* add test
* fix code style
* fix style code
* add sql
* fix sql error
* add api
* add test
* fix code style
* modify api
* delete column , fix mapper
* fix unimport
* fix test
* fix bug of missing param for Python
* fix code style
* fix test
* fix test
Co-authored-by: wangxj <wangxj31>
2021-11-05 17:25:45 +08:00
wind
1ebab356f3
[DS-6660][Improvement] Change the registry default timeout ( #6663 )
...
* [DS-6660][Improvement] Change the registry default timeout
* [DS-6660][Improvement] Change the registry default timeout
* fix Double to Integer
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-04 22:31:13 +08:00
mask
9b1984c244
[fix-6570][task] add sql task udf resource load ( #6570 ) ( #6571 )
...
* [fix-6570][task] add sql task udf resource load (#6570 )
-add sql task udf resource load
-Fix UdfFuncRequest serialization
* fix checkstyle
* add defaultFS in sqlTaskContext
* fix checkstyle
2021-10-21 15:23:49 +08:00
OS
52a550b6ae
[BUG-6543][Master] process instance state is always running when failure task ( #6547 )
...
* fix-6543: process instance state is always running when failure task exists
* code style
* code style
2021-10-15 18:04:04 +08:00
wind
50f1766903
[DS-6499][WorkerServer] report busy state when worker is overload ( #6512 )
...
* [DS-6499][WorkerServer] report busy state when worker is overload
2021-10-13 11:45:56 +08:00
OS
cea8ae6991
6471: cache process definition in master ( #6511 )
2021-10-13 11:36:27 +08:00
wind
fea7874f95
[DS-6502][MasterServer] fix cmdParam NPE when constructProcessInstance on SCHEDULER mode ( #6504 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2021-10-12 14:42:49 +08:00
OS
db04a5b04d
[Feature-6471]Cache Process definition in master ( #6485 )
...
* feature-6471 Cache Process definition in master
2021-10-11 20:22:58 +08:00
