Commit Graph

656 Commits

Author SHA1 Message Date
wind
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null (#7475)
* fix failover when node host is null

* add failover execute thread

* worker handle dead server

* fix task instance failover time check

* fix upgrade sql

* failover logic update

Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-28 19:32:10 +08:00
BaoLiang
605767e47b
[Fix-7654]Dependent node onChangeProjectCode error (#7672)
* fix #7654 Dependent node onChangeProjectCode error

* fix #7654 Dependent node onChangeProjectCode error

* fix #7654 Dependent node onChangeProjectCode error

* update ut
2021-12-28 18:09:42 +08:00
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue (#7491)
* add task group

* modify task group

* pull dev

* add license header

* modify code style

* fix code style

* fix sql error

* fix error

* fix test

* fix test

* fix test

* fix test

* fix code style

* fix ut

* code style

* fix unit test

* test ut

* ut

* add unittest

* test ut

* modify back ut

* majorization code

* fix conflict

* fix ut

* add task group api

* reset file

* fix ut

* fix lost column

* fix ut

* fix ut

* fix ut

* fix ut

* delete duplicate code

* fix code style 、name

* fix ut

* fix mapper

Co-authored-by: wangxj <wangxj31>
2021-12-24 16:37:59 +08:00
wind
41bf1a955e
[Improvement-7213][MasterServer] execute thread pool code optimization (#7258)
* threadpool optimization

* threadpool params

* rebase dev

* ut check fix

* add return

* rebase dev

* event loop

Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-16 23:45:32 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
Kerwin
4e6265df11
fix Property or field 'all' cannot be found (#7246) 2021-12-08 09:00:00 +08:00
Manhua
2dc09c627f
[Fix-5551][common] Fix ExecutionStatus for hive-on-Tez application (#6289)
* add ENDED state

* Update HadoopUtils.java
2021-12-05 14:03:29 +08:00
zwZjut
af9ed35295
[Feature][MasterServer] add cache for workGroup and schedule #6987 (#7161)
* to #6987

* to #6987

* to #6987

* to #6987

* to #6987

* to #6987: @Param -> @Name

* to #6987: fix Sonar

* to #6987: fix Sonar

Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
2021-12-04 16:59:05 +08:00
wind
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow (#7090)
* add cache manager for workflow

* [DS-6988][MasterServer] add cache manager for workflow

* cache evict code optimization

* test

Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-03 23:13:11 +08:00
wangxj3
2b519f31e9
[Feature-#6422] task group queue (#6722)
* add task group

* modify task group

* pull dev

* add license header

* modify code style

* fix code style

* fix sql error

* fix error

* fix test

* fix test

* fix test

* fix test

* fix code style

* fix ut

* code style

* fix unit test

* test ut

* ut

* add unittest

* test ut

* modify back ut

* majorization code

* fix conflict

* fix ut

Co-authored-by: wangxj <wangxj31>
2021-12-03 14:02:31 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time (#7054) 2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config (#6947)
* [DS-5229][fix] server port custom config
This closes #5229

* [DS-5229][feat] Implement server port custom config
This closes #5229

* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862)

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md

* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852)

Co-authored-by: caishunfeng <534328519@qq.com>

* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229

Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-01 00:21:03 +08:00
Kerwin
f0630bc9a0
Fix view task status module on project homepage cannot read property label of undefined (#6956) 2021-11-30 10:00:54 +08:00
Hua Jiang
a0955a078a
[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. (#7046) 2021-11-30 09:59:48 +08:00
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user (#6974)
* cache manager for tenant and user

* add test

* fix checkstyle

* add dependencies

* fix sonar check

* use spring boot cache configuration

* add license

* check style

* check style

* add license

* change proxy name to processor

* add license header

* check style

* add license file

* add license file

* add license

Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-30 08:29:25 +08:00
lvshaokang
1a88b1863b
[Improvement-5785][common] Fix code format (#6864)
* [DS-5785][style] Fix common module code format

* [DS-5785][style] Fix common module code format
2021-11-26 14:43:23 +08:00
kezhenxu94
ef0cb46661
Fix shell tools for database schema cannot run (#7003) 2021-11-26 11:10:38 +08:00
wind
e6239e808b
[DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852)
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-21 17:51:56 +08:00
JinYong Li
4bec792160
[Fix][Common] rewrite code generate,fix bit shift (#6914)
* rewrite code generate,fix bit shift

* fix ut

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* fix ut
2021-11-19 16:04:05 +08:00
Kerwin
802fc498b5
[Fix-#6605][Common] Fix the situation where the obtained string property is empty. (#6661)
* Fix the situation where the obtained string property is empty.

* recovery test

* update code style.

* megre dev.
2021-11-19 10:58:17 +08:00
chouc
f564687a89
[Improvement][Log] optimize task log path to reduce task running time (#6731)
* [Improvement][Log] optimize task log path to reduce task running time
* Reduce the creation of log directories


Co-authored-by: Kirs <acm_master@163.com>
2021-11-17 17:08:44 +08:00
OS
880c1d28ff
[Feature][master] change sub process id to code. (#6846)
* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* code style

* fix ut process

* fix ut process
2021-11-15 01:00:44 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
mask
a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490) (#6828)
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490)

* fix pom dep scope

* filter alert

* filter alert

* filter alert

* fix checkstyle

* remove hikaricp version

* update pom

* add properties

* add properties

* rename properties

* fix e2e fail

* rename properties
2021-11-14 13:02:58 +08:00
Assert
089f73ebe4
[Fix-6707][common] Fix complement data error because of global variables (#6759)
* [Fix-6707][common] Fix complement data error because of global variables

* [Fix-6707][common] Fix complement data error because of global variables

* [Fix-6707][common] Fix complement data error because of global variables

* [Fix-6707][common] Fix complement data error because of global variables

Co-authored-by: shangeyao <sgy960921>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
2021-11-13 17:06:15 +08:00
Jiajie Zhong
6045728428
[ci] Add checkstyle prohibit rule System.out.println (#6778)
* [ci] Add checkstyle prohibit rule System.out.println

* Correct RegexpSingleline place
2021-11-13 16:47:29 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage (#6767) 2021-11-10 18:29:17 +08:00
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. (#6071) (#6361)
* [feature-6071][plugin] add datasource plugin. (#6071)

* remove the dependency of datasource in common and task-plugin module.

* the new datasource-api module provides data source connection acquisition.

* add sqlserver data source plug-in.

* add ds license

* fix check style

* update spi pom

* frx conflict resolution

* fix check style

* fix datasource ut

* fix code check style

* fix code check style

* fix datasource ut

* add pg hive mysql oracle clickhouse db2  datasource

* fix datasource package

* update datasource pom

* update datasource pom

* update datasource pom

* update datasource pom

* update pom

* update pom dep

* update pom dep

* fix license

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* add oracle package

* modify comments

* conflict resolution

* rename log comment

* rename log comment
2021-11-09 20:20:07 +08:00
kezhenxu94
05bae0b226
Refactor alert plugins and simplify its usage (#6723) 2021-11-09 18:59:34 +08:00
kezhenxu94
02350ea37b
Fix datasource not work in tests and standalone server, and run all tests by default (#6743) 2021-11-09 16:57:26 +08:00
JinYong Li
be28d2c63b
[Bug][SnowFlakeUtils] fix snowFlake bug (#6744)
* fix snowFlake bug

* fix hash machine
2021-11-09 10:59:16 +08:00
kezhenxu94
63602bf77c
Clean up the utility codes (#6732) 2021-11-08 23:22:07 +08:00
Wenjun Ruan
0f94577d26
Use nanoTime replace currentTimemill to avoid clock backwards (#6740) 2021-11-08 22:26:17 +08:00
OS
24bd8a16e1
[Bug][Master] The task of dependent should use taskCode (#6730)
* fix #6682 The task of dependent should use taskCode

* fix #6682 The task of dependent should use taskCode
2021-11-07 23:20:35 +08:00
Wenjun Ruan
0afb780718
[Improvement-5807][Dao] Use h2 in UT (#6500)
* Use H2 in UT

* change scope

* Change profile name to h2

* Add datasource-h2.properties, datasource-postgresql.properties

* Fix UT

* Resolve conflict

* Remove startDatabase
2021-11-07 23:19:19 +08:00
kezhenxu94
5400abc74f
Refactor registry plugin and simplify its usage (#6712) 2021-11-07 15:18:10 +08:00
wind
72daba58cd
[DS-6694][Master] Reduce the refresh resource interval of LowerWeight (#6695)
* [DS-6694][Master] Reduce the refresh resource interval of LowerWeight

* add server node update event handle

Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-05 19:10:36 +08:00
wangxj3
dc8b87e473
[Feature-#6268][server-master] Serial execte process (#6185)
* add serial processInstance

* fix bug

* add test

* fix code style

* fix style code

* add sql

* fix sql error

* add api

* add test

* fix code style

* modify api

* delete column , fix mapper

* fix unimport

* fix test

* fix bug of missing param for Python

* fix code style

* fix test

* fix test

Co-authored-by: wangxj <wangxj31>
2021-11-05 17:25:45 +08:00
Jiajie Zhong
dd6ed36f65
Add Python API implementation of workflows-as-code (#6269)
* Init DS python SDK pydolphinscheduler: python code definition

* Doc first

* Add quick start and developer doc

* Java documentation change

* Add LICENSE-py4j.txt

* Add py4j to release-docs/LICENSE

* Move dependency version to parent pom

* Remove outdated code

* Add tenant parameter to tutorial
2021-10-31 20:35:46 +08:00
aCodingAddict
f60e3c219a
[DS-6606] [common] JSONUtils#getNodeString String type (#6618) 2021-10-30 16:02:02 +08:00
mask
9b1984c244
[fix-6570][task] add sql task udf resource load (#6570) (#6571)
* [fix-6570][task] add sql task udf resource load (#6570)

-add sql task udf resource load
-Fix UdfFuncRequest serialization

* fix checkstyle

* add defaultFS in sqlTaskContext

* fix checkstyle
2021-10-21 15:23:49 +08:00
wind
50f1766903
[DS-6499][WorkerServer] report busy state when worker is overload (#6512)
* [DS-6499][WorkerServer] report busy state when worker is overload
2021-10-13 11:45:56 +08:00
calvinit
59daf3c837
[DS-6276][fix] specify UTF-8 encoding to read file contents (#6466)
This closes #6276
2021-10-09 00:40:10 +08:00
百岁
d284e44d52
Rename tis to pigeon from backend layer (#6437)
* add mock getTaskLogName() for UT test pass

* make task-tis rename to task-pigeon

* rename module name 'tis' to pigeon

* rename module name 'tis' to pigeon,make sanity check pass

* remove execute console.log()
2021-10-01 14:54:04 +08:00
Shukun Zhang
6e453de241
[Feature] workflow level task dry run (#6104)
* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run
2021-09-30 11:18:06 +08:00
OS
c013b49e72
[Fix-6347][Master-API]fix bug #6347 the first schedule_time is error in complement data (#6389)
* fix bug #6347  complement data errors
2021-09-28 17:00:46 +08:00
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT (#6383) 2021-09-28 12:00:43 +08:00
百岁
34352c1bf8
Enhance using experience of DataX by introduce TIS (#6229)
* Enhance using experience of DataX by introduce TIS [Feature-5992]

* Enhance using experience of DataX by introduce TIS [Feature-5992]

* fix stylecheck error

* make testCase pass

* make dev testCase pass

* add new java dependency Java-WebSocket

* modfiy TISParameters.java avoid to name confliction

* add InterruptedException checking

* reAdd async-http-client annotation

* in order to staisfy the coverage degree add test which has been removed

* make testCase pass

* make testCase pass

* add jacoco dependency

* make code duplications be more lower

* add Java-WebSocket dependency in root pom

* remove useless code comment

* make tis http apply path and post body configurable ,the params save in config.properites

* Remove the dangerous instance of double-checked locking
2021-09-26 18:40:41 +08:00
HanayoZz
4625783479
[Fix-6072][common] Remove Generic method JsonUtils.toMap (#6283)
* [Fix][common] Remove Generic method JsonUtils.toMap, Replace by parseObj(String json, TypeReference type)
2021-09-23 23:06:53 +08:00
LiuBodong
c4375a54c1
[Fix][Task] Task lost resource information (#6292) (#6295)
Co-authored-by: liubodong <liubodong>
2021-09-23 22:23:17 +08:00