test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-05 05:38:30 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
5,515 Commits 27 Branches 59 Tags 276 MiB
ba2b2a67c2
Commit Graph

648 Commits

Author SHA1 Message Date
wind
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow (#7090) 
* add cache manager for workflow

* [DS-6988][MasterServer] add cache manager for workflow

* cache evict code optimization

* test

Co-authored-by: caishunfeng <534328519@qq.com>
 2021-12-03 23:13:11 +08:00
wangxj3
2b519f31e9
[Feature-#6422] task group queue (#6722) 
* add task group

* modify task group

* pull dev

* add license header

* modify code style

* fix code style

* fix sql error

* fix error

* fix test

* fix test

* fix test

* fix test

* fix code style

* fix ut

* code style

* fix unit test

* test ut

* ut

* add unittest

* test ut

* modify back ut

* majorization code

* fix conflict

* fix ut

Co-authored-by: wangxj <wangxj31>
 2021-12-03 14:02:31 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time (#7054) 2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config (#6947) 
* [DS-5229][fix] server port custom config
This closes #5229

* [DS-5229][feat] Implement server port custom config
This closes #5229

* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862)

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md

* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852)

Co-authored-by: caishunfeng <534328519@qq.com>

* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229

Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
 2021-12-01 00:21:03 +08:00
Kerwin
f0630bc9a0
Fix view task status module on project homepage cannot read property label of undefined (#6956) 2021-11-30 10:00:54 +08:00
Hua Jiang
a0955a078a
[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. (#7046) 2021-11-30 09:59:48 +08:00
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user (#6974) 
* cache manager for tenant and user

* add test

* fix checkstyle

* add dependencies

* fix sonar check

* use spring boot cache configuration

* add license

* check style

* check style

* add license

* change proxy name to processor

* add license header

* check style

* add license file

* add license file

* add license

Co-authored-by: caishunfeng <534328519@qq.com>
 2021-11-30 08:29:25 +08:00
lvshaokang
1a88b1863b
[Improvement-5785][common] Fix code format (#6864) 
* [DS-5785][style] Fix common module code format

* [DS-5785][style] Fix common module code format
 2021-11-26 14:43:23 +08:00
kezhenxu94
ef0cb46661
Fix shell tools for database schema cannot run (#7003) 2021-11-26 11:10:38 +08:00
wind
e6239e808b
[DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852) 
Co-authored-by: caishunfeng <534328519@qq.com>
 2021-11-21 17:51:56 +08:00
JinYong Li
4bec792160
[Fix][Common] rewrite code generate,fix bit shift (#6914) 
* rewrite code generate,fix bit shift

* fix ut

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* fix ut
 2021-11-19 16:04:05 +08:00
Kerwin
802fc498b5
[Fix-#6605][Common] Fix the situation where the obtained string property is empty. (#6661) 
* Fix the situation where the obtained string property is empty.

* recovery test

* update code style.

* megre dev.
 2021-11-19 10:58:17 +08:00
chouc
f564687a89
[Improvement][Log] optimize task log path to reduce task running time (#6731) 
* [Improvement][Log] optimize task log path to reduce task running time
* Reduce the creation of log directories


Co-authored-by: Kirs <acm_master@163.com>
 2021-11-17 17:08:44 +08:00
OS
880c1d28ff
[Feature][master] change sub process id to code. (#6846) 
* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* 6597 change sub process id to code.

* code style

* fix ut process

* fix ut process
 2021-11-15 01:00:44 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
mask
a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490) (#6828) 
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490)

* fix pom dep scope

* filter alert

* filter alert

* filter alert

* fix checkstyle

* remove hikaricp version

* update pom

* add properties

* add properties

* rename properties

* fix e2e fail

* rename properties
 2021-11-14 13:02:58 +08:00
Assert
089f73ebe4
[Fix-6707][common] Fix complement data error because of global variables (#6759) 
* [Fix-6707][common] Fix complement data error because of global variables

* [Fix-6707][common] Fix complement data error because of global variables

* [Fix-6707][common] Fix complement data error because of global variables

* [Fix-6707][common] Fix complement data error because of global variables

Co-authored-by: shangeyao <sgy960921>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
 2021-11-13 17:06:15 +08:00
Jiajie Zhong
6045728428
[ci] Add checkstyle prohibit rule System.out.println (#6778) 
* [ci] Add checkstyle prohibit rule System.out.println

* Correct RegexpSingleline place
 2021-11-13 16:47:29 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage (#6767) 2021-11-10 18:29:17 +08:00
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. (#6071) (#6361) 
* [feature-6071][plugin] add datasource plugin. (#6071)

* remove the dependency of datasource in common and task-plugin module.

* the new datasource-api module provides data source connection acquisition.

* add sqlserver data source plug-in.

* add ds license

* fix check style

* update spi pom

* frx conflict resolution

* fix check style

* fix datasource ut

* fix code check style

* fix code check style

* fix datasource ut

* add pg hive mysql oracle clickhouse db2  datasource

* fix datasource package

* update datasource pom

* update datasource pom

* update datasource pom

* update datasource pom

* update pom

* update pom dep

* update pom dep

* fix license

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* fix ut

* add oracle package

* modify comments

* conflict resolution

* rename log comment

* rename log comment
 2021-11-09 20:20:07 +08:00
kezhenxu94
05bae0b226
Refactor alert plugins and simplify its usage (#6723) 2021-11-09 18:59:34 +08:00
kezhenxu94
02350ea37b
Fix datasource not work in tests and standalone server, and run all tests by default (#6743) 2021-11-09 16:57:26 +08:00
JinYong Li
be28d2c63b
[Bug][SnowFlakeUtils] fix snowFlake bug (#6744) 
* fix snowFlake bug

* fix hash machine
 2021-11-09 10:59:16 +08:00
kezhenxu94
63602bf77c
Clean up the utility codes (#6732) 2021-11-08 23:22:07 +08:00
Wenjun Ruan
0f94577d26
Use nanoTime replace currentTimemill to avoid clock backwards (#6740) 2021-11-08 22:26:17 +08:00
OS
24bd8a16e1
[Bug][Master] The task of dependent should use taskCode (#6730) 
* fix #6682 The task of dependent should use taskCode

* fix #6682 The task of dependent should use taskCode
 2021-11-07 23:20:35 +08:00
Wenjun Ruan
0afb780718
[Improvement-5807][Dao] Use h2 in UT (#6500) 
* Use H2 in UT

* change scope

* Change profile name to h2

* Add datasource-h2.properties, datasource-postgresql.properties

* Fix UT

* Resolve conflict

* Remove startDatabase
 2021-11-07 23:19:19 +08:00
kezhenxu94
5400abc74f
Refactor registry plugin and simplify its usage (#6712) 2021-11-07 15:18:10 +08:00
wind
72daba58cd
[DS-6694][Master] Reduce the refresh resource interval of LowerWeight (#6695) 
* [DS-6694][Master] Reduce the refresh resource interval of LowerWeight

* add server node update event handle

Co-authored-by: caishunfeng <534328519@qq.com>
 2021-11-05 19:10:36 +08:00
wangxj3
dc8b87e473
[Feature-#6268][server-master] Serial execte process (#6185) 
* add serial processInstance

* fix bug

* add test

* fix code style

* fix style code

* add sql

* fix sql error

* add api

* add test

* fix code style

* modify api

* delete column , fix mapper

* fix unimport

* fix test

* fix bug of missing param for Python

* fix code style

* fix test

* fix test

Co-authored-by: wangxj <wangxj31>
 2021-11-05 17:25:45 +08:00
Jiajie Zhong
dd6ed36f65
Add Python API implementation of workflows-as-code (#6269) 
* Init DS python SDK pydolphinscheduler: python code definition

* Doc first

* Add quick start and developer doc

* Java documentation change

* Add LICENSE-py4j.txt

* Add py4j to release-docs/LICENSE

* Move dependency version to parent pom

* Remove outdated code

* Add tenant parameter to tutorial
 2021-10-31 20:35:46 +08:00
aCodingAddict
f60e3c219a
[DS-6606] [common] JSONUtils#getNodeString String type (#6618) 2021-10-30 16:02:02 +08:00
mask
9b1984c244
[fix-6570][task] add sql task udf resource load (#6570) (#6571) 
* [fix-6570][task] add sql task udf resource load (#6570)

-add sql task udf resource load
-Fix UdfFuncRequest serialization

* fix checkstyle

* add defaultFS in sqlTaskContext

* fix checkstyle
 2021-10-21 15:23:49 +08:00
wind
50f1766903
[DS-6499][WorkerServer] report busy state when worker is overload (#6512) 
* [DS-6499][WorkerServer] report busy state when worker is overload
 2021-10-13 11:45:56 +08:00
calvinit
59daf3c837
[DS-6276][fix] specify UTF-8 encoding to read file contents (#6466) 
This closes #6276
 2021-10-09 00:40:10 +08:00
百岁
d284e44d52
Rename tis to pigeon from backend layer (#6437) 
* add mock getTaskLogName() for UT test pass

* make task-tis rename to task-pigeon

* rename module name 'tis' to pigeon

* rename module name 'tis' to pigeon,make sanity check pass

* remove execute console.log()
 2021-10-01 14:54:04 +08:00
Shukun Zhang
6e453de241
[Feature] workflow level task dry run (#6104) 
* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run

* feat workflow level task dry run
 2021-09-30 11:18:06 +08:00
OS
c013b49e72
[Fix-6347][Master-API]fix bug #6347 the first schedule_time is error in complement data (#6389) 
* fix bug #6347  complement data errors
 2021-09-28 17:00:46 +08:00
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT (#6383) 2021-09-28 12:00:43 +08:00
百岁
34352c1bf8
Enhance using experience of DataX by introduce TIS (#6229) 
* Enhance using experience of DataX by introduce TIS [Feature-5992]

* Enhance using experience of DataX by introduce TIS [Feature-5992]

* fix stylecheck error

* make testCase pass

* make dev testCase pass

* add new java dependency Java-WebSocket

* modfiy TISParameters.java avoid to name confliction

* add InterruptedException checking

* reAdd async-http-client annotation

* in order to staisfy the coverage degree add test which has been removed

* make testCase pass

* make testCase pass

* add jacoco dependency

* make code duplications be more lower

* add Java-WebSocket dependency in root pom

* remove useless code comment

* make tis http apply path and post body configurable ,the params save in config.properites

* Remove the dangerous instance of double-checked locking
 2021-09-26 18:40:41 +08:00
HanayoZz
4625783479
[Fix-6072][common] Remove Generic method JsonUtils.toMap (#6283) 
* [Fix][common] Remove Generic method JsonUtils.toMap, Replace by parseObj(String json, TypeReference type)
 2021-09-23 23:06:53 +08:00
LiuBodong
c4375a54c1
[Fix][Task] Task lost resource information (#6292) (#6295) 
Co-authored-by: liubodong <liubodong>
 2021-09-23 22:23:17 +08:00
JinyLeeChina
07d6542feb
[Fix-6139][API] fix workFlowLineage relation (#6217) 
* fix bug of view-tree api

* fix bug of view-tree api

* fix ut

* fix ut

* fix master buildFlowDag error

* fix workFlowLineage relation

* fix ut

* fix ut

* fix ut

Co-authored-by: JinyLeeChina <297062848@qq.com>
 2021-09-15 17:42:21 +08:00
OS
8694b9c8f9
fix bug-6208 json parse error in sub process (#6211) 2021-09-15 10:29:26 +08:00
Jiajie Zhong
170280fab2
Simplify constants variables (#6183) 
* Combine `alter` and `registry` properties,
  bese on https://github.com/apache/dolphinscheduler/pull/6180
* Remove duplicate variable `SLASH` and `SINGLE_SLASH`
 2021-09-13 22:03:09 +08:00
JinyLeeChina
14e4d7086d
[Fix-6139][API] fix bug of view-tree api (#6188) 
* fix bug of view-tree api

* fix bug of view-tree api

* fix ut

* fix ut

Co-authored-by: JinyLeeChina <297062848@qq.com>
 2021-09-13 20:40:52 +08:00
kezhenxu94
ea2a9dbcb2
Clean up third-party files that are not mentioned in LICENSE (#6165) 2021-09-11 11:27:42 +08:00
kezhenxu94
a632be73c0
Clean up unused util classes... (#6154) 2021-09-10 00:35:39 +08:00
JinyLeeChina
e8ddc9103d
[Bug][API] fix genTaskCodeList return same code and save proces error (#6150) 
* fix mysql create sentence bug

* fix mysql create sentence bug

* fix genTaskCodeList return same code and save proces error

Co-authored-by: JinyLeeChina <297062848@qq.com>
 2021-09-09 22:01:22 +08:00
CalvinKirs
dbb4adf6c3 Merge remote-tracking branch 'upstream/dev' into task-end 
# Conflicts:
#	.asf.yaml
#	dolphinscheduler-alert/src/main/java/org/apache/dolphinscheduler/alert/utils/FuncUtils.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/SchedulerController.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java
#	dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/AlertGroupControllerTest.java
#	dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorService2Test.java
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/StateEventType.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Command.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ErrorCommand.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskDefinition.java
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ScheduleMapper.xml
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionMapper.xml
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/ConditionsTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/DependentTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/DependentTaskProcessor.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/WorkerServer.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/ConditionsTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/DependentTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/MasterBaseTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/MasterTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/SubProcessTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/WorkerManagerThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/python/PythonTask.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/WorkflowExecuteThreadTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/processor/TaskKillProcessorTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThreadTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/runner/WorkerManagerThreadTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/PythonCommandExecutorTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTaskTest.java
#	dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
#	dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/cron/CronUtils.java
#	dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java
#	dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/DolphinSchedulerPlugin.java
#	dolphinscheduler-standalone-server/pom.xml
#	dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-procedure/src/main/java/org/apache/dolphinscheduler/plugin/task/procedure/ProcedureTaskChannelFactory.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-python/src/main/java/org/apache/dolphinscheduler/plugin/task/python/PythonCommandExecutor.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-shell/src/main/java/org/apache/dolphinscheduler/plugin/task/shell/ShellTask.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopTask.java
#	dolphinscheduler-ui/package.json
#	dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/dag.scss
#	dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/dag.vue
#	dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/formModel.vue
#	sql/dolphinscheduler_h2.sql
 2021-09-08 16:44:07 +08:00