test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-04 05:09:48 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
7,667 Commits 27 Branches 59 Tags 276 MiB
938f13b568
Commit Graph

820 Commits

Author SHA1 Message Date
fuchanghai
7bf3e3cdd6
[improve-#13201] update pid during running (#13206) 2023-02-22 14:22:49 +08:00
Aaron Wang
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes (#13550) 2023-02-21 23:26:21 +08:00
fuchanghai
701d67c831
[improve-#13045] after a submit failure, stop the processInstance to avoid an endless loop (#13051) 
* [improve-#13045] add max submit number of workflow

* [fix-13405] remove max times

* Update dolphinscheduler-master/src/main/resources/application.yaml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

---------

Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
 2023-02-20 14:10:59 +08:00
Rick Cheng
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS (#13332) 2023-02-13 16:38:26 +08:00
kezhenxu94
e368bde7ee
Publish Helm Chart snapshot to ghcr.io (#13546) 2023-02-11 12:40:35 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger (#13509) 2023-02-07 20:32:53 +08:00
Wenjun Ruan
b11431db95
Use single thread to refresh kerberos (#13456) 2023-02-01 20:41:05 +08:00
Wenjun Ruan
12dd60fa46
Fix task wake up failed will block the event (#13466) 2023-02-01 15:31:04 +08:00
Jay Chung
78e5569afd
[feat] Add type local turn on resource center by default (#13303) 2023-01-31 17:39:20 +08:00
Tq
78ba71bdbd
[Feature][task]add azure datafactory task type (#13417) 
* init data factory

* add task-type-config.yaml

* add data factory ui

* submit doc

* submit license

---------

Co-authored-by: devosend <devosend@gmail.com>
 2023-01-30 11:15:20 +08:00
Wenjun Ruan
713fdbc444
Split the upgrade code (#13436) 
1. Split the upgrade code into different version
2. Log the dml/ddl sql in origin friendly format
3. Fix ddl of 3.0.0 and 3.1.0
4. Fix search resource regular expression injection
 2023-01-20 20:48:07 +08:00
qianli2022
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId (#13184) 
* add sql

* add mapper

* add dao

* add excutor


Co-authored-by: qianl4 <qianl4@cicso.com>
 2023-01-18 17:58:32 +08:00
Tq
3a6e10397f
[feature][api-datasource]add redshift auth mode to get connection by access key (#13379) 
* submit redshift auth mode

* add front end

* add doc and fix frontend

Co-authored-by: devosend <devosend@gmail.com>
 2023-01-16 16:21:26 +08:00
Wenjun Ruan
6634ff98af
Add source env list in common.properties (#13317) 2023-01-04 20:53:52 +08:00
lenian
bcf03ad4d1
[Feature-9467] add DAMENG DataSource (#12860) 
* [Feature-9467][DataSource] add DAMENG （达梦） data source

* [Feature-9467] add DM datasource doc (Issue #9467)

* [Feature-9467] add DM IDataBaseOptionKeys

* [Feature-9467] add DM DataSource: DmDataSourceChannelFactoryTest remove testGetDmConnection method

* [Feature-9467] modification package dm to dameng;

* [Feature-9467] add Dameng DmJdbcDriver18 License

* [Feature-9467] add DM DataSource: remove DruidDataSourceClient

Co-authored-by: lenian <yuenian@mananacare.com>
 2023-01-04 15:00:24 +08:00
Aaron Wang
ccad56e88e
[Improvement][Master] Validate same content of input file when using task cache (#13298) 
* support file content checksum

* fix inject null storageOperate bug
 2023-01-03 11:38:13 +08:00
Wenjun Ruan
fd3afd84ad
Remove alert when delete workflow instance (#13281) 2022-12-28 11:40:12 +08:00
Wenjun Ruan
14ec4a2398
Remove dao module in worker (#13242) 2022-12-22 12:25:29 +08:00
Tq
f0dd4f081f
[Feature][task-plugins-12820] azure azure-sql datasource plugins (#13209) 
* submit azure sql code

* change mssql-jdbc version

* merge changes to delete other

* merge changes to delete other

* add dependencies
 2022-12-19 10:27:03 +08:00
JieguangZhou
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks (#13194) 
* Supports task instance cache operation

* add task plugin cache

* use SHA-256 to generate key

* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>

* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>

* Optimizing database Scripts

* Optimize clear cache operation

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
 2022-12-18 18:17:09 +08:00
JieguangZhou
e4b9b67255
Allow execute task in workflow instance (#13103) 2022-12-13 16:43:44 +08:00
Wenjun Ruan
70ccffeee2
Format task parameter as pretty json (#13173) 2022-12-13 16:30:21 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 (#13002) 2022-12-10 23:50:19 +08:00
insist777
1064680ded
[feat][API] New restful API for workflow state (#13031) 2022-12-01 14:21:01 +08:00
rickchengx
38b876733c
[Feature-10498] Mask the password in the log of sqoop task (#11589) 2022-11-24 14:54:54 +08:00
insist777
b40c63f02d
[Feature][DataSource]Add trino datasource plugin (#12911) 2022-11-23 10:09:02 +08:00
simsicon
43d69ae157
Upgrade clickhouse jdbc driver (#12639) 2022-11-11 13:38:58 +08:00
Wenjun Ruan
ff59acd02f
Splite cpu/memory info in OSUtils#isOverload (#12663) 2022-11-02 21:15:03 +08:00
Wenjun Ruan
9e0c9af1a5
Fix the waiting strategy cannot recovery if the serverstate is already in running (#12651) 2022-11-02 14:06:01 +08:00
amao
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init (#11759) 
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
 2022-10-31 22:35:11 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds (#12197) 
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.

Co-authored-by: gabrywu <gabrywu@apache.com>
 2022-10-31 16:52:53 +08:00
Wenjun Ruan
a0d37fc7ae
Set tenantDir permission (#12486) 2022-10-23 21:54:01 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils (#12502) 
* improve JSONUtil

* merge spi.utils into common.utils

* split constants && remove some copy utils method

* remove StringUtils
 2022-10-23 21:23:54 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE (#12449) 2022-10-20 14:26:56 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412) 
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
 2022-10-19 11:08:04 +08:00
Eric Gao
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom (#12311) 
* move mockito dependencies from sub-modules to root pom

* Add check in CI to block the usage of powermock
 2022-10-14 12:06:51 +08:00
Yann Ann
b357781df2
[Migrate][Test] Migrate junit4 -> junit5 test cases in common, service, and spi modules (#12363) 2022-10-14 11:06:29 +08:00
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes (#12321) 2022-10-13 10:23:27 +08:00
Yann Ann
17cd644506
[Fix-#11669][Workflow Instance Page] Fix the duration in Workflow Instance page. (#12264) 
* fix process duration issue

* use processInstance.getState().isFinished() replace new isFinish function

* add WorkflowUtils
 2022-10-13 09:50:31 +08:00
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project (#12244) 
* Fully remove the usage of powermock from the whole project

* Upgrade org.reflections to 0.10.12
 2022-10-10 10:43:38 +08:00
calvin
b4947b471c
fix this issue (#12281) 2022-10-10 09:34:32 +08:00
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones (#12239) 
* fix DS support cross time zone, use timestamp replace date
 2022-10-01 12:50:42 +08:00
Tq
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts (#12126) 
* fix htrace dependency conflicts

* replace htrace-core with hbase-noop-htrace
 2022-09-26 13:49:08 +08:00
Yann Ann
289e1ecdc6
[Bug] [Common] update IPv4 Pattern (#11762) 
* [Bug] [Common] IPv4 Pattern is wrong. #11758
 2022-09-26 09:10:22 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
JieguangZhou
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use (#12071) 
optimize code
 2022-09-21 10:45:05 +08:00
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center (#11708) 
* Add support for Alibaba Cloud OSS as storage of resource center

* Add dependency licenses

* Refactor code, add unit tests and related docs
 2022-09-20 22:44:06 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task (#11670) 
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
 2022-09-17 13:40:56 +08:00
Wenjun Ruan
d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
jackfanwan
71b4087421
Example Change the regular expression of the data source password (#11459) 2022-09-16 13:34:22 +08:00