深刻
4c0993cdf4
[Fix-5781][UT] Fix test coverage in sonar ( #5817 )
...
* build(UT): make jacoco running in offline-instrumentation
issue: #5781
* build(UT): remove the jacoco agent dependency in microbench
issue: #5781
2021-07-26 22:04:36 +08:00
wangxj3
6df58e8d78
[Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies ( #5603 )
...
* add globalParams new plan with varPool
* add unit test
* add python task varPoolParams
Co-authored-by: wangxj <wangxj31>
2021-06-25 11:37:23 +08:00
kyoty
75813e7866
[bug fix] Some password attributes can be viewd directly in the log ( #5638 )
...
* fix the password display incorrectly
* update
* update
* fic code style
fic code style
* Update SensitiveDataConverterTest.java
* Update SensitiveDataConverter.java
* Update SensitiveDataConverter.java
* Update SensitiveDataConverterTest.java
* Update pom.xml
2021-06-18 19:10:57 +08:00
blackberrier
79eb2e85d7
[Improvement-5539][Master] Check status of taskInstance from cache ( #5572 )
...
* improvement:check status of taskInstance from cache
* issue5572 use timer instead of while&sleep; consider concurrent modification
* use computeifpresent instead of lock
* simplify getByTaskInstanceId function
* add ut for TaskInstanceCacheManagerImpl; fix bug in TaskInstanceCacheManagerImpl
* add Apache license header;add test class in root pom
2021-06-10 09:39:12 +08:00
Kirs
e2243d63be
[Feature-#3961][Registry]Registry-SPI ( #5562 )
...
[Feature#3961]Registry SPI
All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
2021-06-09 15:21:02 +08:00
Shiwen Cheng
b537f4eb26
[Improvement][Docker/K8s] Support more configurations and service accesses, reduce the duplication configurations, improve image for python and the compatibility of python home ( #5393 )
...
* [Improvement][Docker] Support more configurations and improve image for python
* [Improvement][K8s] Support more configurations, more service accesses, and reduce the duplication configurations
* [Improvement][Python] Improve the compatibility of python home
* [Improvement][Install] Fix install config and install.sh
* [Improvement][Install] Fix workflow ut
* [Improvement][Docker] Optimize docker-swarm/check script
* [Improvement][DB] Update default username and password of database
* Update comments in master/worker.properties
* Specify version for all images
* [Improvement][Docker] Optimize PS1 and WORKDIR
* [Improvement][SQL] Reduce heap size to 64m in create-dolphinscheduler.sh and upgrade-dolphinscheduler.sh
* [Fix-5431][K8s] Fix master and worker cannot get the right address with custom DNS
2021-05-24 00:03:35 +08:00
QuakeWang
46660b58ed
[Test] Add SparkParameters test ( #5478 )
...
* Create SparkparametersTest.java
Add a test case
* Update and rename SparkparametersTest.java to SparkParametersTest.java
* Update pom.xml
add SparkParametersTest.java in pom file
2021-05-22 23:56:39 +08:00
zhuangchong
1b7dcea610
[Improvement-5440][Maven] Project module maven dependency optimization ( #5441 )
...
* Optimize dependencies between modules.
* update module-alert dependency.
* add repository in the pom xml.
* update Central Repository in the pom.xml.
2021-05-19 10:05:00 +08:00
baoliang
156dadac69
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 10:08:28 +08:00
ruanwenjun
25727d2d95
[Improvement][Alert] Replace HSSFWorkbook ( #5400 )
...
* [Improvement][Alert] Replace HSSFWorkbook
* add license
2021-05-10 09:57:49 +08:00
zhuangchong
9c77faa8ac
[Improvement-5058][Master] The master-server creates the DB data source twice ( #5139 )
...
* reduce the number of data sources created twice by master server.
* update ProcessAlertManagerTest test class to increase coverage.
* add DolphinSchedulerPluginLoaderTest test ignore.
Co-authored-by: zhuang chong
2021-05-10 09:56:17 +08:00
QuakeWang
f88bbbd637
[Improvement] Create RollViewLogRequestCommandTest.java ( #5417 )
...
* Create RollViewLogRequestCommandTest.java
Add a test case about RollViewLogRequestCommand
* Update pom.xml
add the test files in the parent pom
2021-05-07 20:58:23 +08:00
baoliang
b594afc94c
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/procedure/ProcedureTask.java
2021-05-07 01:38:31 +08:00
ruanwenjun
ee95d07a90
[Feature][Alert plugin] Support slack alert ( #5382 ) ( #5384 )
...
* [Feature][Alert plugin] Support slack alert (#5382 )
* fix code smell
* add markdown table
2021-04-30 10:11:32 +08:00
zhuangchong
88a07f7b55
[Feature-4093][server] Support for stored procedures and stored function calls and data source supports DB2 ( #4094 )
...
* Support for stored procedures and stored function calls and data source supports DB2.
Co-authored-by: zhuangchong <zhuangchong8@163.com>
2021-04-29 18:29:44 +08:00
baoliang
746d9eca1d
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionVersionServiceImpl.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectServiceImpl.java
# dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionControllerTest.java
# dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProcessDefinitionVersionServiceTest.java
# dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/model/TaskNode.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/processdure/ProcedureTask.java
# dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
2021-04-26 14:14:05 +08:00
Shiwen Cheng
6b8461e901
Revert "[Improvement][DAO] CreateDolphinScheduler ( #5357 ) ( #5358 )" ( #5383 )
...
This reverts commit b108ac43b8
2021-04-25 23:29:00 +08:00
ruanwenjun
e6d8da484f
[Improvement][API] Create Datasource #5083 ( #5116 )
...
* [Improvement][API] Create Datasource #5083
* Refact create/ update datasource API
* fix Code Smells
* fix code smell
* resolve code smell
* use ConnectionParam to replace BaseDatasource
* solve license
* add ut
* code check
* add ut
* fix ut coverage
* fix ut
2021-04-25 13:51:59 +08:00
Shiwen Cheng
2e1d6975fd
[Improvement][Build] Optimize long release name and remove unused nginx release and upgrade spring boot version ( #5308 )
...
* [Build] Optimize long release name and remove unused nginx release
* [Build] Upgrade spring boot version
* [Improvement] Improve code style
2021-04-25 09:55:38 +08:00
ruanwenjun
b108ac43b8
[Improvement][DAO] CreateDolphinScheduler ( #5357 ) ( #5358 )
...
* [Improvement][DAO] Create CreateDolphinScheduler (#5357 )
* add ut
* fix code smell
2021-04-23 22:03:47 +08:00
Wenjun
abdd2337b1
[Bug][worker] Fix sql task query result is empty ( #5322 ) ( #5324 )
...
* [Bug][worker] Fix sql task query result is empty (#5322 )
* add ut
* remove duplicate
* add ut
2021-04-22 12:02:32 +08:00
didiaode18
f82da57277
There is a vulnerability in postgresql 42.1.4 ,upgrade recommended #5292 ( #5318 )
...
* There is a vulnerability in postgresql 42.1.4 ,upgrade recommended #5292
2021-04-20 15:04:29 +08:00
zhuangchong
fe144e46a3
[Improvement-4898][API] the paging query interface validates page params ( #4899 )
...
* fix api list-paging method check page params.
* update process definition version class check page params.
* add ProcessInstanceControllerTest in the pom xml file.
* solve sonar exception.
* add project controller test class.
* update create project return result
* update project controller test class code style.
* update sonar problem.
* update QueueController test class.
2021-04-19 11:01:52 +08:00
Shiwen Cheng
e92e29ef9a
[Fix-5309][Common] Fix the wrong memory ( #5312 )
...
* [Fix-5309][Common] Fix the wrong memory
* [Fix-5309][Common] Fix unit test and remove unused code
2021-04-19 10:24:54 +08:00
Shiwen Cheng
94a08c8f32
[Improvement] Remove incubating ( #5305 )
2021-04-16 22:56:51 +08:00
wenjun
7843ed4087
[Improvement][ApiServer] Traffic limit #5235 ( #5307 )
...
* [Improvement][ApiServer] Traffic limit #5235
* code style check
2021-04-16 22:41:02 +08:00
Kirs
ac15f45b0b
Update the main repository link ( #5286 )
2021-04-16 10:56:48 +08:00
Shiwen Cheng
e39409a412
[Improvement-5275][Server] Fix wrong excludeFilters and move zk/ZKMasterClient and registry/ServerNodeManager into master ( #5278 )
...
* [Improvement-5275][Server] Move zk/ZKMasterClient and registry/ServerNodeManager into master
* [Improvement-5275][Server] Fix code smells
* [Improvement-5275][Server] Add ZKMasterClientTest
2021-04-16 00:17:21 +08:00
wenjun
f3dfaacca2
[Improvement][Service] Optimize query log ( #5205 )
...
* [Improvement][Service] Optimize query log
* remove IPUtils
2021-04-15 20:25:01 +08:00
Shiwen Cheng
1c1f6663a6
[Improvement][WorkerGroup] Optimize and reduce database and zookeeper queries and replace worker address input with multiple-select ( #5182 )
...
* [Improvement][Server] Optimize and reduce database and ZooKeeper query
* [Improvement][WorkerGroup] Replace worker address input with multiple-select
* [Improvement][WorkerGroup] Fix unit test
* [Improvement][API&UI] Optimize long host to short host and long host display in k8s
2021-04-14 21:39:39 +08:00
wenjun
cc7a4446f5
[Improvement][Task] DataX Node doesn't Support Clickhouse as datasource ( #5180 ) ( #5243 )
...
* [Improvement][Module Name] DataX Node doesn't Support Clickhouse as the datasource (#5180 )
* add ut
* add ut
* resolve code smell
2021-04-14 11:31:40 +08:00
lenboo
c8ac6f99fe
Merge remote-tracking branch 'upstream/dev' into spilit
...
# Conflicts:
# dolphinscheduler-common/src/main/resources/common.properties
2021-03-30 11:31:42 +08:00
wenjun
a8b47e1d4f
[Bug][Master] Typo in MasterConfig #5141 ( #5142 )
2021-03-26 09:47:21 +08:00
lenboo
010b49c933
merge from dev
2021-03-22 15:18:16 +08:00
Shiwen Cheng
8ac72e80e6
[Feature-5087][SqlTask] Add a switch to send mail and print head logs in SqlTask ( #5088 )
...
* [Feature-5087][SqlTask] Add the switch to send mail and print head logs in SqlTask
* [Feature-5087][SqlTask] Add SqlParametersTest
2021-03-19 15:07:39 +08:00
Shiwen Cheng
5856a12855
[Improvement-4984][Worker] Refactor and Improve worker load balance ( #4996 )
...
* [Improvement][Remote] Improve unit tests
* [Improvement][Worker] Improve worker load balance
* [Improvement][Worker] Fix code smells
* [Improvement][Worker] Rename weight to hostWeight
2021-03-18 10:03:58 +08:00
lenboo
cf63fc1fee
Merge remote-tracking branch 'upstream/dev' into spilit
2021-03-16 15:50:06 +08:00
Kirs
6bf5031f45
[FIX] fix MySQL datasource jdbc connect parameters ( #4851 )
...
* [FIX-CVE-2020-11974] fix MySQLDataSource Security
* remove useless code
* add ut
* A more reasonable way
* code style
* fix test error
* code style
* update exception msg
2021-03-16 14:40:07 +08:00
lenboo
a574d2e38c
Merge remote-tracking branch 'upstream/dev' into spilit
2021-03-12 10:44:43 +08:00
wenjun
ea6b1de120
[Improvement][worker] Optimize KillYarnJob (4939) ( #4943 )
2021-03-11 22:13:44 +08:00
lenboo
6ef54ebe9b
Merge remote-tracking branch 'upstream/dev' into spilit
2021-03-11 17:01:29 +08:00
Shiwen Cheng
beb5adb985
[Feature][MR] Support mapreduce name ( #5029 )
2021-03-11 12:35:08 +08:00
Simon
bd4b2f85a9
[Feature][jsonsplit]add ProcessTaskRelationServiceImplTest/TaskDefinitionServiceImplTest ( #4949 )
...
* Modify Project and ProjectUser Mapper
* Modify Project and ProjectUser Mapper
* project_code is bigint(20)
* modify ERROR name
* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation
* modify import/export processdefinition, add genProcessData
* fix ut and bug
* code style
* repalce project_id with code
* conflicts solve
* conflicts solve
* conflicts solve
* bugfix
* modify listResources mothod and remove getResourceIds mothod
* 1
* conflicts solve
* modify listResources mothod and remove getResourceIds mothod
* modify listResources mothod and remove getResourceIds mothod
* replace processDefinitionVersion with processDefinitionLog
* codestyle
* codestyle
* add ProcessTaskRelationServiceImplTest/TaskDefinitionServiceImplTest
* conflicts solve
* conflicts solve
* conflicts solve
* add ProcessTaskRelationServiceImplTest/TaskDefinitionServiceImplTest
* codestyle
2021-03-06 10:56:49 +08:00
zhuangchong
5bfe3fca67
[Fix][SqlTask] fix hive sqltask invalid connection parameters. ( #4917 )
...
* fix hive sqltask invalid connection parameters.
* add setConnParams test method.
* update base datasource test class code style.
* add BaseDataSourceTest in the pom xml.
* update hive conf list jdbc url stitching.
* update hive datasource test class.
2021-03-03 09:15:37 +08:00
lgcareer
43ba29a2dd
[Improvement-4624] When the server exist in the dead server list of zk,need stop service byself ( #4626 )
...
* [Improvement-4624] When the server exist in the dead server list of zk,need stop service byself
* [Improvement-4624]fix check style and add MaterRegistryTest
* [Improvement-4624]fix check style and add ZookeeperRegistryCenterTest
* [Improvement-4624]fix check style and add ZookeeperRegistryCenterTest
* [Improvement-4624]add RegisterOperatorTest
* [Improvement-4624]update RegisterOperatorTest
* [Improvement-4624]resolve code smell
* [Improvement-4624]revert LICENSE-@form-create-element-ui
2021-03-02 14:05:00 +08:00
CalvinKirs
b4c7e35340
Merge branch 'dev' into rpc
...
# Conflicts:
# tools/dependencies/known-dependencies.txt
2021-02-26 18:10:49 +08:00
Kirs
a9e1d316f5
[Improvement][SPI]Migrate related classes loaded by SPI classes to public modules ( #4586 )
...
* Migrate related classes loaded by SPI classes to public modules
* Migrate related classes loaded by SPI classes to public modules
* remove guava
* update guava
* update guava
* move to common model
* code style
* code style
* resolved conflicting file
* remove ignore test
* add jar
* fix ut error
* fix test
* fix error
* test
* exclude asm、resolver、aether-api
* code style
* remove default plugin jar config
* add LICENSE file
* add version
* add version
* remove ignore ut
* fix ut error
* add ut
* code style
* add license
2021-02-25 11:39:46 +08:00
dailidong
551759965d
Merge pull request #4743 from apache/params-trans
...
[Feature-3805][mater-worker-ui] The implement of passing variables between tasks
2021-02-23 19:06:13 +08:00
guohaozhang
ffcb1c22e1
[Improvement-#3735] Make task delayed execution more efficient ( #4812 )
...
* [Improvement-3735] improve implementation of delay task execution
* [Improvement][worker] delay task compatible with dev branch and fix test
Co-authored-by: vanilla111 <1115690319@qq.com>
2021-02-23 10:10:11 +08:00
wangxj3
874ca07e6e
[Feature-3805][mater-worker-ui]fix function of passing variables between tasks bug ( #4811 )
...
* global initParam and set Param
* fix dataFormat error
* fix deal outParams bug
* fix code style
* fix code style
* fix code style
* fix code style
* fix code style
* fix code style
* fix code style
* add test
* fix code style (variable name)
* fix reset globalParams bug
* fix format bug
* fix SonarCloud Code Analysis
* add test
* fix code style
* fix code style
* modify test
* fix test Coverage
* fix code style
* 4 test
* add test pom
* add header
Co-authored-by: wangxj <wangxj31>
2021-02-20 14:32:46 +08:00
