Tandoy
846d408e66
[Improvement][Master]fix typo ( #5934 )
...
·fix typo in MasterBaseTaskExecThread
2021-08-02 22:34:46 +08:00
kyoty
bca92157a0
[Fix-5808][Server] When we try to transfer data using datax between different types of data sources, the worker will exit with ClassCastException ( #5809 )
...
* bug fix
* fix bug
* simplify the code format
* add a new parameter to make it easier to understand.
2021-07-27 15:46:15 +08:00
深刻
4c0993cdf4
[Fix-5781][UT] Fix test coverage in sonar ( #5817 )
...
* build(UT): make jacoco running in offline-instrumentation
issue: #5781
* build(UT): remove the jacoco agent dependency in microbench
issue: #5781
2021-07-26 22:04:36 +08:00
Hua Jiang
4a68bfbe1c
[Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. ( #5867 )
...
* provide two system parameters to support the rest of type of tasks
* provide two system parameters to support the rest of type of tasks
* improve test conversion
2021-07-21 18:36:20 +08:00
wen-hemin
9ae2266cd4
fix: Remove unused class ( #5833 )
...
Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-07-17 16:23:39 +08:00
wen-hemin
596821a5b6
[Improvement][Api Module] refactor registry client, remove spring annotation ( #5814 )
...
* fix: refactor registry client, remove spring annotation
* fix UT
* fix UT
* fix checkstyle
* fix UT
* fix UT
* fix UT
* fix: Rename RegistryCenterUtils method name
Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-07-14 11:54:41 +08:00
wen-hemin
e78344df2e
fix: Remove duplicate "registryClient.close" method calls ( #5805 )
...
Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-07-13 01:14:07 +08:00
傅双波
1e3ff41928
fix ( #5803 )
...
Co-authored-by: shuangbofu <fusb@tuya.com>
2021-07-12 18:04:17 +08:00
kyoty
7bffe0ac85
[Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. ( #5796 )
...
* improve timestamp format
make the startime in the log of httptask to be easier to read.
* fix bad code smell and update the note.
2021-07-12 14:31:48 +08:00
Wenjun Ruan
30af55b82a
[Improvement][Worker] Task log may be lost #5775 ( #5783 )
2021-07-10 01:14:59 +08:00
kyoty
626c47399a
[FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE
2021-07-10 01:08:16 +08:00
Hua Jiang
9fd5145b66
[Improvement-5773][server] need to support two parameters related to task ( #5774 )
...
* add some new parameter for task
* restore official properties
* improve imports
* modify a variable's name
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
2021-07-09 17:00:32 +08:00
Tanvi Moharir
ab527a5e5a
[Improvement#5741][Worker] Improve task process status log ( #5776 )
2021-07-09 00:28:00 +08:00
wangxj3
7a18adae52
[FIX-#5721][master-server] Global params parameter missing ( #5757 )
...
Co-authored-by: wangxj <wangxj31>
2021-07-07 17:34:38 +08:00
wangxj3
6df58e8d78
[Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies ( #5603 )
...
* add globalParams new plan with varPool
* add unit test
* add python task varPoolParams
Co-authored-by: wangxj <wangxj31>
2021-06-25 11:37:23 +08:00
ji04xiaogang
3b80760c42
Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log ( #5691 )
...
Co-authored-by: shenglm <shenglm840722@126.com>
2021-06-25 09:13:26 +08:00
Kirs
f508788689
[BUG-#5678][Registry]fix registry init node miss ( #5686 )
2021-06-24 15:23:29 +08:00
Shiwen Cheng
93f1d4df7d
[Improvement-5405][Docker/K8s] Adapt to registry SPI ( #5681 )
2021-06-23 11:30:15 +08:00
Squidyu
664206cb81
wrong word: 'WAITTING' ->'WAITING' ( #5682 )
...
* Add:Name verification remove the first and last spaces.
* Update: wrong word: 'WAITTING' ->'WAITING'
Co-authored-by: Squid <2824638304@qq.com>
2021-06-22 21:56:21 +08:00
kyoty
75813e7866
[bug fix] Some password attributes can be viewd directly in the log ( #5638 )
...
* fix the password display incorrectly
* update
* update
* fic code style
fic code style
* Update SensitiveDataConverterTest.java
* Update SensitiveDataConverter.java
* Update SensitiveDataConverter.java
* Update SensitiveDataConverterTest.java
* Update pom.xml
2021-06-18 19:10:57 +08:00
justoy
0d7c32a1e8
Update TaskInstanceCacheManagerImpl.java ( #5656 )
2021-06-17 18:45:34 +08:00
zhuangchong
3026f04d85
[Improvement-5452][Task] ds flink task support submit a PyFlink job via the CLI ( #5453 )
...
* flink task support submit a PyFlink job via the CLI.
* optimize attribute name.
* Modify pyflink parameter judgment logic
2021-06-17 15:19:25 +08:00
LOUKHNATI Mohamed Khalil
86ffc1f3f2
[Bug][WorkerServer] SqlTask NullPointerException ( #5556 )
...
* [Bug][WorkerServer] add null checks to sqlParamsMap and sqlParamsMap.get(i), Add test to verify it
* Change Test Name
* Fix Code style issues, Modify checking null for Sql params in sqlParamsMap with clearer logging.
2021-06-16 18:21:41 +08:00
kyoty
67711442d5
[Improvement][SQL] Query return number In SQL should be configurable ( #5632 )
...
* [Improvement][SQL] Query return number should be configurable #5580
* Update SqlTaskTest.java
* Update sql.vue
2021-06-16 09:40:21 +08:00
zhuangchong
e2d6265e26
[Improvement][registry-plugin] Optimize registry plugin loading and initial installation #5635
2021-06-15 20:51:52 +08:00
Shiwen Cheng
8bf042ae6e
[Fix-5596][Python] Fix conflict between python_home and datax_home configuration in dolphinscheduler_env.sh ( #5612 )
2021-06-12 01:23:18 +08:00
blackberrier
79eb2e85d7
[Improvement-5539][Master] Check status of taskInstance from cache ( #5572 )
...
* improvement:check status of taskInstance from cache
* issue5572 use timer instead of while&sleep; consider concurrent modification
* use computeifpresent instead of lock
* simplify getByTaskInstanceId function
* add ut for TaskInstanceCacheManagerImpl; fix bug in TaskInstanceCacheManagerImpl
* add Apache license header;add test class in root pom
2021-06-10 09:39:12 +08:00
Kirs
e2243d63be
[Feature-#3961][Registry]Registry-SPI ( #5562 )
...
[Feature#3961]Registry SPI
All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
2021-06-09 15:21:02 +08:00
Chouc
75be09735a
[DS-5559][fix][Master Server] Master Server was shutdown but the process still in system ( #5588 )
...
* Close spring context to destory beans that has running thread
2021-06-04 13:27:18 +08:00
Shiwen Cheng
a5a0c7c5f8
[Fix-5570][Worker] Fix worker.groups in worker.properties is still commented and improve install scripts and configurations ( #5571 )
2021-06-02 11:21:01 +08:00
geosmart
56273319a2
[Bug][Master] remove check with executePath when kill yarn job ( #5552 )
...
* [Bug][Master] remove check with executePath when kill yarn job
* [Bug][Master] init execute path if not exist
* [Bug][Master] move createWorkDirIfAbsent out side of executePath calculate part
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
2021-05-29 12:14:46 +08:00
ruanwenjun
842c5400e6
[Improvement][Server] Must restart master if Zk reconnect ( #5210 ) ( #5211 )
2021-05-25 05:03:29 +08:00
ruanwenjun
9d0c816cee
[Bug][worker] task throw ConcurrentModifiedException #5528 ( #5530 )
...
* [Bug][worker] task throw ConcurrentModifiedException #5528
* fix code smell
2021-05-25 04:53:56 +08:00
Shiwen Cheng
b537f4eb26
[Improvement][Docker/K8s] Support more configurations and service accesses, reduce the duplication configurations, improve image for python and the compatibility of python home ( #5393 )
...
* [Improvement][Docker] Support more configurations and improve image for python
* [Improvement][K8s] Support more configurations, more service accesses, and reduce the duplication configurations
* [Improvement][Python] Improve the compatibility of python home
* [Improvement][Install] Fix install config and install.sh
* [Improvement][Install] Fix workflow ut
* [Improvement][Docker] Optimize docker-swarm/check script
* [Improvement][DB] Update default username and password of database
* Update comments in master/worker.properties
* Specify version for all images
* [Improvement][Docker] Optimize PS1 and WORKDIR
* [Improvement][SQL] Reduce heap size to 64m in create-dolphinscheduler.sh and upgrade-dolphinscheduler.sh
* [Fix-5431][K8s] Fix master and worker cannot get the right address with custom DNS
2021-05-24 00:03:35 +08:00
zhuangchong
1b7dcea610
[Improvement-5440][Maven] Project module maven dependency optimization ( #5441 )
...
* Optimize dependencies between modules.
* update module-alert dependency.
* add repository in the pom xml.
* update Central Repository in the pom.xml.
2021-05-19 10:05:00 +08:00
Yao WANG
bc22ae7c91
[Improvement][Task] Remove deprecated TaskRecordDao and simply the after() method in the AbstractTask class ( #5492 )
...
* remove the depreated data quality module
* fix lint
Co-authored-by: Yao <ywang46@paypal.com>
2021-05-18 17:00:03 +08:00
baoliang
dd80f98196
update sleep exception
2021-05-12 16:50:39 +08:00
baoliang
7aa8b518ac
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 13:55:27 +08:00
baoliang
2dd0d936fb
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 11:45:40 +08:00
baoliang
156dadac69
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/DependentExecute.java
2021-05-11 10:08:28 +08:00
JinyLeeChina
515fa4912e
[Feature][JsonSplit] Fix ProcessService ut ( #5435 )
...
* Fix ProcessService ut
* codeStyle
* Fix AlertPlugin ut
* Fix AlertPlugin ut
* Fix server module ut
* Fix server module ut
* Fix api ut
* codeStyle
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-05-10 23:17:56 +08:00
zhuangchong
9c77faa8ac
[Improvement-5058][Master] The master-server creates the DB data source twice ( #5139 )
...
* reduce the number of data sources created twice by master server.
* update ProcessAlertManagerTest test class to increase coverage.
* add DolphinSchedulerPluginLoaderTest test ignore.
Co-authored-by: zhuang chong
2021-05-10 09:56:17 +08:00
baoliang
c850b85484
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/procedure/ProcedureTask.java
2021-05-07 01:38:53 +08:00
baoliang
b594afc94c
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/procedure/ProcedureTask.java
2021-05-07 01:38:31 +08:00
zhuangchong
88a07f7b55
[Feature-4093][server] Support for stored procedures and stored function calls and data source supports DB2 ( #4094 )
...
* Support for stored procedures and stored function calls and data source supports DB2.
Co-authored-by: zhuangchong <zhuangchong8@163.com>
2021-04-29 18:29:44 +08:00
baoliang
746d9eca1d
Merge remote-tracking branch 'upstream/dev' into json_split
...
# Conflicts:
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionVersionServiceImpl.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectServiceImpl.java
# dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionControllerTest.java
# dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProcessDefinitionVersionServiceTest.java
# dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/model/TaskNode.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/processdure/ProcedureTask.java
# dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
2021-04-26 14:14:05 +08:00
JinyLeeChina
c3558965c1
update taskParams/add task delayTime/fix conditionType bug ( #5385 )
...
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-04-26 11:28:35 +08:00
ruanwenjun
e6d8da484f
[Improvement][API] Create Datasource #5083 ( #5116 )
...
* [Improvement][API] Create Datasource #5083
* Refact create/ update datasource API
* fix Code Smells
* fix code smell
* resolve code smell
* use ConnectionParam to replace BaseDatasource
* solve license
* add ut
* code check
* add ut
* fix ut coverage
* fix ut
2021-04-25 13:51:59 +08:00
bao liang
75d1ad645c
fix bug: 5349, work flow would running if task is killed by manual. ( #5362 )
...
Co-authored-by: baoliang <baoliang@analysys.com.cn>
2021-04-23 14:29:26 +08:00
JinyLeeChina
862565a7c7
[Feature][JsonSplit] Fix dependTask bug ( #5360 )
...
* update SnowFlake
* update processDefinite from processInstance
* update processDefinite from processInstance
* Fix task logger path
* Fix dependTask bug
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-04-23 14:23:04 +08:00
