Commit Graph

5065 Commits

Author SHA1 Message Date
wen-hemin
15b2b1c0ec
[FixBug][Jsonsplit] When to save process definition, upload task code field #4417 (#5178)
* fix: task page, rename "description" to "desc"

* fix: The default value of field dependence is null.

* fix: frontend compile check

* fix: When create process, the version button is set to unavailable

* fix: sql condition bug

* fix: When to save process definition, upload task code field

Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-03-30 14:28:08 +08:00
lenboo
c8ac6f99fe Merge remote-tracking branch 'upstream/dev' into spilit
# Conflicts:
#	dolphinscheduler-common/src/main/resources/common.properties
2021-03-30 11:31:42 +08:00
JinyLeeChina
689106d114
transform taskCode from long to string (#5174)
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-03-30 11:07:28 +08:00
KonsonChow
9c3cec5bbd
[DS-4891][docs] update DolphinScheduler slack invitation url (#5167)
update DolphinScheduler slack invitation url

Co-authored-by: KonsonChow <KonsonChow@gmail.com>
2021-03-29 23:47:52 +08:00
wen-hemin
f960eacdcd
[FixBug][Jsonsplit] sql condition bug #4417 (#5171)
* fix: task page, rename "description" to "desc"

* fix: The default value of field dependence is null.

* fix: frontend compile check

* fix: When create process, the version button is set to unavailable

* fix: sql condition bug

Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-03-29 22:17:28 +08:00
wen-hemin
0edd783373
[FIxBug][JsonSplit] When create process, the version button is set to unavailable #5137 (#5168)
* fix: task page, rename "description" to "desc"

* fix: The default value of field dependence is null.

* fix: frontend compile check

* fix: When create process, the version button is set to unavailable

Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-03-29 17:24:01 +08:00
JinyLeeChina
a467fffd7b
[Feature][JsonSplit] fix taskId in json (#5165)
* modify checkDAGRing and ProcessService method

* merge

* modify dagRing

* modify process instance for project home page

* fix save process bug

* codeStyle

* Fix logical bug in saving process definition

* codeSytle

* Fix bug in interface of  queryProcessDefinitionList

* codeSytle

* Fix api bug"

* fix taskId in processDefinitionJson

* fix json taskId

* codeStyle

Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-03-29 14:22:40 +08:00
guohaozhang
a6a1b94df4
[Feature-#5128]Support turning off sudo permissions (#5129)
* [Feature-#5128]Support turning off sudo permissions

* [Feature-#5128] without sudo permission do not create tenant user
2021-03-29 11:07:39 +08:00
xiaojingXU
9c04396218
[Bug-5132][server] stop the take immediately if an exception occurs in the taskExecuteThread (#5133)
* [DS-5132][fix] stop the take immediately if an exception occurs in the taskExecuteThread
- let softkill() method return true when the process is stopped,else return false;
2021-03-29 10:41:51 +08:00
wenjun
573252f3e1
[Bug][Server] DB transaction failure #5150 (#5152) 2021-03-29 10:30:16 +08:00
wenjun
87ed189ef5
[Improvement] print exception when task execute (#5159) 2021-03-29 10:15:18 +08:00
zhuangchong
f6d62a4924
[Improvement-5144][UI] error v-show directives cannot be put on template tags (#5145)
* fix error v-show directives cannot be put on template tags.

* rerun e2e.
2021-03-26 16:42:13 +08:00
Shiwen Cheng
492b318bd3
[Fix][Server] Fix clear task execute path is related to master (#5123) 2021-03-26 10:11:56 +08:00
wenjun
a8b47e1d4f
[Bug][Master] Typo in MasterConfig #5141 (#5142) 2021-03-26 09:47:21 +08:00
lenboo
0914699fc5 Merge branch 'json_split' of https://github.com/apache/incubator-dolphinscheduler into spilit 2021-03-26 00:06:26 +08:00
lenboo
f05ba84968 Merge remote-tracking branch 'upstream/dev' into spilit
# Conflicts:
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProcessInstanceMapper.java
#	dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapperTest.java
2021-03-25 23:54:01 +08:00
Simon
cd19507271
[feature][jsonsplit]fix copy/export/import processDefinition bugs (#5146)
* Modify Project and ProjectUser Mapper

* Modify Project and ProjectUser Mapper

* project_code is bigint(20)

* modify ERROR name

* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation

* modify import/export processdefinition, add genProcessData

* fix ut and bug

* code style

* repalce project_id with code

* conflicts solve

* conflicts solve

* conflicts solve

* bugfix

* modify listResources mothod and remove getResourceIds mothod

* 1

* conflicts solve

* modify listResources mothod and remove getResourceIds mothod

* modify listResources mothod and remove getResourceIds mothod

* replace processDefinitionVersion with processDefinitionLog

* codestyle

* codestyle

* add mapper module ut

* codestyle

* fix ProcessInstanceMapperTest

* codestyle

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* fix ProcessInstanceMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix TaskInstanceMapperTest

* add TaskDefinitionLogMapperTest/TaskDefinitionMapperTest and bugfix

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add ProcessServiceTest

* 1

* 1

* fix copy/export/import processDefinition bugs

* 1

* codestyle
2021-03-25 18:27:47 +08:00
Simon
3c9b214da7
[Feature][jsonsplit]add ProcessServiceTest (#5097)
* Modify Project and ProjectUser Mapper

* Modify Project and ProjectUser Mapper

* project_code is bigint(20)

* modify ERROR name

* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation

* modify import/export processdefinition, add genProcessData

* fix ut and bug

* code style

* repalce project_id with code

* conflicts solve

* conflicts solve

* conflicts solve

* bugfix

* modify listResources mothod and remove getResourceIds mothod

* 1

* conflicts solve

* modify listResources mothod and remove getResourceIds mothod

* modify listResources mothod and remove getResourceIds mothod

* replace processDefinitionVersion with processDefinitionLog

* codestyle

* codestyle

* add mapper module ut

* codestyle

* fix ProcessInstanceMapperTest

* codestyle

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* fix ProcessInstanceMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix TaskInstanceMapperTest

* add TaskDefinitionLogMapperTest/TaskDefinitionMapperTest and bugfix

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add ProcessServiceTest

* 1

* 1
2021-03-24 15:14:22 +08:00
wen-hemin
e53367bdcc
[Feature][JsonSplit] task page, rename field "description" to "desc" (#5137)
* fix: task page, rename "description" to "desc"

* fix: The default value of field dependence is null.

* fix: frontend compile check

Co-authored-by: wen-hemin <wenhemin@apache.com>
2021-03-24 15:05:54 +08:00
Simon
21b7b77d50
[Feature][JsonSplit]fix modifyBy field (#5112)
* Modify Project and ProjectUser Mapper

* Modify Project and ProjectUser Mapper

* project_code is bigint(20)

* modify ERROR name

* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation

* modify import/export processdefinition, add genProcessData

* fix ut and bug

* code style

* repalce project_id with code

* conflicts solve

* conflicts solve

* conflicts solve

* bugfix

* modify listResources mothod and remove getResourceIds mothod

* 1

* conflicts solve

* modify listResources mothod and remove getResourceIds mothod

* modify listResources mothod and remove getResourceIds mothod

* replace processDefinitionVersion with processDefinitionLog

* codestyle

* codestyle

* add mapper module ut

* codestyle

* fix ProcessInstanceMapperTest

* codestyle

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* fix ProcessInstanceMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix TaskInstanceMapperTest

* add TaskDefinitionLogMapperTest/TaskDefinitionMapperTest and bugfix

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add ProcessServiceTest

* 1

* 1

* fix modifyBy field

* 1

* ut fix
2021-03-24 14:54:34 +08:00
JinyLeeChina
1eaf1f6234
[Feature][JsonSplit] fix taskId in processDefinitionJson (#5136)
* modify checkDAGRing and ProcessService method

* merge

* modify dagRing

* modify process instance for project home page

* fix save process bug

* codeStyle

* Fix logical bug in saving process definition

* codeSytle

* Fix bug in interface of  queryProcessDefinitionList

* codeSytle

* Fix api bug"

* fix taskId in processDefinitionJson

Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-03-24 11:44:14 +08:00
wenjun
5d264c9f20
[Improvement][Api] Optimize query processInstanceList #5134 (#5135)
* Batch query user rather than several query
2021-03-23 23:48:13 +08:00
Shiwen Cheng
0d065c0712
[Feature-5062][*] Create/Edit/Delete Worker Group in Web UI (#5113)
* [Improvement][Server] Rename MASTER_PREFIX to MASTER_TYPE and WORKER_PREFIX to WORKER_TYPE

* [Feature][Api&Dao] Support api and dao of worker group

* [Feature][UI] Support to create, edit and delete worker group

* [Feature][API&UI] Support work groups from database and zookeeper to display together in worker group management

* [Feature][API&Dao] Support long ip list and check request parameters when saving worker group

* [Feature][*] Refactor ipList to addrList

* [Feature][Server] Support host manager to select worker host by worker group in database

* [Improvement][Server] Improve the config of master and worker

* [Improvement][*] Rename zkRegistered to systemDefault

* [Improvement][Test] Fix unit test and improve ui

* [Improvement][Server] Improve getServerMaps

Co-authored-by: dailidong <dailidong66@gmail.com>
2021-03-23 23:47:41 +08:00
Kirs
f109a758f8
[Bug][PGSQL] Delete the stopped workflow without deleting the correspon… (#5066)
* [Bug][Api] Delete the stopped workflow without deleting the corresponding sub process, the workflow instance name is not displayed in the sub process

* fix ut

* fix ut err

* sql style

* fix error
2021-03-23 11:26:11 +08:00
JinyLeeChina
d99934d0da
[Feature][JsonSplit] Fix api bug (#5102)
* modify checkDAGRing and ProcessService method

* merge

* modify dagRing

* modify process instance for project home page

* fix save process bug

* codeStyle

* Fix logical bug in saving process definition

* codeSytle

* Fix bug in interface of  queryProcessDefinitionList

* codeSytle

* Fix api bug"

Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-03-23 10:44:25 +08:00
lenboo
010b49c933 merge from dev 2021-03-22 15:18:16 +08:00
zhuangchong
ba039dc252
[Improvement][dao] Resource features SQL slow query optimization. (#4947)
* Resourse features SQL slow query optimization.

* update resource mapper queryResourcePaging method.

* split resource and resource_uesr sql statement.

* determine the size of the collection.

* update process service class code style.

* update ResourceUserMapper java class code style.

* add resourceUserMapper in the test config file.

* solve code smell

* split paging query join statement.

* update ResourceUserMapper class.

* remove sql join statement.

* update resource test class code style.

* update ResourcesServiceTest test class.

* rerun ut.

* update user service test class.

* solve query resource list method code smell.
2021-03-20 12:50:26 +08:00
Shiwen Cheng
8ac72e80e6
[Feature-5087][SqlTask] Add a switch to send mail and print head logs in SqlTask (#5088)
* [Feature-5087][SqlTask] Add the switch to send mail and print head logs in SqlTask

* [Feature-5087][SqlTask] Add SqlParametersTest
2021-03-19 15:07:39 +08:00
guohaozhang
4a6e8b7afa
[Fix-#5103] fix resource rename cause can not update (#5107) 2021-03-19 14:51:12 +08:00
Shiwen Cheng
91c29e6ca3
[Improvement-5094][UI] Improve to add a main switch for refresh_in_switched_tab (#5095) 2021-03-19 11:58:33 +08:00
Shiwen Cheng
9008fa4b0c
[Fix-5089][UI] Fix code mirror cannot be displayed normally in sqoop task (#5090)
* [Improvement][UI] Rename from-mirror to form-mirror

* [Fix-5089][UI] Fix code mirror cannot be displayed normally in sqoop task
2021-03-19 11:57:46 +08:00
wenjun
086e716441
[Improvement][Dao] SQL optimization - check exist #5069 (#5070) 2021-03-18 22:49:12 +08:00
Shiwen Cheng
f94cfc620d
[Feature][Worker] Add a configuration item to set whether the tenant is automatically created on Worker (#5007) 2021-03-18 18:34:42 +08:00
wangxj3
29d42fd92d
[Feature-#3805][server-master] Task parameter transfer (#5077)
* fix out param format bug

Co-authored-by: wangxj <wangxj31>
2021-03-18 11:50:10 +08:00
Shiwen Cheng
5856a12855
[Improvement-4984][Worker] Refactor and Improve worker load balance (#4996)
* [Improvement][Remote] Improve unit tests

* [Improvement][Worker] Improve worker load balance

* [Improvement][Worker] Fix code smells

* [Improvement][Worker] Rename weight to hostWeight
2021-03-18 10:03:58 +08:00
zhuangchong
705715c313
[Improvement-5059][API] Verify that the new workflow definition version is the current version before removing it #5059 (#5060)
* this process definition version is used, not allowed to delete.

* update ProcessDefinitionVersionServiceTest test class.

* update check has associated process definition sql.
2021-03-18 09:37:17 +08:00
Ts686
6216817861
[Improvement-4971][alert|dao|service]Remove redundant PropertyUtils.class (#5020)
* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.

* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.

* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.

* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.
2021-03-17 15:28:07 +08:00
Shiwen Cheng
1cd62b4a5e
[Improvement][Datax] Re-assign DATAX_HOME to /opt/soft/datax instead of /opt/soft/datax/bin/datax.py (#5019) 2021-03-17 14:38:04 +08:00
mrproliu
0ba120e8ab
Fix shell task ${setValue(key,value)} is not working (#5067)
Co-authored-by: dalekliuhan <dalekliuhan@didiglobal.com>
2021-03-16 21:56:05 +08:00
lenboo
cf63fc1fee Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-16 15:50:06 +08:00
Simon
00dcc3f78e
[Feature][jsonsplit]add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest (#5047)
* Modify Project and ProjectUser Mapper

* Modify Project and ProjectUser Mapper

* project_code is bigint(20)

* modify ERROR name

* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation

* modify import/export processdefinition, add genProcessData

* fix ut and bug

* code style

* repalce project_id with code

* conflicts solve

* conflicts solve

* conflicts solve

* bugfix

* modify listResources mothod and remove getResourceIds mothod

* 1

* conflicts solve

* modify listResources mothod and remove getResourceIds mothod

* modify listResources mothod and remove getResourceIds mothod

* replace processDefinitionVersion with processDefinitionLog

* codestyle

* codestyle

* add mapper module ut

* codestyle

* fix ProcessInstanceMapperTest

* codestyle

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* fix ProcessInstanceMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix TaskInstanceMapperTest

* add TaskDefinitionLogMapperTest/TaskDefinitionMapperTest and bugfix

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest

* add WorkFlowLineageMapperTest/WorkFlowLineageServiceTest
2021-03-16 15:44:45 +08:00
JinyLeeChina
d98c2c5cfe
[Feature][JsonSplit] Fix bug in interface of queryProcessDefinitionList (#5043)
* modify checkDAGRing and ProcessService method

* merge

* modify dagRing

* modify process instance for project home page

* fix save process bug

* codeStyle

* Fix logical bug in saving process definition

* codeSytle

* Fix bug in interface of  queryProcessDefinitionList

* codeSytle

Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-03-16 15:44:03 +08:00
Kirs
6bf5031f45
[FIX] fix MySQL datasource jdbc connect parameters (#4851)
* [FIX-CVE-2020-11974] fix MySQLDataSource Security

* remove useless code

* add ut

* A more reasonable way

* code style

* fix test error

* code style

* update exception msg
2021-03-16 14:40:07 +08:00
lenboo
6ab3892989 Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-15 17:38:49 +08:00
游侠
2d8c4ec86b
[Improvement][Worker-Task] Python Command (#5036)
* create the interpreter for python command

* Update README.md

Co-authored-by: zhouhong <zhouhong@zhangyue.com>
2021-03-15 10:22:13 +08:00
lenboo
c33cc87af4 Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-14 21:47:30 +08:00
guohaozhang
a3153b5e44
[Bug][Master] Fix First master fault tolerance when startup (#5045)
* [Fix-#5044] Fix First master fault tolerance when startup

* [Fix-#5044] Fix spell error
2021-03-13 00:01:18 +08:00
Simon
22ec6c069d
[Feature][jsonsplit]Refactor process lineage (#5034)
* Modify Project and ProjectUser Mapper

* Modify Project and ProjectUser Mapper

* project_code is bigint(20)

* modify ERROR name

* modify saveProcessDefine, remove the duplicate code with createTaskAndRelation

* modify import/export processdefinition, add genProcessData

* fix ut and bug

* code style

* repalce project_id with code

* conflicts solve

* conflicts solve

* conflicts solve

* bugfix

* modify listResources mothod and remove getResourceIds mothod

* 1

* conflicts solve

* modify listResources mothod and remove getResourceIds mothod

* modify listResources mothod and remove getResourceIds mothod

* replace processDefinitionVersion with processDefinitionLog

* codestyle

* codestyle

* add mapper module ut

* codestyle

* fix ProcessInstanceMapperTest

* codestyle

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* conflicts solve

* fix ProcessInstanceMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix ProjectMapperTest/ProjectUserMapperTest/ScheduleMapperTest

* fix TaskInstanceMapperTest

* add TaskDefinitionLogMapperTest/TaskDefinitionMapperTest and bugfix

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage

* codestyle

* codestyle

* Refactor process lineage

* Refactor process lineage
2021-03-12 18:22:34 +08:00
Chenshuai
cc2a94a973
[Improvement-4988][docs] Uppercase the first letter the state of Process Instance (#4990)
* [DS-4988][docs] Uppercase the first letter the state of Process Instance

* [DS-4988][docs] Fix uppercase the first letter the state of Process Instance

* Fix uppercase the first letter the state of Process Instance

* Fix uppercase the first letter the state of Process Instance

* delete duplicate key name and delete

Co-authored-by: jim <ccit_cs@qq.com>
2021-03-12 16:48:33 +08:00
lenboo
a574d2e38c Merge remote-tracking branch 'upstream/dev' into spilit 2021-03-12 10:44:43 +08:00