test/DolphinScheduler
1
0
Fork 0
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-12-01 11:47:51 +08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
7,563 Commits 27 Branches 59 Tags 276 MiB
06b77b34c7
Commit Graph

1224 Commits

Author SHA1 Message Date
Wenjun Ruan
14ec4a2398
Remove dao module in worker (#13242) 2022-12-22 12:25:29 +08:00
Jay Chung
5cd8a02f87
[fix] Remove tenant arg in workflow definition (#13212) 
current tenant in workflow only work when the first time
user do not exist, when user change the tenant in workflow
but tenant exist, it will be ignore, so we try to remove it
from workflow, and in #40 we try to create both user and tenant
vis cli instead of auto create
 2022-12-19 16:06:41 +08:00
JieguangZhou
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks (#13194) 
* Supports task instance cache operation

* add task plugin cache

* use SHA-256 to generate key

* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>

* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>

* Optimizing database Scripts

* Optimize clear cache operation

Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
 2022-12-18 18:17:09 +08:00
youzipi
042ec74a24
[refactor] cache compiled regex pattern (#13208) 
* [refactor] cache compiled regex pattern

* [refactor] cache compiled regex pattern

* [refactor] cache compiled regex pattern
 2022-12-18 11:24:11 +08:00
insist777
0e3a924a37
move v2folder (#13192) 2022-12-16 10:30:01 +08:00
insist777
ca5af013ae
[Fix] openapi version inconsistency when updating workflows, tasks and relationship (#13094) 
Co-authored-by: xiangzihao <460888207@qq.com>
 2022-12-14 14:57:57 +08:00
insist777
631e3a81d1
[Feature][API] New restful API for taskInstance (#13070) 2022-12-14 14:47:36 +08:00
JieguangZhou
e4b9b67255
Allow execute task in workflow instance (#13103) 2022-12-13 16:43:44 +08:00
xuhhui
0db1bb6ba8
[Improvement][ApiServer]Task defination list paging sorted by updateTime not effect (#12622) 2022-12-13 16:35:54 +08:00
Wenjun Ruan
ab96a3d0c0
Fix datasource jdbc url param doesn't work (#13098) 2022-12-12 11:46:40 +08:00
Wenjun Ruan
f5801a0e1c
Fix task group might be rob by multiple task (#13115) 2022-12-11 22:58:21 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 (#13002) 2022-12-10 23:50:19 +08:00
J·Y
f9cf9331a6
Throws the correct exception for the updateResource interface (#13120) 
Co-authored-by: rqyin <rqyin@easipass.com>
 2022-12-09 16:17:20 +08:00
jackfanwan
cb8d125e0f
[Improvement] optimization task definition & fix in the task definition list, if one task have more pre task, the task list can't show all task (#13106) 
* fix in the task definition list, if one task have more pre task, the task list can't show all task

* modify code style

* in the task definition, delete search workflow name filter, and  fix in the task definition list, if one task have more pre task, the task list can't show all task

* modify code style

* modify code style

* delete useless select sql

* add annotation

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
 2022-12-07 09:52:53 +08:00
Wenjun Ruan
8a152aebc7
Fix task log file might not be clear (#13102) 2022-12-06 11:23:20 +08:00
Wenjun Ruan
fed1e85fb8
Fix the invalidate task instance never be deleted (#13091) 2022-12-04 18:38:54 +08:00
Wenjun Ruan
169168ef34
Add plugin-all module (#13079) 2022-12-02 23:19:08 +08:00
Jay Chung
1192720ddc
[ci] Update skywalking-eye and fix block CI (#13076) 2022-12-02 17:54:00 +08:00
Jay Chung
f5a59982b3
[fix] Python gateway can not upload to resource center (#13042) 
in #12076 we refactor our resource center, remove the resource table,
it is a good refactor but it failed python api upload, this patch try
to fix python api upload function

ref: #12076
 2022-12-02 10:11:57 +08:00
insist777
1064680ded
[feat][API] New restful API for workflow state (#13031) 2022-12-01 14:21:01 +08:00
Eric Gao
c2ae76c0bd
Expand all static imports and add CI check to block wildcard static imports (#13067) 2022-12-01 14:15:28 +08:00
Wenjun Ruan
ffc9fb280a
Add gc timestampt (#13059) 2022-12-01 10:00:27 +08:00
insist777
a0eb45b9e4
[Feature][API] New restful API for workflowInstance (#12990) 2022-11-28 12:58:46 +08:00
rickchengx
7336afaa65
[Fix-12916] Add permission check when query or download log (#12917) 2022-11-25 17:59:28 +08:00
rickchengx
d09e02e5a9
[Improvement-12804][UT] Remove the unused method in DataAnalysisControllerTest (#12805) 2022-11-21 20:59:30 +08:00
rickchengx
20518682bb
[Improvement-12853][UT] Improve the ut of ResourcePermissionCheckServiceTest (#12854) 2022-11-19 15:12:12 +08:00
insist777
528f45acc5
[improve]Source skip check (#12900) 
* Source skip check
 2022-11-17 14:23:12 +08:00
Assert
2dbc79693e
[feature][task plugin] Add new task plugin for apache linkis (#12693) 2022-11-17 10:14:36 +08:00
Jay Chung
f20c9b3102
[chore][python] Change name from process definition to workflow (#12918) 
only change its name in python gateway server code, incluing

* Function name: all related to process definition
* Parameter name and comment related

ref: apache/dolphinscheduler-sdk-python#22
 2022-11-16 21:23:39 +08:00
Zzih
0186413a9f
[Feature][Api] Refactor org.apache.dolphinscheduler.api.controller.TaskInstanceController (#11747) 
* Feature ：Refactor org.apache.dolphinscheduler.api.controller.TaskInstanceController
 2022-11-16 18:32:42 +08:00
Jay Chung
6d8befa075
[fix] Add token as authentication for python gateway (#12893) 
separate from #6407. Authentication, add secret to ensure only trusted people could connect to gateway.

fix: #8255
 2022-11-14 18:43:08 +08:00
Tq
d324cce6f1
[Feature-12526][Api] add dynamic task function: shell task (#12840) 
* submit dynamic task code

* submit shell.json

* add dynamic-task-type-config.yaml to standalone server
 2022-11-14 13:52:44 +08:00
rickchengx
813c44b22b
[Improvement-12838][UT] Improve the ut of datasource (#12839) 2022-11-14 11:30:10 +08:00
Kerwin
e3cf72cbcd
[Fix-12832][API] Fix update worker group exception group name already exists. (#12874) 2022-11-14 11:20:51 +08:00
Jay Chung
87a88e3662
[feat] Support set execute type to pydolphinscheduler (#12871) 
Up to now, we can only submit workflow with parallel mode. this patch give users ability specific execute type to workflow
 2022-11-12 10:31:08 +08:00
JieguangZhou
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2022-11-11 16:08:38 +08:00
Jay Chung
1347a8f94c
[chore] Separate Python API into another repository (#12779) 
Currently, our Python API code is a module in apache/dolphinscheduler codebase,
each time users change Python API code, they need to run all requests CI check
for dolphinscheduler and Python API, But if the user does only change Python
code, it could be merged if Python API CI pass and do not dependent on others CI.

Besides, we release Python API as the same version of dolphinscheduler. It is
easy for user to match Python API version. But when Python API does not change
any code, but dolphinscheduler release a bugfix version, Python API has to
release the new version to match dolphinscheduler. This happened when we
released Python API 2.0.6 and 2.0.7. 2.0.6 and 2.0.7 is bugfix version, and
Python API does not change any code, so the PyPI package is the same.

Separate Python API also makes our code more sense, we will have more
distinguished code in dolphinscheduler and Python API new repository.
Have separate issue tracker and changelog for information to users.

ref PR in other repository: apache/dolphinscheduler-sdk-python#1

see more detail in mail thread: https://lists.apache.org/thread/4z7l5l54c4d81smjlk1n8nq380p9f0oo
 2022-11-09 21:20:15 +08:00
jackfanwan
641a2559ea
[fix-12687]when timeout_notify_strategy is null, modify timeout_notify_strategy (#12689) 
* According to timeoutFlag, assign timeoutNotifyStrategy value

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
 2022-11-09 11:48:12 +08:00
houshitao
04c6b5e413
[Bug][Api] Added param description. (#12813) 
Co-authored-by: houshitao <shitaohou@163.com>
 2022-11-08 21:17:40 +08:00
jackfanwan
23ead4f0e8
[Bug] [dolphinscheduler-api] in the task define list, when edit task, should show pre node list (#12695) 
* backend : add pre node list

* resolve workflow error

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
 2022-11-08 13:31:20 +08:00
Eric Gao
6d79414491
Add instructions on how to configure logging level for debugging purpose (#12699) 2022-11-07 10:30:56 +08:00
JiPeng Wang
606a5aeee5
[Bug] Resource default auth function disabled false. (#12467) 
* resource function disable check fix

Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
 2022-11-06 21:15:27 +08:00
jackfanwan
f249f2b257
[fix-12675]edit workflow related task, workflow's task version change (#12692) 
* edit workflow related task, workflow's task version change

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
 2022-11-06 20:49:07 +08:00
Aaron Wang
04e23ed88f
Update Status.java (#12724) 2022-11-05 21:09:17 +08:00
jackfanwan
7d0e2cbbb9
when delete workflow, delete related task (#12681) 
* when delete workflow, delete related task

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
 2022-11-04 14:40:15 +08:00
JieguangZhou
a47b4d0672
[Feature][Task] Transfer files between tasks (#12552) 
* add data transfer between tasks

* add delete DATA_TRANSFER API

* convert Result to DeleteDataTransferResponse

* add api UT

* fix final

* fix doc
 2022-11-03 16:33:39 +08:00
rickchengx
c69376e74c
[Improvement-12658][Permission] Improve the readability of setPerm() in grantProjectbyCode() (#12660) 2022-11-03 09:29:04 +08:00
rickchengx
44e0935f56
[Improvement-12650][Permission] Improve the check of resourcePermissionCheck() (#12652) 2022-11-03 09:12:56 +08:00
rickchengx
3ff328c961
[Improvement-12623][Permission]Remove the unused projectIds in operationPermissionCheck() (#12624) 2022-11-02 11:20:00 +08:00
calvin
d84f1ef269
[Improvement][UI] Support to view the process variables on the page of DAG. (#12609) 2022-10-30 20:31:40 +08:00