Commit Graph

734 Commits

Author SHA1 Message Date
Jiajie Zhong
82ddd72e4a
[Feature][API] New restful API for workflow and schedule (#11912)
* [feat] New restful API for workflow and schedule

CURD for workflow and schedule, different with exists
API, this new restful api only operate single resource
in each request, and return the latest. For example,
previous workflow should also need to post tasks definition
and tasks relation definition, but this patch will allow
you to create workflow without task relate information

* use checkProjectAndAuthThrowException, and fix CI error

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ScheduleV2Controller.java

* change method name
from createProcessDefinitionV2 to createSingleProcessDefinition
from updateProcessDefinitionV2 to updateSingleProcessDefinition

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-23 22:25:19 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
limaiwang
ed1d1e8856
add(dependent node's add unprivileged project list): dependent node's add unprivileged project list (#12073) 2022-09-21 09:28:22 +08:00
juzimao
c8884e4f2f
and task instance date range search (#11963) 2022-09-19 17:50:24 +08:00
insist777
5b384f3fab
[Feature-11473][Task]Support test task (#11670)
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
Wenjun Ruan
864a90820d
Fix insertOrUpdate plugin may failed due to concurrent operation (#11471) 2022-09-16 15:39:04 +08:00
Wenjun Ruan
d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
Wenjun Ruan
4ad34483a5
Fix alert_send_status may throw duplicate key exception and add limit for query alert (#11953) 2022-09-16 13:26:33 +08:00
caishunfeng
925e2fa551
[Bug][Workflow Definition] fix ordinary users can not create depend task (#11961) 2022-09-16 09:12:33 +08:00
wendongdi
e70c501f9a
[Fix][db] fix init&upgrade mysql-meta-schema bugs (#11887) 2022-09-14 18:03:07 +08:00
juzimao
b1068e27ba
Optimize date range condition (#11911) 2022-09-14 16:55:10 +08:00
Wenjun Ruan
8a47785f42
Fix listing processDefinition the schedulerReleaseState will never be null (#11888) 2022-09-13 10:07:12 +08:00
Yann Ann
e20f17a7b7
[Improvement-#11768][Monitor] Support monitor h2 database in monitor page (#11813) 2022-09-09 09:43:56 +08:00
Stalary
37325b4c34
[Bug](dependent) Dependent downstream trigger error when schedule cycle not day. (#11734)
* FIX: dependent

* FIX: version

* MOD: for review
2022-09-08 15:08:10 +08:00
caishunfeng
2e61c76c22
[Improvement] Add remote task model (#11767) 2022-09-08 13:41:36 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
fuchanghai
7c0c1f4b78
Update dolphinscheduler_ddl.sql fix alert t_ds_user add time_zone is not correct (#11795) 2022-09-06 23:14:41 +08:00
Tq
4283cfd7a1
[fix][API]fix task types sequence (#11775)
* fix task types sequence

* fix mapper table name t_ds_fav_task

* fix sequence
2022-09-05 20:29:03 +08:00
caishunfeng
f8d46a26c1
[Feature-11530] add state history for process instance (#11757)
* [Feature] add state history for process instance (#97)

* add state history for process instance

* upsertProcessInstance

* remove unuse method

* fix UT

Co-authored-by: caishunfeng <534328519@qq.com>
2022-09-03 10:14:05 +08:00
Tq
b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav (#11727)
* feature: add task type list and enable user to add task type to fav

* make DependentExecute.java variable protected

* add standalone server fileSet
2022-09-02 11:40:23 +08:00
juzimao
d0f5e7edf3
optimize the process instance query, change the date time range (#11719) 2022-08-31 19:53:41 +08:00
seedscoder
145e441afb
[Fix-11633] The data type of the PostgreSQL database column is wrong. (#11633) (#11634) 2022-08-29 16:33:16 +08:00
SongTao Zhuang
6a5367fd57
[Improvement-11662][dao] fix foreach null items in mapper.xml 2022-08-28 20:58:12 +08:00
JiPeng Wang
961e67f524
[Improvement][User] Query all users filter un enabled. (#11504)
* user query all filter un enabled
* restore permissions
2022-08-24 11:55:18 +08:00
longtb
4893bef5a7
[Improvement][TaskInstance] reduce database queries (#11522)
* [Improvement][TaskInstance] reduce database queries

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

* [Improvement][TaskInstance] queryByInstanceIdsAndCodes -> queryByProcessInstanceIdsAndTaskCodes

Co-authored-by: zhangshunmin <zhangshunmin@kezaihui.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-08-23 14:27:26 +08:00
insist777
abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters (#11542)
* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [improvement] The default worker grouping supports editing

* [improvement] The default worker grouping supports editing

* Update WorkerGroupServiceImpl.java

* Update WorkerGroupServiceImpl.java

* The default worker grouping supports editing

* update sql file

* update

* update

* update

* update
2022-08-23 10:54:17 +08:00
kezhenxu94
0b82755796
Fix some resource leak (#11576) 2022-08-21 01:19:16 +08:00
Wenjun Ruan
2862f5b67c
Set process instance and task priority default value as 2 (#11539) 2022-08-18 16:21:12 +08:00
Jiajie Zhong
5983e41f4b
[fix] Database change not sync (#11258)
* [fix] Database change not sync

* Make del sql upgrade same as release version
* Correct data quality dml sql
* Upgrade database DDL to avoid log path too long error

* Add missing if exists
2022-08-17 10:50:34 +08:00
juzimao
f3d28112ce
optimize sorting priority for workflow define (#11462) 2022-08-13 13:28:35 +08:00
insist777
8217784044
[Fix-1360]: Fix Description length check (#11371)
* [Fix-1360]: Fix Description length check

* Update dolphinscheduler_h2.sql

h2 Increase the length of some fields

* Update dolphinscheduler_ddl.sql
2022-08-11 13:26:50 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task (#11350)
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
Wenjun Ruan
acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop (#11254)
* Fix TaskGroup cannot work and will cause master dead loop

* Remove acquireTaskGroupAgain in ProcessServiceImpl
2022-08-04 14:43:45 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db (#11221) 2022-08-01 13:04:27 +08:00
zhuxt2015
4d9db3451e
[Bug] [API] If the single log length is long, it will causes the View Log page to hang (#10612) 2022-07-26 10:14:45 +08:00
jackfanwan
ccb7641468
修改数据质量查询的权限判断 (#10977)
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-07-25 09:35:53 +08:00
WangJPLeo
fd59f0bb32
[Fix-11031] AccessToken can only be used by the creator. (#11032)
* AccessToken can only be used by the creator.

* supplement ut

* checkstyle fix
2022-07-20 16:32:03 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
Jiajie Zhong
81930e5420
[python] Fix tasks with multiple upstream and workflow query error (#10941)
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed
2022-07-14 22:06:07 +08:00
Assert
4ff8eaa996
[Fix-10595][api] Fix Dependent Node Only View Projects Created By Self (#10834)
* [Fix-10595][api] Fix Dependent Node Only View Projects Created By Self
2022-07-14 19:03:34 +08:00
SongTao Zhuang
99a134febb
fix: typo task definition. (#10958) 2022-07-14 11:40:46 +08:00
WangJPLeo
9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. (#10922)
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
Jiajie Zhong
0db9bbd538
[fix] delete workflow or task used by other tasks (#10873)
currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
2022-07-12 21:22:06 +08:00
Jiajie Zhong
a6154220dc
[fix] update queue and tenant with same code error (#10877)
in ae6aa53f I use the same logic test to create or update
tenant and queue which failed create with the same value,
this patch will fix it. and also I use the constructor to reduce
the new object logic here
2022-07-11 11:06:30 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866)
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
WangJPLeo
3756a233d8
[Fix-10863] Command execution result data statistics remove userId condition. (#10864)
* Command execution result data statistics remove userId condition.

* e2e rerun
2022-07-10 17:17:35 +08:00
Jiajie Zhong
ae6aa53f96
[python] Fix submit and run error (#10792)
because we add permission, so it failed out
createQueue method. this patch fix it and
do some refactor of our tenant and queue
validator code
2022-07-08 10:54:41 +08:00
qianli2022
1e48bbc5c6
[Feature-10629][Improvement] Support multi cluster environments - namespace use cluster (#10650)
* server code

* namespace ui

* fix dao test

* fix test

* fix

* docs and image update

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-07-07 16:05:52 +08:00
WangJPLeo
b8a90d801e
[Improvement] Resource authority module database table unique index increased. (#10807)
* Resource authority module database table unique index increased.

* add unique in sql file

* rerun

* ut fix

* e2e rerun

* dao un fix
2022-07-07 14:35:50 +08:00