Commit Graph

711 Commits

Author SHA1 Message Date
JiPeng Wang
961e67f524
[Improvement][User] Query all users filter un enabled. ()
* user query all filter un enabled
* restore permissions
2022-08-24 11:55:18 +08:00
longtb
4893bef5a7
[Improvement][TaskInstance] reduce database queries ()
* [Improvement][TaskInstance] reduce database queries

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

* [Improvement][TaskInstance] queryByInstanceIdsAndCodes -> queryByProcessInstanceIdsAndTaskCodes

Co-authored-by: zhangshunmin <zhangshunmin@kezaihui.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-08-23 14:27:26 +08:00
insist777
abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters ()
* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [improvement] The default worker grouping supports editing

* [improvement] The default worker grouping supports editing

* Update WorkerGroupServiceImpl.java

* Update WorkerGroupServiceImpl.java

* The default worker grouping supports editing

* update sql file

* update

* update

* update

* update
2022-08-23 10:54:17 +08:00
kezhenxu94
0b82755796
Fix some resource leak () 2022-08-21 01:19:16 +08:00
Wenjun Ruan
2862f5b67c
Set process instance and task priority default value as 2 () 2022-08-18 16:21:12 +08:00
Jiajie Zhong
5983e41f4b
[fix] Database change not sync ()
* [fix] Database change not sync

* Make del sql upgrade same as release version
* Correct data quality dml sql
* Upgrade database DDL to avoid log path too long error

* Add missing if exists
2022-08-17 10:50:34 +08:00
juzimao
f3d28112ce
optimize sorting priority for workflow define () 2022-08-13 13:28:35 +08:00
insist777
8217784044
[Fix-1360]: Fix Description length check ()
* [Fix-1360]: Fix Description length check

* Update dolphinscheduler_h2.sql

h2 Increase the length of some fields

* Update dolphinscheduler_ddl.sql
2022-08-11 13:26:50 +08:00
caishunfeng
0464123c2b
[Feature-11223] support stream task ()
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus () 2022-08-10 11:00:23 +08:00
Wenjun Ruan
acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop ()
* Fix TaskGroup cannot work and will cause master dead loop

* Remove acquireTaskGroupAgain in ProcessServiceImpl
2022-08-04 14:43:45 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db () 2022-08-01 13:04:27 +08:00
zhuxt2015
4d9db3451e
[Bug] [API] If the single log length is long, it will causes the View Log page to hang () 2022-07-26 10:14:45 +08:00
jackfanwan
ccb7641468
修改数据质量查询的权限判断 ()
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-07-25 09:35:53 +08:00
WangJPLeo
fd59f0bb32
[Fix-11031] AccessToken can only be used by the creator. ()
* AccessToken can only be used by the creator.

* supplement ut

* checkstyle fix
2022-07-20 16:32:03 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version () 2022-07-20 10:37:31 +08:00
Jiajie Zhong
81930e5420
[python] Fix tasks with multiple upstream and workflow query error ()
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed
2022-07-14 22:06:07 +08:00
Assert
4ff8eaa996
[Fix-10595][api] Fix Dependent Node Only View Projects Created By Self ()
* [Fix-10595][api] Fix Dependent Node Only View Projects Created By Self
2022-07-14 19:03:34 +08:00
SongTao Zhuang
99a134febb
fix: typo task definition. () 2022-07-14 11:40:46 +08:00
WangJPLeo
9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. ()
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
Jiajie Zhong
0db9bbd538
[fix] delete workflow or task used by other tasks ()
currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
2022-07-12 21:22:06 +08:00
Jiajie Zhong
a6154220dc
[fix] update queue and tenant with same code error ()
in ae6aa53f I use the same logic test to create or update
tenant and queue which failed create with the same value,
this patch will fix it. and also I use the constructor to reduce
the new object logic here
2022-07-11 11:06:30 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ()
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
WangJPLeo
3756a233d8
[Fix-10863] Command execution result data statistics remove userId condition. ()
* Command execution result data statistics remove userId condition.

* e2e rerun
2022-07-10 17:17:35 +08:00
Jiajie Zhong
ae6aa53f96
[python] Fix submit and run error ()
because we add permission, so it failed out
createQueue method. this patch fix it and
do some refactor of our tenant and queue
validator code
2022-07-08 10:54:41 +08:00
qianli2022
1e48bbc5c6
[Feature-10629][Improvement] Support multi cluster environments - namespace use cluster ()
* server code

* namespace ui

* fix dao test

* fix test

* fix

* docs and image update

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-07-07 16:05:52 +08:00
WangJPLeo
b8a90d801e
[Improvement] Resource authority module database table unique index increased. ()
* Resource authority module database table unique index increased.

* add unique in sql file

* rerun

* ut fix

* e2e rerun

* dao un fix
2022-07-07 14:35:50 +08:00
WangJPLeo
d2fe16d252
[Optimization] Config module resource relation fix. ()
* Config module resource relation fix.

* Boolean judge fix

* e2e rerun

* e2e rerun

* e2e rerun

* ArrayList fix

* change variable name.

* Add transaction processing on create data resource method
2022-07-05 15:51:23 +08:00
Wenjun Ruan
1b86394d3c
Optimize sql script use insert ignore () 2022-06-30 19:04:57 +08:00
WangJPLeo
bc4ceda4cf
TaskGroup remove query func permission check. () 2022-06-23 10:00:31 +08:00
WangJPLeo
20348578a2
[Feature] [Permission]Data permission module added and Query class api remove verification. ()
* Data permission module added

* ut fix.

* queueController ut fix.

* queue service ut fix.

* e2e rerun
2022-06-22 15:53:36 +08:00
chuxing
0dd6f4008e
[fix][master]: fix condition npe ()
* [fix][master]: fix condition npe

* [fix][master]: fix condition npe

Co-authored-by: huaangcg <huangcg@getui.com>
2022-06-17 13:40:11 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ()
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
SmallProgrammer
7aa9fa4965
[imporve] Add index to table t_ds_relation_process_instance () 2022-06-16 15:24:05 +08:00
qianli2022
ff065d8e5b
[Feature][Improvement] Support multi cluster environments - k8s type ()
* service code

* [Feature][UI] Add front-end for cluster manage

* fix e2e

* remove comment on cluster controller

* doc

* img

* setting e2e.yaml

* test

* rerun e2e

* fix bug from comment

* Update index.tsx

use Nspace instead of css.

* Update index.tsx

Remove the style.

* Delete index.module.scss

Remove the useless file.

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>
2022-06-15 13:39:20 +08:00
陈家名
b86dc53ad1
[fix][Python] Support same task name in project defferent process definition ()
close: 
2022-06-14 13:53:18 +08:00
WangJPLeo
9782fe4e7c
[Fix-10367] Permission check related key adjustment. ()
* Permission check related key adjustment.

* resource create or upload fix.

* Analysis Permission fix.

* Ut coverage add.

* rerun

* source permission fix.

* ut coverage is achieved

* Reference permission package path modification

* k8s path bug fix.

* k8s path bug fix.

* Corrected controller method name and rerun.
2022-06-09 19:10:34 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup () 2022-06-08 19:42:05 +08:00
WangJPLeo
e3e39cbdea
[Fix] Privilege Reconstruction Data Query Fix & Resource Creation Post Oper… ()
* Privilege Reconstruction Data Query Fix & Resource Creation Post Operations.

* Resource Retry

* e2e retry

* create project log add

* project list query log add

* clean project log

* delete delay

* delete delay

* remove post handle

* project e2e

* browser refresh

* browser refresh

* e2e fix

* e2e browser refresh

* rowk flow e2e fix

* mapper deduplication

* udf e2e

* e2e
2022-06-01 20:29:10 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues () 2022-05-31 15:22:41 +08:00
Wenjun Ruan
3a9e67325e
Fix database initialize late in standalone. () 2022-05-31 11:52:00 +08:00
WangJPLeo
a4948f58e6
[Feature][Permission] Reconstruction of permissions of resource center and monitoring center. ()
* Reconstruction of permissions of resource center and monitoring center.

* clear local logs.

* resource query fix
2022-05-31 08:14:45 +08:00
hstdream
53ab6f7b92
[Feature][API] Modify the permissions of project management, security center, data source center and data quality module. ()
* Resource userName is null

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.

Co-authored-by: houshitao <shitaohou@163.com>
2022-05-30 23:32:25 +08:00
WangJPLeo
7ac8ea493c
[Feature][Permission] Project management module authority control refactoring, authority abstract interface setting. ()
* Create process definition with task group and task group priority cannot be save into db

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* The result of the assertion cannot be changed. The only difference is the judgment condition.

* test recovery.

* controller test fix.

* ProjectE2ETest fix.

* WebElement import.

* Prioritize problem solving.

* Judging the length of the set fix

Co-authored-by: houshitao <shitaohou@163.com>
Co-authored-by: hstdream <33045461+hstdream@users.noreply.github.com>
2022-05-29 16:14:16 +08:00
Jiajie Zhong
d8db2b57d2
[fix] Task group queue query all list error ()
Add support query all task queue by not given specific groupId

close: 
2022-05-25 16:33:23 +08:00
longtb
bd34e66b8a
[Fix-10076] queryUserByToken use server timezone ()
* [Fix-10076] queryUserByToken use server timezone

* [Fix-10076] queryUserByToken use server timezone
2022-05-23 09:54:11 +08:00
BaoLiang
ee2b855ced
[BUG][TaskGroup] Task group does not take effect ()
* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect

* fix 10092: Task group does not take effect
2022-05-18 18:40:36 +08:00
hstdream
701b535ae6
[Fix-10082]Resource userName is null and udf function manager add a username ()
* Resource userName is null

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

* [Fix-10082]Resource userName is null and udf function manager add a username

Co-authored-by: houshitao <shitaohou@163.com>
2022-05-18 11:33:08 +08:00
hstdream
cd70ad584a
Create process definition with task group and task group priority cannot be save into db ()
Co-authored-by: houshitao <shitaohou@163.com>
2022-05-18 09:41:03 +08:00
qianli2022
bc1c15b142
[Feature-8252][doc] K8s and namespace manager docs and web page update ()
* test

* test

* doc

* fix image error

* fix

* Update docs/docs/en/guide/security.md

* Update docs/docs/en/guide/security.md

* Update docs/docs/en/guide/security.md

* Update docs/docs/zh/guide/security.md

* fix doc

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-11 16:40:30 +08:00