Commit Graph

6910 Commits

Author SHA1 Message Date
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979)
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed

* Move heartbeat error threshold to applicaiton.yml
2022-07-15 20:06:53 +08:00
SongTao Zhuang
dad8981cfe
fix: typo executor service (#10980) 2022-07-15 18:58:39 +08:00
Wenjun Ruan
998e4d74dd
Kill task when we do master failover to avoid task lost (#10997) 2022-07-15 18:44:42 +08:00
labbomb
d41dac0602
feat: Add trim method to all input (#10984) 2022-07-15 17:14:47 +08:00
JieguangZhou
11a354927d
[fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms (#10986) (#10987)
* Optimize the download method of preset project

* fix checkstyle
2022-07-15 15:38:13 +08:00
songjianet
fd5fb85858
[Refactor][UI] Refactor audio log using NSpace component. (#10991) 2022-07-15 15:10:48 +08:00
zhuxt2015
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters (#10950)
* datax task plugin support custom parameters
2022-07-15 09:27:23 +08:00
Eric Gao
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#10925)
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814)
2022-07-15 09:26:28 +08:00
Jiajie Zhong
81930e5420
[python] Fix tasks with multiple upstream and workflow query error (#10941)
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed
2022-07-14 22:06:07 +08:00
labbomb
05308ccfe6
feat: Added global trim method (#10975) 2022-07-14 20:44:41 +08:00
Assert
4ff8eaa996
[Fix-10595][api] Fix Dependent Node Only View Projects Created By Self (#10834)
* [Fix-10595][api] Fix Dependent Node Only View Projects Created By Self
2022-07-14 19:03:34 +08:00
songjianet
89780e296d
[Feat][UI] Add title display to the table in the security center. (#10969) 2022-07-14 18:02:29 +08:00
rockfang
895dca3e93
[Fix-10890] fix some bugs in the Resource page (#10907)
* [Fix-10890] fix some bugs in the Resource page

* fix: cancel the unnecessary verifications and reset the form when canceling FileUploadModal

* fix: use se `resetForm` method to reset the form
2022-07-14 16:17:06 +08:00
SongTao Zhuang
99a134febb
fix: typo task definition. (#10958) 2022-07-14 11:40:46 +08:00
Mr.An
ed7ec8ac10
[fix-10954]Querying the task group list returns empty data (#10955)
* fix task group query page retrun is null
2022-07-14 11:16:28 +08:00
Amy0104
c2dfda80f6
[Fix][UI] Set the next node to not required. (#10932)
* [Fix][UI] Set the next node to not required.

* [Fix][UI] Remove the unuse i
2022-07-14 10:15:25 +08:00
Wenjun Ruan
b245e7c973
Fix kill yarn job error when failover caused by doesn't set ProcessDefinition (#10948) 2022-07-13 22:32:30 +08:00
Wenjun Ruan
df0416c193
Fix taskInstance's host is not worker nettyServer address (#10926)
* Fix taskInstance's host is not worker nettyServer address

* Remove unnecessary mock
2022-07-13 20:46:33 +08:00
WangJPLeo
9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. (#10922)
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
Jiajie Zhong
2d7a48b0f0
[fix] Can not query exists environments (#10884)
always return empty set
2022-07-13 17:21:29 +08:00
pinkhello
7a15877fa4
[FIX-10784][Bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop (#10888)
* closed 10784 [Bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop
2022-07-13 14:27:44 +08:00
chuxing
427c58546c
[fix][plugin]: fix spark sql use custom parma. (#10821)
* [fix][plugin]: fix spark sql use custom parma.

* [fix][plugin] roll back part of the code

Co-authored-by: huaangcg <huangcg@getui.com>
2022-07-13 11:44:13 +08:00
WangJPLeo
2397423eb6
Solve transaction problems under multi-threading (#10917) 2022-07-13 11:23:47 +08:00
Wenjun Ruan
2a67866718
Catch exception when check state in StateWheelExecuteThread (#10908)
* Catch exception when check state
2022-07-13 10:51:20 +08:00
Amy0104
51d6eae7f4
[Fix][UI] Add validator to the next node field of the switch task. (#10898) 2022-07-13 10:21:18 +08:00
Devosend
0ea5809ff8
fix bug where task name display in gantt (#10900) 2022-07-13 09:55:10 +08:00
zhuxt2015
c9afc1a1f4
display timing record start time (#10915) 2022-07-13 09:53:48 +08:00
Jiajie Zhong
0db9bbd538
[fix] delete workflow or task used by other tasks (#10873)
currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
2022-07-12 21:22:06 +08:00
JieguangZhou
02b2de2a1e
[fix] mlflow project success status (#10897)
close: #10902
2022-07-12 20:58:53 +08:00
陈家名
59cd86157f
[Python] Supports creating or editing resources. (#10823) 2022-07-12 20:44:59 +08:00
Tq
499e5b1307
[bug][task-plugins]fix switch class name and add param check to the switch task plugin (#10894)
* fix switch class name and add param check to the switch task plugin
2022-07-12 16:41:15 +08:00
Jiajie Zhong
08cfbc73e0
[doc] Add missing doc change by snehal (#10892) 2022-07-12 14:16:00 +08:00
Wenjun Ruan
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master (#10886)
* Fix network error cause worker cannot send message to master
2022-07-12 14:08:42 +08:00
Eric Gao
2f7281c2d2
[Feature][Metrics] Add resource download related metrics for workers (#10749)
* [Feature][Metrics] Add resource download related metrics for workers (#9324)

* [Feature][Metrics] Fix bugs and add grafana demos for worker resource download metrics (#9324)

* [Feature][Metrics] Add docs to resource related metrics (#9324)

* [Feature][Metrics] Use tags to indicate status in metrics (#9324)

* [Feature][Metrics] Fix demos, docs and remove redundant code (#9324)

* [Feature][Metrics] Remove .pnpm-debug.log (#9324)

* [Feature][Metrics] Fix style check (#9324)

* [Feature][Metrics] Replace KB with bytes for the unit of resource file size in metrics (#9324)

* [Feature][Metrics] Make code neat (#9324)
2022-07-12 11:44:34 +08:00
Devosend
56fe11ecf9
[Fix][UI] Fix hive datasource doesn't show principal bug (#10876)
* fix hive datasource doesn't show principal bug

* delete IDataBase

* restore DataSourceController.java
2022-07-12 09:18:10 +08:00
songjianet
1edbb5e333
[Feat][UI] Uniform search button size. (#10880) 2022-07-12 06:51:34 +08:00
caishunfeng
aa8b88a875
[Feature-10871] add workflow executing data query (#10875)
* add workflow executing data query
* fix sonar check for interrupted
2022-07-11 16:55:55 +08:00
songjianet
553159f780
[Refactor][UI] Refactor tenant manage using NSpace component. (#10879) 2022-07-11 15:46:04 +08:00
Jiajie Zhong
a6154220dc
[fix] update queue and tenant with same code error (#10877)
in ae6aa53f I use the same logic test to create or update
tenant and queue which failed create with the same value,
this patch will fix it. and also I use the constructor to reduce
the new object logic here
2022-07-11 11:06:30 +08:00
Wenjun Ruan
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866)
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
WangJPLeo
3756a233d8
[Fix-10863] Command execution result data statistics remove userId condition. (#10864)
* Command execution result data statistics remove userId condition.

* e2e rerun
2022-07-10 17:17:35 +08:00
Jiajie Zhong
598290487c
[python] Integrate test run strategy change (#10825)
this patch change python integrate test strategy,
before that, only content change in dir
dolphinscheduler-python will run the test. And it
will fail our python api startter when some code
change in java gateway. So we should start the
integrate test except docs change.

and in the future, we may add python ci to
.asf.yaml to make sure it pass

close: #10488
2022-07-10 11:06:27 +08:00
ZhaoGuodong
8eaf5a2309
[Feature-10219][EMR] EMR supports use <add-Steps> to add steps to an existing cluster (#10657)
* Add the ProgramType parameter to distinguish task types
* EmrAddStepsTask supports Add-Steps
* UI supports Add-Steps
* EmrTask modify the name of the class to EmrJobFlowTask
* add ERM Task abstract base class AbstractEmrTask
* add testcase for EmrAddStepsTask
* Modifying help Documents
2022-07-10 10:15:25 +08:00
houshitao
cc41485d50
[Bug][Api] Complete Data workflow remove duplicates. (#10868)
Co-authored-by: houshitao <shitaohou@163.com>
2022-07-10 01:31:44 +08:00
pinkhello
030fb89d6e
[Fixed-10833] [Bug] [Quartz] timezone display doesn't match the next_fire_time in ds 3.0.0-beta1 version (#10865)
* closed [10619]  [Improvement][Master] batch remove TaskInstaceId and workflowInstanceId

* fixed # 10833 timezone display error

* checkstyle
2022-07-09 22:28:08 +08:00
houshitao
73f4846c49
[Bug][Api] Complete Data workflow remove duplicates. (#10850)
* [Bug][Api] Complete Data workflow remove duplicates.

Co-authored-by: houshitao <shitaohou@163.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-07-09 21:29:11 +08:00
WangJPLeo
b5653ea7f2
Complement numbers will run in a loop under the serial strategy fixed. (#10862)
* Complement numbers will run in a loop under the serial strategy fixed.

* e2e rerun
2022-07-09 16:41:51 +08:00
Luke Yan
057a9f6b24
[Fix-10830][Doc] Change the start or install script command sh in the document to bash (#10831)
* Change sh to bash

Change the script execution command sh to bash

* Change sh to bash

Change the script execution command sh to bash

* Change sh to bash

Change the script execution command sh to bash

* Change sh to bash

Change the script execution command sh to bash

* Change sh to bash

Co-authored-by: luke <luke.yan@china-entercom.net>
2022-07-09 15:41:56 +08:00
Chris Ho
0cf31232de
[Feature][dependencies] Bump spring-core from 5.3.12 to 5.3.19 (#10857) 2022-07-09 15:19:46 +08:00
Eric Gao
1387fb3ccf
[Doc][Upgrade] Fix tiny error in upgrade docs (#10847)
* [Doc][Upgrade] Fix tiny error in upgrade docs (#10846)

* [Doc][Upgrade] Add useSSL=false and fix typo in upgrade docs (#10846)
2022-07-09 14:53:05 +08:00