Commit Graph

157 Commits

Author SHA1 Message Date
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477)
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
kezhenxu94
fa19006a18
Add a property to enable/disable metrics module (#8484) 2022-02-24 10:21:47 +08:00
Kerwin
36d526d3a8
[Improvement-8284][Alert] Dingtalk alert plugin supports markdown message type (#8285)
* add msgtype in the dingtalk alert plugin

* update markdown msgtype 'at persion'
2022-02-07 22:07:37 +08:00
pinkhello
84b004d960
[Feature-8245][Alert] Add Alert Plugin Telegram (#8246) 2022-02-07 09:51:18 +08:00
wangyang
49d581d72e
[DS-8215][feat] Support WebexTeams Plugin (#8217)
Co-authored-by: qingwli <qingwli@cisco.com>
2022-01-29 11:36:38 +08:00
wind
128ede3ee9
Fix start standalone fail because db not init yet (#8191) 2022-01-25 18:10:02 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
wangyang
14908a25e8
[Feature-7992][Alert] Support PagerDuty Plugin (#8120)
Co-authored-by: qingwli <qingwli@cisco.com>
2022-01-25 10:04:21 +08:00
Kerwin
1fd748f376
[Feature-8138][Task] add at@ msg in the dingtalk task plugin (#8139) 2022-01-21 17:09:39 +08:00
Kerwin
cb8c875fa8
[Feature-7804][Task] DingTalk alert plugin adds signature. (#7805)
* DingTalk alarm component adds signature.

* fix code style.

* fix code style.

* add dingtalk document link

* fix DingTalkAlertChannelFactoryTest

* update NAME_DING_TALK_SECRET
2022-01-12 15:24:25 +08:00
springmonster
4f4d9d12ce
[Bug-7849] Add unit test for alert server, fix #7849 (#7850) 2022-01-11 12:46:21 +08:00
kezhenxu94
68906f1b31
Remove fast return for alert plugins (#7723) 2021-12-30 15:02:03 +08:00
springmonster
2b05ebf47e
[Refactor] Canonical enum naming (#7676)
* [Refactor] Refactor comments
2021-12-28 19:07:53 +08:00
Jiezhi.G
6562bc9145
fix http alarm msg ignored (#7523)
See https://github.com/apache/dolphinscheduler/issues/7522
2021-12-21 16:04:12 +08:00
SbloodyS
86b476a160
[Feature-7460][Alert] Wechat alert add send to group chat (#7465)
* feature_7460

* feature_7460

Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-18 10:26:16 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
Zonglei Dong
fa8ccd3855
Refactor code, add class import, keep ‘logger’ variable names uniform. (#7164) 2021-12-04 21:18:43 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time (#7054) 2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config (#6947)
* [DS-5229][fix] server port custom config
This closes #5229

* [DS-5229][feat] Implement server port custom config
This closes #5229

* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862)

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md

* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852)

Co-authored-by: caishunfeng <534328519@qq.com>

* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229

Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-01 00:21:03 +08:00
kezhenxu94
2de4a3cbb5
Add a demo docker-compose for testing the meter module (#6938) 2021-11-24 17:22:30 +08:00
LinKai
ad33d43498
[Improvement][Dao] SQL optimization - limit one piece of data #5061 (#6940)
* [Feature][unittest] Recover UT in AlertPluginManagerTest.java #6619

* use throws exception instead of try-catch to catch exceptions in AlertPluginManagerTest.java

* change the assert result from assertNull to assertNotNull

* [Improvement][Dao] SQL optimization - limit one piece of data #5069

* Improve the code to A rate

Co-authored-by: linkai <linkai@bigo.sg>
2021-11-22 20:27:29 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
mask
e36d18b588
[Fix-6792] resolve API and alert port conflicts (#6792) (#6815)
* [Fix-6792]Multiple ApplicationRunners on classpath make SpringApplication.run only run one (#6792)
2021-11-12 10:51:11 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage (#6767) 2021-11-10 18:29:17 +08:00
kezhenxu94
05bae0b226
Refactor alert plugins and simplify its usage (#6723) 2021-11-09 18:59:34 +08:00
kezhenxu94
02350ea37b
Fix datasource not work in tests and standalone server, and run all tests by default (#6743) 2021-11-09 16:57:26 +08:00
kezhenxu94
63602bf77c
Clean up the utility codes (#6732) 2021-11-08 23:22:07 +08:00
Wenjun Ruan
0afb780718
[Improvement-5807][Dao] Use h2 in UT (#6500)
* Use H2 in UT

* change scope

* Change profile name to h2

* Add datasource-h2.properties, datasource-postgresql.properties

* Fix UT

* Resolve conflict

* Remove startDatabase
2021-11-07 23:19:19 +08:00
jon-qj
86c85114b2
make alert instance page display more friendly (#6635)
* make alert instance page display more friendly

* Update EmailAlertChannelTest.java
2021-10-31 21:16:10 +08:00
Jiajie Zhong
cbeedba3e2
Remove unused try block in UT (#6182)
closes https://github.com/apache/dolphinscheduler/issues/6181
2021-10-27 19:01:33 +08:00
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT (#6383) 2021-09-28 12:00:43 +08:00
kezhenxu94
a632be73c0
Clean up unused util classes... (#6154) 2021-09-10 00:35:39 +08:00
CalvinKirs
dbb4adf6c3 Merge remote-tracking branch 'upstream/dev' into task-end
# Conflicts:
#	.asf.yaml
#	dolphinscheduler-alert/src/main/java/org/apache/dolphinscheduler/alert/utils/FuncUtils.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/SchedulerController.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
#	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java
#	dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/AlertGroupControllerTest.java
#	dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorService2Test.java
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/StateEventType.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Command.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ErrorCommand.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java
#	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskDefinition.java
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ScheduleMapper.xml
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionMapper.xml
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/ConditionsTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/DependentTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/DependentTaskProcessor.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/WorkerServer.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/ConditionsTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/DependentTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/MasterBaseTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/MasterTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/SubProcessTaskExecThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/WorkerManagerThread.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/python/PythonTask.java
#	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/WorkflowExecuteThreadTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/processor/TaskKillProcessorTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThreadTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/runner/WorkerManagerThreadTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/PythonCommandExecutorTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java
#	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTaskTest.java
#	dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
#	dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/cron/CronUtils.java
#	dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java
#	dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/DolphinSchedulerPlugin.java
#	dolphinscheduler-standalone-server/pom.xml
#	dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-procedure/src/main/java/org/apache/dolphinscheduler/plugin/task/procedure/ProcedureTaskChannelFactory.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-python/src/main/java/org/apache/dolphinscheduler/plugin/task/python/PythonCommandExecutor.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-shell/src/main/java/org/apache/dolphinscheduler/plugin/task/shell/ShellTask.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java
#	dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopTask.java
#	dolphinscheduler-ui/package.json
#	dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/dag.scss
#	dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/dag.vue
#	dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/formModel.vue
#	sql/dolphinscheduler_h2.sql
2021-09-08 16:44:07 +08:00
Kirs
9508d1e978
[SPI][TASK]TaskPlugin (#6122)
* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952)

* [Bug][WorkerServer] SqlTask NullPointerException #5549

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428

* fix the readme typing issue (#5998)

* Fix unchecked type conversions

* Use indentation level reported by checkstyle

* Reorganize CI workflows to fasten the wasted time and resources (#6011)

* Add standalone server module to make it easier to develop (#6022)

* Task node of SWITCH (#5939)

* [Feature-#5273][server-master] Task node of SWITCH (#5922)



Co-authored-by: wangxj <wangxj31>

* remove description of bonecp (#6030)

Co-authored-by: shaojwu <shaojwu@ebay.com>

* [Improvement][Api Module]split alert group list-paging interface (#5941)

* [Improvement][Api Module]split alert group list-paging interface

* [FIX-#6007]Wrong complement date (#6026)

* [FIX-#6007]Wrong complement date

* [style]Wrong complement date

* [Improvement-6024][dist] Remove useless packaging commands (#6029)

·Remove useless packaging commands in dolphinscheduler-bin.xml

This closes #6024

Co-authored-by: mask <liuhu@zhiyoutec.com>

* [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909)

* fix the npe in MasterExec

* fix the compile error

* Add `.asf.yaml` to easily set the GitHub metadata (#6035)

* fix dead server cannot stop (#6046)

* Enhancement Translation (#6042)

* replaced Loading... with i18n

* modified Edit zh_CN translation

* Delete zh_CN.js

Co-authored-by: David <dailidong66@gmail.com>

* fix bug #6053 zh_CN.js is lost

* [Fix-6038][ui] width of "SQL Statement" in Dag FormLineModal will be shrunk if sql line is too long (#6040)

This closes #6038

* [Improvement] Fix inefficient map iterator (#6004)

* Fix inefficient map iterator

* Use forEach and remove call to valueOf

* Modify AbstractParameters

* Enhance `StandaloneServer` so that we don't need to update the version number manually (#6074)

* Remove invalid character in `.asf.yaml` (#6075)

* Remove invalid character `\n` in `.asf.yaml` (#6077)

It turns out that the invalid character is `\n`

* Add alert server into standalone-server as well and some minor polish (#6087)

* Support starting standalone server in Docker image (#6102)

Also remove unused class

* [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6095)

* [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6085)

* master refactor:
   1. spi for task submit and other actions(pause, kill)
   2. remove threads for process instance and task instance.
   3. add events for process instance and task instance

* ut npe

* add try catch

* code style

* fix critical bugs

* fix critical bugs

* fix critical bugs

* fix critical bugs

* Remove unused params in SwitchTaskTest (#6109)

* [Feature-5987][Server] Support to set multiple environment configs for a certain worker. (#6082)

* support multi environments

* add some test cases

* add an environment vue component

* improve environment form

* improve environment form

* add environment worker group relation

* add environment worker group relation

* add the environment choice for formModel

* set an environment for the task

* modify the modal form of starting process

* add the environment config to TaskExecutionContext

* add the environment config to the timing form

* fix conflicts

* fix issues of the code style

* fix some issues of the code style

* fix some issues of the code style

* fix some issues of the code style

* fix some issues of the code style

* fix some issues of the code style

* fix some bugs in the code review

* add the same table and columns to support H2

* fix some bugs

* [Plugin][Task]Task Spi

* [Plugin][Task]Task Spi

* [Plugin][Task]Task Spi

* [Plugin][Task]Task Spi

* [Plugin][Task]Task Spi

* [Plugin][Task]Task Spi

Co-authored-by: didiaode18 <563646039@qq.com>
Co-authored-by: Roy <yongjuncao1213@gmail.com>
Co-authored-by: lyxell <alyxell@kth.se>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: myangle1120 <942542838@qq.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: shaojwu <shaojwu@ebay.com>
Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com>
Co-authored-by: linquan <1175687813@qq.com>
Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com>
Co-authored-by: mask <liuhu@zhiyoutec.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com>
Co-authored-by: David <dailidong66@gmail.com>
Co-authored-by: lenboo <baoliang.leon@gmail.com>
Co-authored-by: lilyzhou <lj_zhou@outlook.com>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
Co-authored-by: junfan.zhang <zuston.shacha@gmail.com>
Co-authored-by: Hua Jiang <jianghuachinacom@163.com>
2021-09-08 10:40:40 +08:00
JinyLeeChina
33ea3eddb9
[Feature][JsonSplit-api] merge code from dev to json2 (#6115)
* [BUG-#5678][Registry]fix registry init node miss (#5686)

* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)

* improve

edit the userinfo success, but the updatetime is not the latest.

* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)

Co-authored-by: shenglm <shenglm840722@126.com>

* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)

* add globalParams new plan with varPool

* add unit test

* add python task varPoolParams


Co-authored-by: wangxj <wangxj31>

* Issue robot translation judgment changed to Chinese (#5694)



Co-authored-by: chenxingchun <438044805@qq.com>

* the update function should use post instead of get (#5703)

* enhance form verify (#5696)

* checkState only supports %s not {} (#5711)

* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)

* update

* fix the codestyle error

* fix the compile error

* support rollback

* [Fix-5699][UI] Fix update user error in user information (#5700)

* [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676)

* bug fix

the auto generated spi service can't be recongized



* include a  new method

* [Improvement-5622][project management] Modify the title (#5723)

* [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715)



* add a new init method.

* [Fix#5758] There are some problems in the api documentation that need to be improved (#5759)

* add the necessary parameters

* openapi improve

* fix code style error

* [FIX-#5721][master-server] Global params parameter missing (#5757)



Co-authored-by: wangxj <wangxj31>

* [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move`  doesn't work. (#5739)

* Update relatedItems.vue

* Update relatedItems.vue

* [Improvement#5741][Worker] Improve task process status log  (#5776)

* [Improvement-5773][server] need to support two parameters related to task (#5774)

* add some new parameter for task

* restore official properties

* improve imports

* modify a variable's name

Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>

* [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE

* [Improvement][Worker] Task log may be lost #5775 (#5783)

* [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789)

* [Imporvement #5725][CheckStyle] upgrade checkstyle file
  Upgrade checkstyle.xml to support checkstyle version 8.24+

* change ci checkstyle version

* [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected.  (#5796)

* improve timestamp format

make the startime in the log of httptask to be easier to read.


* fix bad code smell and update the note.

* [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797)

·the list of workflow instances is sorted by start time and end time
·This closes #5621

* fix (#5803)

Co-authored-by: shuangbofu <fusb@tuya.com>

* fix: Remove duplicate "registryClient.close" method calls (#5805)

Co-authored-by: wen-hemin <wenhemin@apache.com>

* [Improvement][SPI] support load single plugin (#5794)

change load operation of 'registry.plugin.dir'

* [Improvement][Api Module] refactor registry client, remove spring annotation (#5814)

* fix: refactor registry client, remove spring annotation

* fix UT

* fix UT

* fix checkstyle

* fix UT

* fix UT

* fix UT

* fix: Rename RegistryCenterUtils method name

Co-authored-by: wen-hemin <wenhemin@apache.com>

* [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735)

* [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727)

* enhance the validation in UI

* enchance form verifaction

* simplify disable condition

* fix: Remove unused class (#5833)

Co-authored-by: wen-hemin <wenhemin@apache.com>

* [fix-5737] [Bug][Datasource] datsource other param check error (#5835)

Co-authored-by: wanggang <wanggy01@servyou.com.cn>

* [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes

[Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes

* [Fix-5825][BUG][WEB] the resource tree in the process definition of latest dev branch can't display correctly (#5826)

* resoures-shows-error

* fix codestyle error

* add license header for new js

* fix codesmell

* [Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867)

* provide two system parameters to support the rest of type of tasks

* provide two system parameters to support the rest of type of tasks

* improve test conversion

* [Improvement][Fix-5769][UI]When we try to delete the existing dag, the console in web browser would shows exception (#5770)

* fix bug

* cache the this variable

* Avoid self name

* fix code style compile error

* [Fix-5781][UT] Fix test coverage in sonar (#5817)

* build(UT): make jacoco running in offline-instrumentation

issue: #5781

* build(UT): remove the jacoco agent dependency in microbench

issue: #5781

* [Fix-5808][Server]  When we try to transfer data using datax between  different types of data sources, the worker will exit with ClassCastException (#5809)

* bug fix

* fix bug

* simplify the code format

* add a new parameter to make it easier to understand.

* [Fix-5830][Improvement][UI] Improve the selection style in dag edit dialog (#5829)

* improve the selection style

* update another file

* remove unnecessary css part.

* [Fix-5904][upgrade]fix dev branch upgrade mysql sql script error (#5821)

* fix dev branch upgrade mysql sql script error.

* Update naming convention.

* [Improvement][Api Module] refactor DataSourceParam and DependentParam, remove spring annotation (#5832)

* fix: refactor api utils class, remove spring annotation.

* fix: Optimization comments

Co-authored-by: wen-hemin <wenhemin@apache.com>

* correct the wrong annotion from zk queue implemented to java priority blocking queue (#5906)

Co-authored-by: ywang46 <ywang46@paypal.com>

* Add a Gitter chat badge to README.md (#5883)

* Add Gitter badge

* Update README.md

Co-authored-by: David <dailidong66@gmail.com>

* ci: improve maven connection in CI builds (#5924)

issue: #5921

* [Improvement][Master]fix typo (#5934)

·fix typo in MasterBaseTaskExecThread

* [Fix-5886][server] Enhanced scheduler delete check (#5936)

* Add:Name verification remove the first and last spaces.

* Update: wrong word: 'WAITTING' ->'WAITING'

* Add: Strengthen verification

Co-authored-by: Squid <2824638304@qq.com>

* [Improvement-5880][api] Optimized data structure of pagination query API results (#5895)

* [5880][refactor]Optimized data structure of pagination query API results
- refactor PageInfo and delete returnDataListPaging in API
- modify the related Controller and Service and the corresponding Test

* Merge branch 'dev' of github.com:apache/dolphinscheduler into dev

 Conflicts:
	dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionServiceImpl.java

Co-authored-by: 蔡泽华 <sorea1k@163.com>

* [IMPROVEMENT]fix mysql comment error (#5959)

* [Improvement][Api]fix typo (#5960)

* [Imporvement #5621][job instance] start-time and end-time (#5621)
·the list of workflow instances is sorted by start time and end time
·This closes #5621

* [FIX-5975]queryLastRunningProcess sql in ProcessInstanceMapper.xml (#5980)

* [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism  (#5912)

* update

* web improved

* improve the ui

* add the ability to configure the parallelism

* update variables

* enhance the ut and add necessary note

* fix code style

* fix code style issue

* ensure the complation task in parallel mode can run the right numbers of tasks.

* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952)

* [Bug][WorkerServer] SqlTask NullPointerException #5549

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428

* [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428

* fix the readme typing issue (#5998)

* Fix unchecked type conversions

* Use indentation level reported by checkstyle

* Reorganize CI workflows to fasten the wasted time and resources (#6011)

* Add standalone server module to make it easier to develop (#6022)

* Task node of SWITCH (#5939)

* [Feature-#5273][server-master] Task node of SWITCH (#5922)



Co-authored-by: wangxj <wangxj31>

* remove description of bonecp (#6030)

Co-authored-by: shaojwu <shaojwu@ebay.com>

* [Improvement][Api Module]split alert group list-paging interface (#5941)

* [Improvement][Api Module]split alert group list-paging interface

* [FIX-#6007]Wrong complement date (#6026)

* [FIX-#6007]Wrong complement date

* [style]Wrong complement date

* [Improvement-6024][dist] Remove useless packaging commands (#6029)

·Remove useless packaging commands in dolphinscheduler-bin.xml

This closes #6024

Co-authored-by: mask <liuhu@zhiyoutec.com>

* [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909)

* fix the npe in MasterExec

* fix the compile error

* Add `.asf.yaml` to easily set the GitHub metadata (#6035)

* fix dead server cannot stop (#6046)

* Enhancement Translation (#6042)

* replaced Loading... with i18n

* modified Edit zh_CN translation

* Delete zh_CN.js

Co-authored-by: David <dailidong66@gmail.com>

* fix bug #6053 zh_CN.js is lost

* [Fix-6038][ui] width of "SQL Statement" in Dag FormLineModal will be shrunk if sql line is too long (#6040)

This closes #6038

* [Improvement] Fix inefficient map iterator (#6004)

* Fix inefficient map iterator

* Use forEach and remove call to valueOf

* Modify AbstractParameters

* Enhance `StandaloneServer` so that we don't need to update the version number manually (#6074)

* Remove invalid character in `.asf.yaml` (#6075)

* Remove invalid character `\n` in `.asf.yaml` (#6077)

It turns out that the invalid character is `\n`

* Add alert server into standalone-server as well and some minor polish (#6087)

* Support starting standalone server in Docker image (#6102)

Also remove unused class

* [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6095)

* [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6085)

* master refactor:
   1. spi for task submit and other actions(pause, kill)
   2. remove threads for process instance and task instance.
   3. add events for process instance and task instance

* ut npe

* add try catch

* code style

* fix critical bugs

* fix critical bugs

* fix critical bugs

* fix critical bugs

* Remove unused params in SwitchTaskTest (#6109)

* fix ut

* fix ut

* fix ut

* fix ut

* fix server module ut

* fix ut

Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: Shiwen Cheng <chengshiwen0103@gmail.com>
Co-authored-by: Jianchao Wang <akingchao@qq.com>
Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com>
Co-authored-by: Hua Jiang <jianghuachinacom@163.com>
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
Co-authored-by: Wenjun Ruan <861923274@qq.com>
Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com>
Co-authored-by: 傅双波 <786183073@qq.com>
Co-authored-by: shuangbofu <fusb@tuya.com>
Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com>
Co-authored-by: wen-hemin <wenhemin@apache.com>
Co-authored-by: geosmart <geosmart@hotmail.com>
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
Co-authored-by: AzureCN <colorazure@163.com>
Co-authored-by: 深刻 <tsund@qq.com>
Co-authored-by: zhuangchong <37063904+zhuangchong@users.noreply.github.com>
Co-authored-by: Yao WANG <Yao.MR.CN@gmail.com>
Co-authored-by: ywang46 <ywang46@paypal.com>
Co-authored-by: The Gitter Badger <badger@gitter.im>
Co-authored-by: David <dailidong66@gmail.com>
Co-authored-by: Squidyu <1297554122@qq.com>
Co-authored-by: Squid <2824638304@qq.com>
Co-authored-by: soreak <60459867+soreak@users.noreply.github.com>
Co-authored-by: 蔡泽华 <sorea1k@163.com>
Co-authored-by: yimaixinchen <yimaixinchen@163.com>
Co-authored-by: atai-555 <74188560+atai-555@users.noreply.github.com>
Co-authored-by: didiaode18 <563646039@qq.com>
Co-authored-by: Roy <yongjuncao1213@gmail.com>
Co-authored-by: lyxell <alyxell@kth.se>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: myangle1120 <942542838@qq.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: shaojwu <shaojwu@ebay.com>
Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com>
Co-authored-by: linquan <1175687813@qq.com>
Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com>
Co-authored-by: mask <liuhu@zhiyoutec.com>
Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com>
Co-authored-by: lenboo <baoliang.leon@gmail.com>
Co-authored-by: lilyzhou <lj_zhou@outlook.com>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
Co-authored-by: junfan.zhang <zuston.shacha@gmail.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-09-07 17:15:05 +08:00
Kerwin
22c5292b93
[Feature][SPI] Task (#6044) 2021-08-27 15:02:24 +08:00
JinyLeeChina
7427dc6fd3
[Feature][JsonSplit-api]merge code from dev to json_split_two (#5923)
* [BUG-#5678][Registry]fix registry init node miss (#5686)

* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)

* improve

edit the userinfo success, but the updatetime is not the latest.

* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)

Co-authored-by: shenglm <shenglm840722@126.com>

* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)

* add globalParams new plan with varPool

* add unit test

* add python task varPoolParams


Co-authored-by: wangxj <wangxj31>

* Issue robot translation judgment changed to Chinese (#5694)



Co-authored-by: chenxingchun <438044805@qq.com>

* the update function should use post instead of get (#5703)

* enhance form verify (#5696)

* checkState only supports %s not {} (#5711)

* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)

* update

* fix the codestyle error

* fix the compile error

* support rollback

* [Fix-5699][UI] Fix update user error in user information (#5700)

* [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676)

* bug fix

the auto generated spi service can't be recongized



* include a  new method

* [Improvement-5622][project management] Modify the title (#5723)

* [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715)



* add a new init method.

* [Fix#5758] There are some problems in the api documentation that need to be improved (#5759)

* add the necessary parameters

* openapi improve

* fix code style error

* [FIX-#5721][master-server] Global params parameter missing (#5757)



Co-authored-by: wangxj <wangxj31>

* [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move`  doesn't work. (#5739)

* Update relatedItems.vue

* Update relatedItems.vue

* [Improvement#5741][Worker] Improve task process status log  (#5776)

* [Improvement-5773][server] need to support two parameters related to task (#5774)

* add some new parameter for task

* restore official properties

* improve imports

* modify a variable's name

Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>

* [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE

* [Improvement][Worker] Task log may be lost #5775 (#5783)

* [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789)

* [Imporvement #5725][CheckStyle] upgrade checkstyle file
  Upgrade checkstyle.xml to support checkstyle version 8.24+

* change ci checkstyle version

* [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected.  (#5796)

* improve timestamp format

make the startime in the log of httptask to be easier to read.


* fix bad code smell and update the note.

* [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797)

·the list of workflow instances is sorted by start time and end time
·This closes #5621

* fix (#5803)

Co-authored-by: shuangbofu <fusb@tuya.com>

* fix: Remove duplicate "registryClient.close" method calls (#5805)

Co-authored-by: wen-hemin <wenhemin@apache.com>

* [Improvement][SPI] support load single plugin (#5794)

change load operation of 'registry.plugin.dir'

* [Improvement][Api Module] refactor registry client, remove spring annotation (#5814)

* fix: refactor registry client, remove spring annotation

* fix UT

* fix UT

* fix checkstyle

* fix UT

* fix UT

* fix UT

* fix: Rename RegistryCenterUtils method name

Co-authored-by: wen-hemin <wenhemin@apache.com>

* [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735)

* [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727)

* enhance the validation in UI

* enchance form verifaction

* simplify disable condition

* fix: Remove unused class (#5833)

Co-authored-by: wen-hemin <wenhemin@apache.com>

* [fix-5737] [Bug][Datasource] datsource other param check error (#5835)

Co-authored-by: wanggang <wanggy01@servyou.com.cn>

* [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes

[Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes

* [Fix-5825][BUG][WEB] the resource tree in the process definition of latest dev branch can't display correctly (#5826)

* resoures-shows-error

* fix codestyle error

* add license header for new js

* fix codesmell

* [Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867)

* provide two system parameters to support the rest of type of tasks

* provide two system parameters to support the rest of type of tasks

* improve test conversion

* [Improvement][Fix-5769][UI]When we try to delete the existing dag, the console in web browser would shows exception (#5770)

* fix bug

* cache the this variable

* Avoid self name

* fix code style compile error

* [Fix-5781][UT] Fix test coverage in sonar (#5817)

* build(UT): make jacoco running in offline-instrumentation

issue: #5781

* build(UT): remove the jacoco agent dependency in microbench

issue: #5781

* [Fix-5808][Server]  When we try to transfer data using datax between  different types of data sources, the worker will exit with ClassCastException (#5809)

* bug fix

* fix bug

* simplify the code format

* add a new parameter to make it easier to understand.

* [Fix-5830][Improvement][UI] Improve the selection style in dag edit dialog (#5829)

* improve the selection style

* update another file

* remove unnecessary css part.

* [Fix-5904][upgrade]fix dev branch upgrade mysql sql script error (#5821)

* fix dev branch upgrade mysql sql script error.

* Update naming convention.

* [Improvement][Api Module] refactor DataSourceParam and DependentParam, remove spring annotation (#5832)

* fix: refactor api utils class, remove spring annotation.

* fix: Optimization comments

Co-authored-by: wen-hemin <wenhemin@apache.com>

* correct the wrong annotion from zk queue implemented to java priority blocking queue (#5906)

Co-authored-by: ywang46 <ywang46@paypal.com>

Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: Shiwen Cheng <chengshiwen0103@gmail.com>
Co-authored-by: Jianchao Wang <akingchao@qq.com>
Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com>
Co-authored-by: Hua Jiang <jianghuachinacom@163.com>
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
Co-authored-by: Wenjun Ruan <861923274@qq.com>
Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com>
Co-authored-by: 傅双波 <786183073@qq.com>
Co-authored-by: shuangbofu <fusb@tuya.com>
Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com>
Co-authored-by: wen-hemin <wenhemin@apache.com>
Co-authored-by: geosmart <geosmart@hotmail.com>
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
Co-authored-by: AzureCN <colorazure@163.com>
Co-authored-by: 深刻 <tsund@qq.com>
Co-authored-by: zhuangchong <37063904+zhuangchong@users.noreply.github.com>
Co-authored-by: Yao WANG <Yao.MR.CN@gmail.com>
Co-authored-by: ywang46 <ywang46@paypal.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-07-31 11:19:41 +08:00
深刻
4c0993cdf4
[Fix-5781][UT] Fix test coverage in sonar (#5817)
* build(UT): make jacoco running in offline-instrumentation

issue: #5781

* build(UT): remove the jacoco agent dependency in microbench

issue: #5781
2021-07-26 22:04:36 +08:00
JinyLeeChina
57414c4df7
[Feature][JsonSplit-api] merging from dev to json_split_two (#5712)
* [BUG-#5678][Registry]fix registry init node miss (#5686)

* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)

* improve

edit the userinfo success, but the updatetime is not the latest.

* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)

Co-authored-by: shenglm <shenglm840722@126.com>

* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)

* add globalParams new plan with varPool

* add unit test

* add python task varPoolParams


Co-authored-by: wangxj <wangxj31>

* Issue robot translation judgment changed to Chinese (#5694)



Co-authored-by: chenxingchun <438044805@qq.com>

* the update function should use post instead of get (#5703)

* enhance form verify (#5696)

* checkState only supports %s not {} (#5711)

* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)

* update

* fix the codestyle error

* fix the compile error

* support rollback

Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-06-29 22:55:01 +08:00
kyoty
239cfe5027
checkState only supports %s not {} (#5711) 2021-06-29 01:03:52 +08:00
Shiwen Cheng
93f1d4df7d
[Improvement-5405][Docker/K8s] Adapt to registry SPI (#5681) 2021-06-23 11:30:15 +08:00
kyoty
75813e7866
[bug fix] Some password attributes can be viewd directly in the log (#5638)
* fix the password display incorrectly

* update

* update

* fic code style

fic code style

* Update SensitiveDataConverterTest.java

* Update SensitiveDataConverter.java

* Update SensitiveDataConverter.java

* Update SensitiveDataConverterTest.java

* Update pom.xml
2021-06-18 19:10:57 +08:00
Kirs
30c88032d3 [Feature-#3961][Registry]Registry-SPI (#5562)
[Feature#3961]Registry SPI

All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
2021-06-15 10:30:26 +08:00
Kirs
e2243d63be
[Feature-#3961][Registry]Registry-SPI (#5562)
[Feature#3961]Registry SPI

All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
2021-06-09 15:21:02 +08:00
Shiwen Cheng
b537f4eb26
[Improvement][Docker/K8s] Support more configurations and service accesses, reduce the duplication configurations, improve image for python and the compatibility of python home (#5393)
* [Improvement][Docker] Support more configurations and improve image for python

* [Improvement][K8s] Support more configurations, more service accesses, and reduce the duplication configurations

* [Improvement][Python] Improve the compatibility of python home

* [Improvement][Install] Fix install config and install.sh

* [Improvement][Install] Fix workflow ut

* [Improvement][Docker] Optimize docker-swarm/check script

* [Improvement][DB] Update default username and password of database

* Update comments in master/worker.properties

* Specify version for all images

* [Improvement][Docker] Optimize PS1 and WORKDIR

* [Improvement][SQL] Reduce heap size to 64m in create-dolphinscheduler.sh and upgrade-dolphinscheduler.sh

* [Fix-5431][K8s] Fix master and worker cannot get the right address with custom DNS
2021-05-24 00:03:35 +08:00
Kirs
8a1d849701
[ut]ignore plugin ut (#5532) 2021-05-21 10:36:08 +08:00
zhuangchong
1b7dcea610
[Improvement-5440][Maven] Project module maven dependency optimization (#5441)
* Optimize dependencies between modules.

* update module-alert dependency.

* add repository in the pom xml.

* update Central Repository in the pom.xml.
2021-05-19 10:05:00 +08:00
JinyLeeChina
515fa4912e
[Feature][JsonSplit] Fix ProcessService ut (#5435)
* Fix ProcessService ut

* codeStyle

* Fix AlertPlugin ut

* Fix AlertPlugin ut

* Fix server module ut

* Fix server module ut

* Fix api ut

* codeStyle

Co-authored-by: JinyLeeChina <297062848@qq.com>
2021-05-10 23:17:56 +08:00
Shiwen Cheng
0f3eed9d00
[Fix-5386][Alert] Fix ERROR: relation t_ds_plugin_define does not exist (#5387) 2021-04-29 16:17:01 +08:00
Ts686
6216817861
[Improvement-4971][alert|dao|service]Remove redundant PropertyUtils.class (#5020)
* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.

* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.

* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.

* [Improvement][alert|dao|service]
1. Remove redundant PropertyUtils.class (#4971)
2. Remove useless import package.
2021-03-17 15:28:07 +08:00