Commit Graph

275 Commits

Author SHA1 Message Date
Wenjun Ruan
1f365819a6
Ignore unknown VM options in start.sh (#13719) 2023-03-10 17:34:09 +08:00
旺阳
aea84055fe
Fix webex teams alert plugin null exception error (#13591) 2023-02-21 20:10:48 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger (#13509) 2023-02-07 20:32:53 +08:00
Yann Ann
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 (#13002) 2022-12-10 23:50:19 +08:00
旺阳
602971c724
change alert start.sh (#13100) 2022-12-06 00:26:27 +08:00
Wenjun Ruan
169168ef34
Add plugin-all module (#13079) 2022-12-02 23:19:08 +08:00
Wenjun Ruan
ffc9fb280a
Add gc timestampt (#13059) 2022-12-01 10:00:27 +08:00
旺阳
d02991a2e6
[Bug] [Alert] Ignore alert not write info to db (#12867)
* add alert not match return info
2022-11-17 14:14:12 +08:00
Tq
70fe39bb2d
add spring-context to alert api (#12892) 2022-11-14 17:23:28 +08:00
Tq
250d81bcf8
[bug][alert-12865] translate alert input field tips to chinese when using chinese (#12879)
* submit email ding talk changes

* submit alert messages

* submit format

* submit format
2022-11-14 11:14:14 +08:00
kezhenxu94
065d5caccc
Only expose necessary actuator endpoints (#12571) 2022-10-28 07:40:32 +08:00
caishunfeng
138eab6eb3
fix import (#12529) 2022-10-25 21:39:06 +08:00
pandong
f5c814f23b
[fix#12439] [Alert] fix send script alert NPE (#12495)
* [fix#12439] [Alert] fix send script alert NPE
2022-10-25 15:11:54 +08:00
caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils (#12502)
* improve JSONUtil

* merge spi.utils into common.utils

* split constants && remove some copy utils method

* remove StringUtils
2022-10-23 21:23:54 +08:00
fuchanghai
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE (#12449) 2022-10-20 14:26:56 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412)
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
kezhenxu94
4ea7eec750
Use temurin Java docker images instead of deprecated ones (#12334) 2022-10-17 15:46:21 +08:00
Eric Gao
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom (#12311)
* move mockito dependencies from sub-modules to root pom

* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
Eric Gao
224508850e
[Migrate][Test] Migrate all UT cases from jUnit 4 to jUnit 5 in alert and api module (#12337)
* Migrate all UT cases from jUnit 4 to jUnit 5 in alert and api module

* Fix potential problems
2022-10-13 14:40:57 +08:00
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project (#12244)
* Fully remove the usage of powermock from the whole project

* Upgrade org.reflections to 0.10.12
2022-10-10 10:43:38 +08:00
Jiajie Zhong
ad4f344283
[fix][test] Fix flaky test in CI (#12017) 2022-09-28 14:05:26 +08:00
Eric Gao
b52da64001
[Improvement][UT] Upgrade jUnit to 5.+ (#10976) (#11332)
* [Improvement][UT] Upgrade jUnit to 5.+ (#10976)

* Refactor AlertServerTest with jUnit-5 as an example
2022-09-21 10:15:21 +08:00
Wenjun Ruan
cb4ef54a0e
Script cannot contains ''' in params (#12067) 2022-09-21 08:52:51 +08:00
Wenjun Ruan
4ad34483a5
Fix alert_send_status may throw duplicate key exception and add limit for query alert (#11953) 2022-09-16 13:26:33 +08:00
Daniel Y
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 (#11835)
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9

* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-14 09:29:45 +08:00
kezhenxu94
efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
Wenjun Ruan
ea0b5acccb
Remove not use web service comment in alert server (#11819) 2022-09-07 15:05:18 +08:00
kezhenxu94
277f137358
Add Kubernetes configmap reload to all components (#11730) 2022-09-02 12:03:19 +08:00
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory (#11614)
* Add spi priority factory

* Add doc

* Add override log

* Use lombok

* Add comment
2022-08-24 19:51:47 +08:00
Eric Gao
f93e93cfd2
[Feature][Style] Make sure there is one and only one empty line at the end of each java file (#11520) 2022-08-17 12:59:08 +08:00
Wenjun Ruan
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost (#11368)
* Add waiting strategy to support master/worker can recover from registry lost

* throw exception when zookeeper registry start failed due to interrupted
2022-08-13 09:52:03 +08:00
jackfanwan
abd5798755
[fix-10961][alert server]Change the content of alert to an array (#11033)
* Change the content of alert to an array

* Increase unit test coverage

* Modify the code format

* Modify the code format

* Modify judgment statement

* Deleting invalid References

* Modify the previous code implementation and improve the test data

* Modify the code format

* Move the location of the Content conversion

* Remove useless code

* Change the code format

* Modify unit tests

* Modify unit tests

* Modify unit tests

* Changing the location of the code

* Modify the code

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
Co-authored-by: 范万龙 <fanwanlong@fanwanlongs-MBP.lan>
2022-08-12 09:57:27 +08:00
Wenjun Ruan
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
Eric Gao
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server (#11240)
* [Improvement][Metrics] Add metrics for alert server (#11131)

* Update related docs of metrics

* Add grafana demo dashboards for alert server metrics

* Refactor metric classes with UtilityClass annotation

* Refactor meter names in camelCase for checkstyle
2022-08-03 15:42:06 +08:00
Wenjun Ruan
caa6287d7f
Add alertPluginInstanceId in AlertInfo (#11231) 2022-08-01 17:32:26 +08:00
Wenjun Ruan
8e21c38c00
Write alert result into db (#11221) 2022-08-01 13:04:27 +08:00
SongTao Zhuang
a52d1b4acd
[11130] Refactor alert-api with lombok (#11133) 2022-07-28 08:31:44 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
pinkhello
7a15877fa4
[FIX-10784][Bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop (#10888)
* closed 10784 [Bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop
2022-07-13 14:27:44 +08:00
Eric Gao
43d7cc8a98
[Improvement] [Alert Server] Add default datasource url in alert server application.yaml for consistency (#10790)
* [Improvement] [Alert Server] Add default datasource url in alert server application.yaml for consistency (#10788)

* [Improvement] [Alert Server] Add newline at end of file (#10788)
2022-07-05 17:23:44 +08:00
xiangzihao
1111371c9a
add datasource health check to the healthcheck endpoint (#10588) 2022-06-24 13:29:49 +08:00
xiangzihao
0f38217b12
fix_10514 (#10568) 2022-06-23 16:15:08 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516)
* Optimize master log, add workflow instance id and task instance id in log

* Use MDC to set the workflow info in log4j

* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
Wenjun Ruan
e21d7b1551
[Feature][metrics] Add master, worker metrics (#10326)
* Add mater metrics

* fix UT

* Add url to mysql profile

* Add worker metrics

* Update grafana config

* Add system metrics doc

* Add process failover counter

* Add metrics image

* Change jpg to png

* Add command insert metrics

* Fix UT

* Revert UT
2022-06-09 10:55:39 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
juzimao
62bd9d2c4c
[Fix] Fix the line.separator problem for alert-plugin' test case #10277 (#10278)
* [Fix] Fix the line.separator problem for alert-plugin' test case

* [Fix] Fix the line.separator problem for alert-plugin' test case
2022-05-29 18:44:26 +08:00
旺阳
aba5f8a40e
[improve] Change Mysql Driver (#10220) 2022-05-25 14:09:15 +08:00
旺阳
fa2f20577a
Fix unit test and remove unused import (#10113) 2022-05-19 11:25:12 +08:00
旺阳
844ae42ee4
Fix alert server curator health check not work (#10111) 2022-05-19 10:51:37 +08:00
Paul Zhang
8562f6a878
[Feature][Log]Add timezone information in log output (#9913) 2022-05-06 17:31:44 +08:00
Tq
69810a8a36
[BUG][ALERT-SERVER]validate script before alert script (#9834)
* validate script before alert script

* fix validate script before alert script
2022-04-28 18:16:37 +08:00
Tq
7b0b4c126e
[Bug] [Website-9764]add http alert doc and change body param to unnecessary (#9792)
* add http alert doc and change body param to unnecessary

* submit http en doc and fix doc

* fix http doc
2022-04-27 17:26:58 +08:00
Tq
6788c99578
[Bug] [Alert-9786] Http alert Get method not working (#9787)
* fix http alert issues encode before get request

* add hint to http alert
2022-04-26 17:55:34 +08:00
Jiajie Zhong
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server (#9726)
* [common] Make dolphinscheduler_env.sh work

* Change dist tarball `dolphinscheduler_env.sh` location
  from `bin/` to `conf/`, which users could finish their
  change configuration operation in one single directory.
  and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
  when we start our sever instead of adding both
  `$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
worry
ecf13a8c90
[Improvement][style]add comment and clear warn (#9247) 2022-04-21 11:42:08 +08:00
Tq
51d47de6de
[Bug] [ALERT-9487] change email attachment filename to add random string to avoid issues (#9498)
* change email attachment filename to add random string to avoid always use same file

* change email attachment filename to add random string to avoid always use same file
2022-04-15 10:13:13 +08:00
czeming
706cdb6a8e
[Fix-9174] [Alert] Fix deduplication of alarm information (#9371)
* feat(issue #9174):

Fix-9174
2022-04-14 10:30:25 +08:00
Tq
df791a374e
[FIX][WEBSITE-9224]fix wechat doc and wechat form display (#9439)
Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
2022-04-13 23:35:05 +08:00
guoshupei
ca95d2f928
[Fix-9221] [alert-server] optimization and gracefully close (#9246)
* [Fix-9221] [alert-server] optimization and gracefully close

This closes #9221

* [Fix-9221] [alert-server] remove unused mock data

This closes #9221

* [Fix-9221] [alert-server] remove unused mock data

This closes #9221

* [Fix-9221] [alert-server] remove unnecessary Mockito stubbings

* [Fix-9221] [alert-server] init AlertPluginManager in AlertServer

* [Fix-9221] [alert-server] AlertServerTest add AlertPluginManager installPlugin

* [Fix-9221] [alert-server] replace @Eventlistener with @PostConstruct

* [Fix-9221] [alert-server] sonar check solution

* [Improvement-9221] [alert] update constructor injection and replace IStoppable with Closeable

Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-04-08 10:02:10 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
Tq
c294979e2f
[Bug-9235][Alert]Fix wechat markdown message and change wechat form structure (#9367)
* fix wechat issues:
1. change table msg type to markdown.
2. change userId to not required and enrich hints
3. change 'app id' to 'app id and chat id'

* fix wechat issues:
1. revert table showtype and add markdown showtype.
2. enrich hints.
3. delete 'chatid', rename agentid to weChatAgentIdChatId.
4. modify code to send markdown message.

* fix wechat issues: Change the language pack of agentId to agentId/chatId.

* fix format

* fix param name

Co-authored-by: Amy <amywang0104@163.com>
2022-04-06 20:29:30 +08:00
worry
2bab12f2c8
[Feature-9204][alert] Implement alert send status (#9208)
* [DS-9204][feat][alert,dao] Implement alert send status
- implement alert send status
- add alert send status entity、mapper
- modify alert dao
- modify alert sender
- add test
- add sql

This closes #9204

* [DS-9204][feat][alert,dao] Implement alert send status
- add license header

This closes #9204
2022-04-06 18:08:00 +08:00
worry
b4b5241797
[DS-9266][feat][server]Implement show dolphinscheduler banner at launch (#9273)
- add banner.txt
- modify application.yaml

This closes #9266
2022-03-30 16:45:24 +08:00
Tq
03a0c9f544
[Bug] [ALERT-9224] fix wechat alert functions: app and group chat (#9225)
* fix wechat alert with app and group chat

* add apache header and fix imports

* delete wechat alert 'userSendMsg' property code

* delete wechat alert 'userSendMsg' property code in Test

* delete wechat alert 'userSendMsg' property code in factory Test

* change `getSubWorkFlowParam` from private to protected
2022-03-28 20:33:14 +08:00
caishunfeng
263791a63e
[Bug-9137][Server] fix timezone init (#9219)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
Tq
97807390a2
fix alert instance proxy configs when no username and password (#9171) 2022-03-25 09:43:59 +08:00
Tq
b9448c3f6f
fix email alert session configs (#9142) 2022-03-24 09:50:50 +08:00
wangyang
0d8079a816
[DS-7992][feat] Alert module judging strategy (#8636) 2022-03-14 14:25:27 +08:00
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477)
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
kezhenxu94
fa19006a18
Add a property to enable/disable metrics module (#8484) 2022-02-24 10:21:47 +08:00
Kerwin
36d526d3a8
[Improvement-8284][Alert] Dingtalk alert plugin supports markdown message type (#8285)
* add msgtype in the dingtalk alert plugin

* update markdown msgtype 'at persion'
2022-02-07 22:07:37 +08:00
pinkhello
84b004d960
[Feature-8245][Alert] Add Alert Plugin Telegram (#8246) 2022-02-07 09:51:18 +08:00
wangyang
49d581d72e
[DS-8215][feat] Support WebexTeams Plugin (#8217)
Co-authored-by: qingwli <qingwli@cisco.com>
2022-01-29 11:36:38 +08:00
wind
128ede3ee9
Fix start standalone fail because db not init yet (#8191) 2022-01-25 18:10:02 +08:00
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
wangyang
14908a25e8
[Feature-7992][Alert] Support PagerDuty Plugin (#8120)
Co-authored-by: qingwli <qingwli@cisco.com>
2022-01-25 10:04:21 +08:00
Kerwin
1fd748f376
[Feature-8138][Task] add at@ msg in the dingtalk task plugin (#8139) 2022-01-21 17:09:39 +08:00
Kerwin
cb8c875fa8
[Feature-7804][Task] DingTalk alert plugin adds signature. (#7805)
* DingTalk alarm component adds signature.

* fix code style.

* fix code style.

* add dingtalk document link

* fix DingTalkAlertChannelFactoryTest

* update NAME_DING_TALK_SECRET
2022-01-12 15:24:25 +08:00
springmonster
4f4d9d12ce
[Bug-7849] Add unit test for alert server, fix #7849 (#7850) 2022-01-11 12:46:21 +08:00
kezhenxu94
68906f1b31
Remove fast return for alert plugins (#7723) 2021-12-30 15:02:03 +08:00
springmonster
2b05ebf47e
[Refactor] Canonical enum naming (#7676)
* [Refactor] Refactor comments
2021-12-28 19:07:53 +08:00
Jiezhi.G
6562bc9145
fix http alarm msg ignored (#7523)
See https://github.com/apache/dolphinscheduler/issues/7522
2021-12-21 16:04:12 +08:00
SbloodyS
86b476a160
[Feature-7460][Alert] Wechat alert add send to group chat (#7465)
* feature_7460

* feature_7460

Co-authored-by: SbloodyS <sbloodys@qq.com>
2021-12-18 10:26:16 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
Zonglei Dong
fa8ccd3855
Refactor code, add class import, keep ‘logger’ variable names uniform. (#7164) 2021-12-04 21:18:43 +08:00
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time (#7054) 2021-12-01 13:55:26 +08:00
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config (#6947)
* [DS-5229][fix] server port custom config
This closes #5229

* [DS-5229][feat] Implement server port custom config
This closes #5229

* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862)

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link

* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md

* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852)

Co-authored-by: caishunfeng <534328519@qq.com>

* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229

Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
2021-12-01 00:21:03 +08:00
kezhenxu94
2de4a3cbb5
Add a demo docker-compose for testing the meter module (#6938) 2021-11-24 17:22:30 +08:00
LinKai
ad33d43498
[Improvement][Dao] SQL optimization - limit one piece of data #5061 (#6940)
* [Feature][unittest] Recover UT in AlertPluginManagerTest.java #6619

* use throws exception instead of try-catch to catch exceptions in AlertPluginManagerTest.java

* change the assert result from assertNull to assertNotNull

* [Improvement][Dao] SQL optimization - limit one piece of data #5069

* Improve the code to A rate

Co-authored-by: linkai <linkai@bigo.sg>
2021-11-22 20:27:29 +08:00
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
mask
e36d18b588
[Fix-6792] resolve API and alert port conflicts (#6792) (#6815)
* [Fix-6792]Multiple ApplicationRunners on classpath make SpringApplication.run only run one (#6792)
2021-11-12 10:51:11 +08:00
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage (#6767) 2021-11-10 18:29:17 +08:00
kezhenxu94
05bae0b226
Refactor alert plugins and simplify its usage (#6723) 2021-11-09 18:59:34 +08:00
kezhenxu94
02350ea37b
Fix datasource not work in tests and standalone server, and run all tests by default (#6743) 2021-11-09 16:57:26 +08:00
kezhenxu94
63602bf77c
Clean up the utility codes (#6732) 2021-11-08 23:22:07 +08:00
Wenjun Ruan
0afb780718
[Improvement-5807][Dao] Use h2 in UT (#6500)
* Use H2 in UT

* change scope

* Change profile name to h2

* Add datasource-h2.properties, datasource-postgresql.properties

* Fix UT

* Resolve conflict

* Remove startDatabase
2021-11-07 23:19:19 +08:00
jon-qj
86c85114b2
make alert instance page display more friendly (#6635)
* make alert instance page display more friendly

* Update EmailAlertChannelTest.java
2021-10-31 21:16:10 +08:00
Jiajie Zhong
cbeedba3e2
Remove unused try block in UT (#6182)
closes https://github.com/apache/dolphinscheduler/issues/6181
2021-10-27 19:01:33 +08:00
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT (#6383) 2021-09-28 12:00:43 +08:00
kezhenxu94
a632be73c0
Clean up unused util classes... (#6154) 2021-09-10 00:35:39 +08:00