Commit Graph

93 Commits

Author SHA1 Message Date
Rick Cheng
7ee66f2d02
[Feature-13429][Remote Logging] Add support for writing task logs to Google Cloud Storage (#13777) 2023-04-06 17:49:33 +08:00
Wenjun Ruan
41a8ba9aab
Support alert server HA (#13865)
* Support alert server cluster

* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2023-04-05 17:31:50 +08:00
Wenjun Ruan
c9066e8de9
Use MDC to filter task instance log (#13673)
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2023-03-06 17:44:23 +08:00
Aaron Wang
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes (#13550) 2023-02-21 23:26:21 +08:00
seedscoder
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger (#13509) 2023-02-07 20:32:53 +08:00
Wenjun Ruan
14ec4a2398
Remove dao module in worker (#13242) 2022-12-22 12:25:29 +08:00
Wenjun Ruan
70ccffeee2
Format task parameter as pretty json (#13173) 2022-12-13 16:30:21 +08:00
rickchengx
38b876733c
[Feature-10498] Mask the password in the log of sqoop task (#11589) 2022-11-24 14:54:54 +08:00
John Bampton
27c37b8828
Fix grammar and spelling (#12937) 2022-11-18 23:03:34 +08:00
Jay Chung
6d8befa075
[fix] Add token as authentication for python gateway (#12893)
separate from #6407. Authentication, add secret to ensure only trusted people could connect to gateway.

fix: #8255
2022-11-14 18:43:08 +08:00
Tq
d324cce6f1
[Feature-12526][Api] add dynamic task function: shell task (#12840)
* submit dynamic task code

* submit shell.json

* add dynamic-task-type-config.yaml to standalone server
2022-11-14 13:52:44 +08:00
Eric Gao
6d79414491
Add instructions on how to configure logging level for debugging purpose (#12699) 2022-11-07 10:30:56 +08:00
Wenjun Ruan
e6da1ccf81
Add worker-group-refresh-interval in master config (#12601)
* Add worker-group-refresh-interval in master config

* Set interval cannot smaller than 10s

* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-10-31 09:37:26 +08:00
kezhenxu94
065d5caccc
Only expose necessary actuator endpoints (#12571) 2022-10-28 07:40:32 +08:00
Wenjun Ruan
76a9e6d8a6
Add task executor threads full policy config in worker (#12510) 2022-10-26 10:29:22 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412)
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
kezhenxu94
4ea7eec750
Use temurin Java docker images instead of deprecated ones (#12334) 2022-10-17 15:46:21 +08:00
caishunfeng
40090d459e
[Improvement] remove log-server and server module (#12206)
* remove log-server module

* remove dolphinscheduler-server module
2022-09-30 16:41:41 +08:00
Daniel Y
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 (#11835)
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9

* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-14 09:29:45 +08:00
He Zhao
ca22c083b3
[Fix] [Quartz] cron did not work as expected (#11851)
* [Fix] [Quartz] cron did not work as expected

* [Fix] [Quartz] make the value of batchTriggerAcquisitionMaxCount to 1(default value)

Co-authored-by: hezhao2 <hezhao2@cisco.com>
2022-09-13 16:05:40 +08:00
kezhenxu94
277f137358
Add Kubernetes configmap reload to all components (#11730) 2022-09-02 12:03:19 +08:00
Tq
b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav (#11727)
* feature: add task type list and enable user to add task type to fav

* make DependentExecute.java variable protected

* add standalone server fileSet
2022-09-02 11:40:23 +08:00
kezhenxu94
13e4720aa2
Upgrade spring boot and swagger (#11574) 2022-08-22 10:27:40 +08:00
Wenjun Ruan
3516533017
Remove logger header in task log file (#11555) 2022-08-19 14:01:52 +08:00
kezhenxu94
f6ef628030
Support reading application properties from kubernetes config map (#11402) 2022-08-16 20:25:17 +08:00
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
Wenjun Ruan
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979)
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed

* Move heartbeat error threshold to applicaiton.yml
2022-07-15 20:06:53 +08:00
pinkhello
7a15877fa4
[FIX-10784][Bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop (#10888)
* closed 10784 [Bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop
2022-07-13 14:27:44 +08:00
Wenjun Ruan
2a67866718
Catch exception when check state in StateWheelExecuteThread (#10908)
* Catch exception when check state
2022-07-13 10:51:20 +08:00
Wenjun Ruan
35b25da863
Validate master/worker config (#10649) 2022-06-28 20:17:43 +08:00
xiangzihao
1111371c9a
add datasource health check to the healthcheck endpoint (#10588) 2022-06-24 13:29:49 +08:00
Wenjun Ruan
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516)
* Optimize master log, add workflow instance id and task instance id in log

* Use MDC to set the workflow info in log4j

* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
旺阳
6e4b2e6992
[Fix]Recovery Traffic Control Config (#10505)
* recovery traffic config

* run ci

* update
2022-06-20 18:43:48 +08:00
旺阳
6515c66208
[Improve] Add LDAP user not exsitst action config (#10451) 2022-06-17 14:18:05 +08:00
Wenjun Ruan
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479)
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
旺阳
0efcd5c631
[Fix-10425]Recovery LDAP Config (#10429)
* recovery ladp code

* add ldap config in doc
2022-06-15 09:52:37 +08:00
xiangzihao
f4dad689d4
[Fix-10412] [Standalone] Failed to start standalone server according to the official document (#10414)
* fix bug_10412

* fix bug_10412
2022-06-12 16:16:57 +08:00
Wenjun Ruan
e21d7b1551
[Feature][metrics] Add master, worker metrics (#10326)
* Add mater metrics

* fix UT

* Add url to mysql profile

* Add worker metrics

* Update grafana config

* Add system metrics doc

* Add process failover counter

* Add metrics image

* Change jpg to png

* Add command insert metrics

* Fix UT

* Revert UT
2022-06-09 10:55:39 +08:00
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues (#9944) 2022-05-31 15:22:41 +08:00
lugela
a0771541e5
[Fix-10181] Fix the logic of judging that the tenant does not exist (#10185)
* [Fix-10181] Fix the logic of judging that the tenant does not exist

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.

At present, considering that it is a distributed user, it should not be allowed to create users in linux

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

Add test method

The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.

At present, considering that it is a distributed user, it should not be allowed to create users in linux

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

Add parameter description to configuration.md

Add test method

The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.

At present, considering that it is a distributed user, it should not be allowed to create users in linux

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

Add parameter description to configuration.md

Add test method

The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.

At present, considering that it is a distributed user, it should not be allowed to create users in linux

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

* [Fix-10181] Fix the logic of judging that the tenant does not exist

Add parameter description to configuration.md

Add test method

The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.

At present, considering that it is a distributed user, it should not be allowed to create users in linux

Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap :  uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist  in /etc/passwd file and ldap: id: test: no such user

Temporarily unable to test the system for windows and mac

Co-authored-by: ouyangl <ouyangl@tebon.com.cn>
2022-05-26 14:58:07 +08:00
旺阳
aba5f8a40e
[improve] Change Mysql Driver (#10220) 2022-05-25 14:09:15 +08:00
Paul Zhang
8562f6a878
[Feature][Log]Add timezone information in log output (#9913) 2022-05-06 17:31:44 +08:00
songjianet
7a7807e520
[Feature] Remove old ui. (#9909)
* [Feature] Remove old ui.

* [Docs] Add license file for screenfull.
2022-05-06 14:04:35 +08:00
Kerwin
4f2b9ac886
[Improve][common] code clean in the common module (#9435) 2022-04-29 10:34:03 +08:00
Eric Gao
3a66f05baf
[doc] Add how to change standalone database type (#9376) (#9736) 2022-04-26 09:47:22 +08:00
Jiajie Zhong
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server (#9726)
* [common] Make dolphinscheduler_env.sh work

* Change dist tarball `dolphinscheduler_env.sh` location
  from `bin/` to `conf/`, which users could finish their
  change configuration operation in one single directory.
  and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
  when we start our sever instead of adding both
  `$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
Paul Zhang
2be7183563
[Bug][Standalone Server] Deduplicate the classpath jars in start.sh of the standalone server (#9583) 2022-04-20 11:46:42 +08:00
Jiajie Zhong
3d78859fe5
[python] Recover python release properties (#9444)
This patch recovers the properties `python.sign.skip=false`
when the combined profile `release,python` is used.

also close: #9433
2022-04-12 09:42:59 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
Jiajie Zhong
3457cee960
[python] Migrate pythonGatewayServer into api server (#9372)
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server

The distribute package size before and after this patch is:

```sh
# before
796M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz

# after
647M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
2022-04-07 14:41:15 +08:00