Commit Graph

445 Commits

Author SHA1 Message Date
John Bampton
d803658abd
[Document] Fix spelling (#12935) 2022-11-18 16:01:20 +08:00
John Bampton
22532fe1d8
[Document] Update faq, fix spelling, remove repeated words (#12934) 2022-11-18 15:36:47 +08:00
JieguangZhou
7c90bf01bc
[Improvement][Task Plugin] Improvement Kubeflow task plugin (#12928)
add example
check phase in status:conditions
2022-11-17 15:59:44 +08:00
Assert
2dbc79693e
[feature][task plugin] Add new task plugin for apache linkis (#12693) 2022-11-17 10:14:36 +08:00
xuhhui
9e6f4d2bc1
[Improvement][Document]Update architecture and process-start-flow pic (#12912)
Co-authored-by: xuhaihui <xuhaihui@cmss.chinamobile.com>
2022-11-16 16:03:00 +08:00
baihongbin
0373e06615
[Feature-8030][docs] Add sqoop task doc (#12855)
* [Feature-8030][docs] Add sqoop task doc

* Update docs/docs/zh/guide/task/sqoop.md

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* Update docs/docs/en/guide/task/sqoop.md

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* [Feature-8030][docs] Add sqoop task doc

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2022-11-13 00:09:05 +08:00
JieguangZhou
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2022-11-11 16:08:38 +08:00
Kerwin
019e7475d1
The new Chinese document latest version is 3.1.1 (#12869) 2022-11-11 16:06:20 +08:00
Kerwin
a887303d48
add 3.1.1 release doc (#12864) 2022-11-11 15:39:47 +08:00
kezhenxu94
454f1303a0
Add minio as demo storage for helm charts (#12861) 2022-11-11 10:26:57 +08:00
Jay Chung
6896fa9b47
[fix][doc] Dead link for python api (#12848)
* [fix][doc] Dead link for python api

ref: apache/dolphinscheduler#12779 and
apache/dolphinscheduler-sdk-python#1

* fix dead link
2022-11-10 14:49:11 +08:00
Kerwin
775b3f87b4
[doc] Add skip spotless check during ASF release. (#12835) 2022-11-10 14:26:24 +08:00
Jay Chung
1347a8f94c
[chore] Separate Python API into another repository (#12779)
Currently, our Python API code is a module in apache/dolphinscheduler codebase,
each time users change Python API code, they need to run all requests CI check
for dolphinscheduler and Python API, But if the user does only change Python
code, it could be merged if Python API CI pass and do not dependent on others CI.

Besides, we release Python API as the same version of dolphinscheduler. It is
easy for user to match Python API version. But when Python API does not change
any code, but dolphinscheduler release a bugfix version, Python API has to
release the new version to match dolphinscheduler. This happened when we
released Python API 2.0.6 and 2.0.7. 2.0.6 and 2.0.7 is bugfix version, and
Python API does not change any code, so the PyPI package is the same.

Separate Python API also makes our code more sense, we will have more
distinguished code in dolphinscheduler and Python API new repository.
Have separate issue tracker and changelog for information to users.

ref PR in other repository: apache/dolphinscheduler-sdk-python#1

see more detail in mail thread: https://lists.apache.org/thread/4z7l5l54c4d81smjlk1n8nq380p9f0oo
2022-11-09 21:20:15 +08:00
Tq
9bba4b105c
[fix][doc] Fix sql-hive and hive-cli doc (#12765) 2022-11-07 17:39:02 +08:00
Eric Gao
6d79414491
Add instructions on how to configure logging level for debugging purpose (#12699) 2022-11-07 10:30:56 +08:00
JieguangZhou
a47b4d0672
[Feature][Task] Transfer files between tasks (#12552)
* add data transfer between tasks

* add delete DATA_TRANSFER API

* convert Result to DeleteDataTransferResponse

* add api UT

* fix final

* fix doc
2022-11-03 16:33:39 +08:00
Tq
4e2701b57e
[doc][fix] Add new alert doc Slack (#12567) 2022-11-01 09:49:05 +08:00
amao
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init (#11759)
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2022-10-31 22:35:11 +08:00
rickchengx
3c31ddfd9f
[Improvement-12536][k8s] Support the command for the container in k8s task plugin (#12538) 2022-10-31 19:06:41 +08:00
Aaron Wang
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds (#12197)
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.

Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
Wenjun Ruan
e6da1ccf81
Add worker-group-refresh-interval in master config (#12601)
* Add worker-group-refresh-interval in master config

* Set interval cannot smaller than 10s

* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-10-31 09:37:26 +08:00
Yiming Guo
064fec88b0
[Feature-12040][docs] Update Granted Permissions Guide for PR#12048 (#12587)
* Upload new ui screenshots.

* Update document of Granted Permissions.

* Delete unused images.
2022-10-30 17:28:30 +08:00
Kerwin
207b4e77d3
Fix document styles. (#12600) 2022-10-29 11:31:19 +08:00
houshitao
4a13148bfc
[Bug] [API] Before deleting a worker group, check whether there is environment that reference the worker group. (#12534)
* [Bug] [API] Before deleting a worker group, check whether there is environment that reference the worker group.

Co-authored-by: houshitao <shitaohou@163.com>
2022-10-27 16:34:08 +08:00
JieguangZhou
21caec01db
fix doc CI (#12560) 2022-10-27 12:41:30 +08:00
Kerwin
63244c8065
Optimized complement documentation (#12550) 2022-10-27 09:31:12 +08:00
Wenjun Ruan
76a9e6d8a6
Add task executor threads full policy config in worker (#12510) 2022-10-26 10:29:22 +08:00
Jiajie Zhong
8add9744d4
[doc][feat] New release 2.0.7 (#12507) 2022-10-24 11:38:17 +08:00
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412)
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
simsicon
ba538067f2
[fix] Change the default unix shell executor from sh to bash (#12180)
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-10-18 12:57:37 +08:00
Yann Ann
1d0d26a416
[fix#12170] If the worker's clock early than master node, will ignore dispatch time. (#12219)
* add doc to guide user configure synchronized clocks.
2022-10-17 10:42:51 +08:00
Yann Ann
7aa8a77fb2
[Migrate][Test] Migrate other UT cases from jUnit 4 to 5, change @Ignore to @Disable 2022-10-16 12:49:11 +08:00
limaiwang
a725614429
[docs] change optional parameters --jar to --jars for Spark (#12385) (#12386) 2022-10-16 12:40:17 +08:00
Chris-Arith
b953cdc559
update english oracle.md (#12332) 2022-10-14 11:39:06 +08:00
Eric Gao
c50cc997a8
[Doc][Bug] Fix dead link caused by markdown cross-files anchor (#12357) 2022-10-14 10:57:11 +08:00
Jiajie Zhong
501119bbb6
[doc] Change release process base on new tool (#12324)
ref: #11478 we already have a new tool for releasing
2022-10-12 09:52:14 +08:00
Jiajie Zhong
a973054950
[ci] Milestone and Label are required before the PR merged (#12222)
To make our release process easier, I plan to mark milestone
and label is required before the PR merged, they use for

* label: In this path we required label with at least on with
  'feature|bug|improvement|document|chore' to make our changelog
  more convenient
* milestone: For easier to cherry-pick PR, currently some of
  our milestone only exists in issue and not exists in related
  PR, it may
2022-10-10 15:20:51 +08:00
tuchg
ef328d9cea
Update ZooKeeper minimum version requirements (#12284) (#12288) 2022-10-10 15:08:07 +08:00
Chris-Arith
c8bd106604
create oracle help.md (#12218)
* create oracle help.md

* Add files via upload

allow Oracle to choose schema

* move pic under img and add title for oracle.md
2022-10-10 13:53:20 +08:00
fuchanghai
9b1ee1516f
[doc] Correct descriptions in glossary.md (#12282) 2022-10-10 09:37:15 +08:00
Tq
c7435a51ac
[Feature][task-plugins] aws datasync task plugins (#11865)
* submit datasync task

* add aws software package license
2022-10-09 10:47:01 +08:00
TyrantLucifer
61457d1c2f
[Hotfix][Docs] Fix 404 dead links (#12261) 2022-10-08 19:43:35 +08:00
David
7bdbd62985
[doc] Update slack invitation link (#12258) 2022-10-08 11:59:53 +08:00
Yann Ann
7beff83e9f
[Doc] Add chmod command after unzip DS tar.gz (#12210) 2022-10-07 13:07:36 +08:00
Jiajie Zhong
6fc209ab9d
[fix][doc] Correct release export key step (#12228)
We should only add KEY to release/KEYS instead of
dev/KEYS, according to
https://lists.apache.org/thread/2gppxc04n0r87vq4qy4j3n52zwhqd442
2022-10-01 21:58:59 +08:00
caishunfeng
40090d459e
[Improvement] remove log-server and server module (#12206)
* remove log-server module

* remove dolphinscheduler-server module
2022-09-30 16:41:41 +08:00
Wenjun Ruan
aacabc6ec7
Remove workerGroup in registry (#12217) 2022-09-30 13:58:53 +08:00
caishunfeng
a055cf5791
[Release][Doc] add 3.1.0 release (#12227)
* [Release] add 3.1.0 release to doc
2022-09-30 12:14:29 +08:00
JieguangZhou
308f632fcc
[Feature][TASK-PLUGIN] AWS Database Migration Service (DMS) (#11868)
* add aws dms
2022-09-30 10:16:16 +08:00
Jiajie Zhong
d3f087e29b
[feat][ci] Add pre-commit for easy spotless config (#12192)
* Use pre-commit to handle spotless check
* Migrate exists python pre-commit-config into dir root one
* Change related document
2022-09-29 10:10:53 +08:00