Commit Graph

117 Commits

Author SHA1 Message Date
Jiajie Zhong
a86f4e2693
[fix][python] Task switch branch not show in webui (#12120)
when create task switch in python api, switch branch not show correctly,
due to we add some unnecessary attribute to switch task, this patch we
add `_task_ignore_attr` in class `Task` to ignore those unnecessary
attribute from `_task_default_attr`
2022-09-23 19:41:05 +08:00
chenrj
b1b57c8fb7
[feat][python] new resource plugin GitHub and gitlab (#11831)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: JieguangZhou <jieguang_zhou@163.com>
Co-authored-by: Devosend <devosend@gmail.com>
Co-authored-by: juzimao <578961953@qq.com>
2022-09-22 14:43:00 +08:00
Lyle Shaw
cc492c3e13
[Feature] Add CURD to the project/tenant/user section of the python-DS (#11162)
- Add CURD in project
- Add CURD in tenant
- Add CURD in user
- Add test in user

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-22 11:03:27 +08:00
rickchengx
08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
JieguangZhou
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use (#12071)
optimize code
2022-09-21 10:45:05 +08:00
JieguangZhou
ad683c3c42
[feat][python] Support MLflow task in python api (#11962) 2022-09-18 16:28:18 +08:00
JieguangZhou
bbc228e93f
[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
* add dvc task in pyds

* add BaseDVC class
2022-09-17 16:55:51 +08:00
JieguangZhou
e50a32d9cc
[feat][python] Support OpenMLDB task in python api (#11944)
Using the whole word zookeeper instead of short cut zk
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-17 15:24:17 +08:00
Eric Gao
a44817fc46
[Doc][Style] Fix doc format once for all (#12006) 2022-09-17 11:33:32 +08:00
JieguangZhou
e05f14ba23
[feat][python] Add sagemaker yaml example (#11925) 2022-09-17 10:34:06 +08:00
JieguangZhou
5202e5cfc6
[feat][python] Support Pytorch task in python api (#11975)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-16 20:24:38 +08:00
JieguangZhou
c41fa5a8b1
fix pydolphin yaml doc link (#11817) 2022-09-09 10:33:17 +08:00
chenrj
6d460a3ca2
[DSIP-13][python] New mechanism file plugins to Python API (#11360) 2022-09-07 10:02:37 +08:00
JieguangZhou
38ee91fb1e
[DSIP-11][python] create workflows from YAML configuration (#11611) 2022-09-06 09:28:16 +08:00
JieguangZhou
8988492c43
[Feat][python] Add parameter environment to task (#11763) 2022-09-06 09:26:30 +08:00
Jiajie Zhong
4dca488cd5
[python] Auto location when create process definition (#11681)
* Add auto format when submit process definition from python api
* add UPDATING.md

close: #8254
2022-08-27 23:38:46 +08:00
JieguangZhou
635fde1981
[bug][python] Fix task switch error when default branch not defined last (#11606)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-08-27 18:48:09 +08:00
陈家名
564a815d21
[fix][python] Sql pattern add truncate. (#11666) 2022-08-27 12:17:06 +08:00
Kerwin
c59a8503f4
update PyDolphinScheduler documentation link. (#11474) 2022-08-15 13:33:51 +08:00
Jiajie Zhong
4d427ee215
[python] Add support interpreter version 310 and 311-dev (#11170)
* Change version of package
* Fix tox error
* Change dev status to beta
* py4j not work on py{10,11} OS Windows
2022-08-11 23:39:44 +08:00
Jiajie Zhong
26afd655cb
[python] Add multiple version of document (#11391)
* Using sphinx-multiversion to support multiple version
of python api document
* change python api sidebar link
* fix whitelist_externals deprecated and add git

close: #8227
2022-08-11 12:02:29 +08:00
陈家名
26a21566bf
[python] Add local dev mod runing python integate tests (#11362) 2022-08-10 15:04:56 +08:00
xiangzihao
983f9b41bf
[Doc] Add tools/libs to the init database steps (#11345) 2022-08-08 21:53:23 +08:00
陈家名
3061bbc5c4
[Python] Combine gateway.entry_point call in python api side (#11330) 2022-08-08 16:25:27 +08:00
陈家名
aef2fbf36c
[Python] Migrate exists method call in PythonGateway to corresponding service (#11188)
Migrate all exists method call in PythonGateway to corresponding service
2022-08-04 12:50:21 +08:00
Jiajie Zhong
9e692f919a
[python] Add directly install without clone source code (#11179)
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-07-28 21:40:20 +08:00
Jiajie Zhong
7a766cbcf2
[python] Refactor structure to avoid cycle import (#11167)
Currently, our core and side module dependent on each others. and will cause
cycle import in our codebase, especially in issue #10905, we try to refactor
to solve this problem.

This patch do the following change:

* Rename module `side` to `models`
* Move `core/base` and `core/sidebase` to dir `modules`
* Move `configuration` and `default_config.yaml` to the root of pydolphinscheduler
2022-07-28 19:02:13 +08:00
JieguangZhou
764a24384f
[python] add import SageMaker task (#11121) 2022-07-25 15:46:36 +08:00
JieguangZhou
952aee80e7
[fix] py sagemaker doc (#11109) 2022-07-22 22:57:50 +08:00
JieguangZhou
cc088e0f54
[python] Support SageMaker task type (#11002) 2022-07-22 10:47:08 +08:00
Jiajie Zhong
81930e5420
[python] Fix tasks with multiple upstream and workflow query error (#10941)
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed
2022-07-14 22:06:07 +08:00
陈家名
59cd86157f
[Python] Supports creating or editing resources. (#10823) 2022-07-12 20:44:59 +08:00
Jiajie Zhong
113a67122f
[python] fix doc build error (#10810) 2022-07-06 22:02:08 +08:00
Jiajie Zhong
b4eee7edb6
[python] Correct some docs for python api (#10772)
* Remove add leading `$` from command
* Add copy button from code block
* Using `python -m pip` to install package avoid error
2022-07-05 18:34:31 +08:00
陈家名
7970573365
[python] Support using full-name definitions for resources (#10551) 2022-06-23 22:59:59 +08:00
陈家名
f512e3c31f
[fix][python] Correct variable type of SQL type (#10464) 2022-06-16 09:39:53 +08:00
陈家名
b86dc53ad1
[fix][Python] Support same task name in project defferent process definition (#10428)
close: #10431
2022-06-14 13:53:18 +08:00
Jiajie Zhong
56e0ea802d
[python] Fix process definition attr release state not work (#10151)
This patch fix the error release state not work when it set to
offline and submit it to Java gateway, it error because we do
not pass the attribute to Java gateway function
`createOrUpdateProcessDefinition`

close: #9779
2022-05-24 10:25:19 +08:00
Jiajie Zhong
bf4b8ff0d9
[python] Change distribute name avoid confusion with main pkg (#10134)
rename them and add `python` suffix in the end
close: #9121
2022-05-19 20:27:38 +08:00
陈家名
a2cd2a99e6
[python] Fix pre_submit_check error when local params in tasks (#10032) 2022-05-16 19:51:41 +08:00
陈家名
828034f8b9
[improve][python] Specify warning group when deploy process-definition (#9773) 2022-04-27 22:02:14 +08:00
陈家名
8a8b63cd96
[Improve][python] Support create table syntax and custom sql type param (#9673) 2022-04-25 10:17:20 +08:00
Kerwin
9d11be447a
[Python] Make detached signature during release (#9607) 2022-04-20 13:07:27 +08:00
Jiajie Zhong
1f48601c75
[python] Add task decorator for python function (#9496)
* [python] Add task decorator for python function

* Add decorator `@task`
* Add a tutorial about it
* Change tutorial doc and combine into traditional docs
  * Add sphinx-inline-tab for better view

* revert not need change

* Correct python function indent

* Correct integration test
2022-04-15 15:50:52 +08:00
Jiajie Zhong
59a026d897
[python] Support read config in env variable (#9517)
Add a new method to get config from environment variables
and for now, we have three ways to get config and the
priority is `env-var > custom-config-file > built-in-config-file`.

Environment config setting does not work in CLI, because it will
confuse users when they get config value is `var-env` but value
in the configuration file is `var-in-file`, they may not find
the way how to change it

* Add documentation
* Add it to UPDATING.md

close: #8344
2022-04-15 15:46:44 +08:00
Jiajie Zhong
3d78859fe5
[python] Recover python release properties (#9444)
This patch recovers the properties `python.sign.skip=false`
when the combined profile `release,python` is used.

also close: #9433
2022-04-12 09:42:59 +08:00
Jiajie Zhong
aaf2042ec4
[python] Add missing doc about config and connect remote server (#9443)
which includes `configuration`, `run example`
`how to connect remote server`

close: #9286, #9284, #8917
2022-04-12 09:42:28 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
Jiajie Zhong
3457cee960
[python] Migrate pythonGatewayServer into api server (#9372)
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server

The distribute package size before and after this patch is:

```sh
# before
796M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz

# after
647M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
2022-04-07 14:41:15 +08:00
Eric Gao
4c1578a374
[Feature-8369][Document]Add docs building test instructions in pydolphinscheduler DEVELOP.md (#8369) (#8693) 2022-03-29 10:56:53 +08:00