Commit Graph

100 Commits

Author SHA1 Message Date
Jiajie Zhong
82deeab2fb [python] Add multiple version of document (#11391)
* Using sphinx-multiversion to support multiple version
of python api document
* change python api sidebar link
* fix whitelist_externals deprecated and add git

close: #8227

(cherry picked from commit 26afd655cb)
2022-09-17 21:54:29 +08:00
xiangzihao
92ca3df127 [Doc] Add tools/libs to the init database steps (#11345)
(cherry picked from commit 983f9b41bf)
2022-08-09 13:55:30 +08:00
Jiajie Zhong
889fd580e3 [maven-release-plugin] prepare for next development iteration 2022-08-05 18:16:57 +08:00
Jiajie Zhong
9efd1ace78 [maven-release-plugin] prepare release 3.0.0 2022-08-05 18:16:57 +08:00
Jiajie Zhong
a6db4a591c Some docs version change 2022-08-05 16:03:18 +08:00
Jiajie Zhong
ab73121401 [python] Add directly install without clone source code (#11179)
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

(cherry picked from commit 9e692f919a)
2022-08-02 18:51:46 +08:00
Jiajie Zhong
88712b423a [fix] Some cherry pick error 2022-07-28 11:13:48 +08:00
陈家名
4da1240b0e [Python] Supports creating or editing resources. (#10823)
(cherry picked from commit 59cd86157f)
2022-07-28 11:13:48 +08:00
Jiajie Zhong
e4ab488426 [python] Fix tasks with multiple upstream and workflow query error (#10941)
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed

(cherry picked from commit 81930e5420)
2022-07-28 11:13:48 +08:00
Jiajie Zhong
6bab63cc7b [python] fix doc build error (#10810)
(cherry picked from commit 113a67122f)
2022-07-27 22:36:34 +08:00
Jiajie Zhong
6f5ea6e411 [python] Correct some docs for python api (#10772)
* Remove add leading `$` from command
* Add copy button from code block
* Using `python -m pip` to install package avoid error

(cherry picked from commit b4eee7edb6)
2022-07-27 22:34:40 +08:00
陈家名
dd4b99298a [python] Support using full-name definitions for resources (#10551)
(cherry picked from commit 7970573365)
2022-07-27 22:13:04 +08:00
devosend
7ddaa2f47d [maven-release-plugin] prepare for next development iteration 2022-06-28 16:41:31 +08:00
devosend
0a1b9bdd52 [maven-release-plugin] prepare release 3.0.0-beta-2 2022-06-28 16:41:30 +08:00
devosend
d68dcda2bb [chore] pre-release change pom.xml 2022-06-28 08:24:21 +08:00
devosend
331f68e5b8 [chore] pre-release change 2022-06-27 15:24:08 +08:00
陈家名
6edb3412a9 [fix][python] Correct variable type of SQL type (#10464)
(cherry picked from commit f512e3c31f)
2022-06-18 21:51:25 +08:00
陈家名
476f2395dc [fix][Python] Support same task name in project defferent process definition (#10428)
close: #10431
(cherry picked from commit b86dc53ad1)
2022-06-18 21:47:11 +08:00
Jiajie Zhong
57ade38939 [maven-release-plugin] prepare release 3.0.0-beta-1 2022-05-26 20:26:32 +08:00
Jiajie Zhong
80ebe4a334 [python] Fix process definition attr release state not work (#10151)
This patch fix the error release state not work when it set to
offline and submit it to Java gateway, it error because we do
not pass the attribute to Java gateway function
`createOrUpdateProcessDefinition`

close: #9779
(cherry picked from commit 56e0ea802d)
2022-05-24 22:57:28 +08:00
Jiajie Zhong
bb47ced474 [python] Change distribute name avoid confusion with main pkg (#10134)
rename them and add `python` suffix in the end
close: #9121

(cherry picked from commit bf4b8ff0d9)
2022-05-23 14:51:34 +08:00
陈家名
1b4036e2ce [python] Fix pre_submit_check error when local params in tasks (#10032)
(cherry picked from commit a2cd2a99e6)
2022-05-16 20:17:44 +08:00
Jiajie Zhong
a9fa6b33a4
[chore] Change release version to 3.0.0-beta-1 (#9957) 2022-05-12 21:49:21 +08:00
陈家名
828034f8b9
[improve][python] Specify warning group when deploy process-definition (#9773) 2022-04-27 22:02:14 +08:00
陈家名
8a8b63cd96
[Improve][python] Support create table syntax and custom sql type param (#9673) 2022-04-25 10:17:20 +08:00
Kerwin
9d11be447a
[Python] Make detached signature during release (#9607) 2022-04-20 13:07:27 +08:00
Jiajie Zhong
1f48601c75
[python] Add task decorator for python function (#9496)
* [python] Add task decorator for python function

* Add decorator `@task`
* Add a tutorial about it
* Change tutorial doc and combine into traditional docs
  * Add sphinx-inline-tab for better view

* revert not need change

* Correct python function indent

* Correct integration test
2022-04-15 15:50:52 +08:00
Jiajie Zhong
59a026d897
[python] Support read config in env variable (#9517)
Add a new method to get config from environment variables
and for now, we have three ways to get config and the
priority is `env-var > custom-config-file > built-in-config-file`.

Environment config setting does not work in CLI, because it will
confuse users when they get config value is `var-env` but value
in the configuration file is `var-in-file`, they may not find
the way how to change it

* Add documentation
* Add it to UPDATING.md

close: #8344
2022-04-15 15:46:44 +08:00
Jiajie Zhong
3d78859fe5
[python] Recover python release properties (#9444)
This patch recovers the properties `python.sign.skip=false`
when the combined profile `release,python` is used.

also close: #9433
2022-04-12 09:42:59 +08:00
Jiajie Zhong
aaf2042ec4
[python] Add missing doc about config and connect remote server (#9443)
which includes `configuration`, `run example`
`how to connect remote server`

close: #9286, #9284, #8917
2022-04-12 09:42:28 +08:00
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
Jiajie Zhong
3457cee960
[python] Migrate pythonGatewayServer into api server (#9372)
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server

The distribute package size before and after this patch is:

```sh
# before
796M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz

# after
647M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
2022-04-07 14:41:15 +08:00
Eric Gao
4c1578a374
[Feature-8369][Document]Add docs building test instructions in pydolphinscheduler DEVELOP.md (#8369) (#8693) 2022-03-29 10:56:53 +08:00
caishunfeng
263791a63e
[Bug-9137][Server] fix timezone init (#9219)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
Jiajie Zhong
5289b09817
[python] Fix change exists pd attribute user error (#9140)
* [python] Fix change exists pd attribute user error

* Remove attribute from class ProcessDefinition, cause it should to
  object User instead of object ProcessDefinition.
* Grant project to user if attribute user change for exists ProcessDefinition

close: #8751

* Add py.test conftest.py for package integration
2022-03-24 15:24:02 +08:00
Tq
3c377ff54a
[Bug-9094][API] fix default time-zone conf (#9095)
* fix the API server default time-zone conf

* fix the Python server default time-zone conf
2022-03-22 19:38:46 +08:00
Jiajie Zhong
fd5e79bd80
[python] Separate py from main dist and auto add gpg sign (#8999)
* Add new profile `python` to separate build python
  package from `release`. Now someone who does not interested
  in python api could build package without python environment
* Add auto gpg asc sign when run `mvn instal -Ppython`
* Add property to skip gpg sign `python.sign.skip`
* Set `python.sign.skip=false` sign python dist by default

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-03-19 22:07:24 +08:00
Jiajie Zhong
30746d9762
[python] Add integrated test to python gateway server (#8966)
* [python] Add integrated test to python gateway server

* Build java code and create standalone server image in GA
* Add component start docker in python
* Run example to make sure it work to it

close: #8035

* Fix build docker image working directory

* Fix working directory
2022-03-18 16:24:40 +08:00
Jiajie Zhong
e53ac4e304
[python] Add autoflake to auto rm unused import (#8897)
* Add a new commit hook to pre-commit, to auto
  remove unused imports
* Remove all unused pass statements
* Add autoflake to tox and out CI

close: #8592
2022-03-16 09:33:26 +08:00
JieguangZhou
32a5ccac72
[python] Add custom log for meanful an easier log formatter (#8901)
* Change logging.warning to logger.warning
* Add testing for logging about task code already in process definition

close: #8258
2022-03-15 19:16:33 +08:00
天仇
a308fa3bef
[Feature-8591][Python] Add pre-commit to run basic test before commit (#8843)
* add `.pre-commit-config.yaml` in the pydolphinscheduler module and configured basic check rules like `isort`, `black`, `flake8`
* add some instructions in `DEVELOP.MD`
* Fix some mistake and add some docs on it

Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
2022-03-14 20:51:55 +08:00
Jiajie Zhong
90f9aed11d
[python] Add tox for out of box test (#8837)
* [python] Add tox for out of box test

* Add `tox` for easy and complete test
* Correct package data attribute
* Add more dirs in flake8 ignore

* Change GA from raw to tox

* Fix env setter

* Temp skip sanity

* Rm not exists job

* Add code test

* fix

* rm all need

* fix

* fix

* fix

* fix

* fix

* fix

* Migrate pip upgrede to tox setting

* Change install_commands to commands

* fix config of install command

* Add GA env to do that

* Fix env error

* Finial commit

* remove duplicate

* Change doc

* Change dependence of ci

* Change dependence of ci
2022-03-12 21:27:36 +08:00
Jiajie Zhong
fb772c328d
[python] Change name to_string to __str__ and add __repr__ (#8830)
* Change `yaml_parser.py` method `to_string` to magic method
  `__str__` make it more pythonic.
* Add some tests to magic method `__str__` and `__repr__`
2022-03-12 20:36:51 +08:00
Jiajie Zhong
62e12c73dc
[python] Add LICENSE and NOTICE to python dist package (#8768)
* Add LICENSE and NOTICE to Python API
* Add new license dir `python-api-license` to keep all
  Python API's LICENSE files
* Change LICENSE in distribute package
* Add content about should add LICENSE when new dependence
  adding
2022-03-09 19:50:43 +08:00
Jiajie Zhong
5c640789c3
[python] refactor yaml file parser (#8701)
* [python] refactor yaml file parser

* Change yaml parser package to ruamel.yaml
* Refactor configuration.py module
* And file.py to write file locally
* Add more tests on it

close: #8593

* Fix UT error

* Remove pypyaml from tests

* Fix file error when param create is False

* Fix error logic
* And tests to avoid regression
2022-03-07 11:42:28 +08:00
Jiajie Zhong
aac960d3d2
[python] Increase UT coverage threshold to 0.9 (#8660)
This patch increase Python API unit test coverage
threshold from 85% to 90%. If coverage under 90%
will failed our the test
2022-03-03 12:20:34 +08:00
Jiajie Zhong
f256f3ec6e
[python] Fix source and issue tracker link in PyPI (#8638)
* [python] Fix source and issue tracker link in PyPI

This patch fix error or wrong link in
https://pypi.org/project/apache-dolphinscheduler/

* Fix style
2022-03-02 10:44:18 +08:00
Jiajie Zhong
ec4ce2b573
[python] Add config mechanism and cli subcommand config (#8585)
* [python] Add config mechanism and cli subcommand config

* Add configuration.py mechanism for more easy change config
  and move some configs to it. It mechanism including
  configuration.py module and default_config.yaml file
* Add `config` for cli subcommand allow users initialize, get,
  set configs

close: #8344

* Change setup.py format
2022-03-01 11:16:44 +08:00
Jiajie Zhong
36964c2b5a
[deploy] Make Python API as independent directory in release's dist (#8470)
This patch add python api package as independent
directory in final dist directory when user run
command `mvn -U install package -Prelease`.

There have `tar.gz` and `whl` in the python dist directory

```
<the-old-dist-package>
python
  |--- python-independent-pkg.tar.gz
  |--- python-independent-pkg.whl
```

close: #8343

* Give dedicated control to docker build/push goals

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-02-25 11:06:26 +08:00
xiangzihao
891a002bea
[Feature-#8373][MasterServer] Dependent tasks can re-run automatically in the case of complement (#8496)
* first add feature_8373

* fix code smell

* add blank line

* fix some problems

* fix unit test error
2022-02-25 10:34:02 +08:00