Commit Graph

139 Commits

Author SHA1 Message Date
Jiajie Zhong
9281f2b650 [ci] Make cluster test work on release branch (#11199)
When I try to release the 3.0.0 on branch `3.0.0-prepare`,
I find out cluster-test error due to version change of
tarball, I think we should make cluster test work on both
dev and prepare release branch, so I add this patch

(cherry picked from commit 93a73b1292)
2022-08-02 18:52:29 +08:00
Jiajie Zhong
e4ab488426 [python] Fix tasks with multiple upstream and workflow query error (#10941)
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed

(cherry picked from commit 81930e5420)
2022-07-28 11:13:48 +08:00
Jiajie Zhong
b2ad6f3f8e [python] Integrate test run strategy change (#10825)
this patch change python integrate test strategy,
before that, only content change in dir
dolphinscheduler-python will run the test. And it
will fail our python api startter when some code
change in java gateway. So we should start the
integrate test except docs change.

and in the future, we may add python ci to
.asf.yaml to make sure it pass

close: #10488
(cherry picked from commit 598290487c)
2022-07-28 11:13:48 +08:00
Wenjun Ruan
1e0939d4a2 Add JDK11 on unit-test (#10364) 2022-07-18 16:32:19 +08:00
旺阳
3de7b65f9a [doc] Delete SPRING_DATASOURCE_DRIVER_CLASS_NAME in doc (#10228)
(cherry picked from commit 87d9ec1dee)
2022-06-16 17:02:16 +08:00
xiangzihao
43e1e5567c add_alert_server_cluster_test (#10125)
(cherry picked from commit 8fab224e0d)
2022-05-25 23:42:02 +08:00
xiangzihao
b2058f9532 [Feature-10034][CI] Add postgresql cluster test in CI (#10035)
* add postgresql cluster test

* add postgresql cluster test

* add postgresql cluster test

* add postgresql cluster test

(cherry picked from commit 2423fa50f7)
2022-05-25 23:41:13 +08:00
xiangzihao
944be34177 [Feature-9474] [CI] Add cluster test script verify on shell script (#9997)
* cluster test

* fix init db failed

* fix init db failed

* fix init db failed

* fix init db failed

* fix init db failed

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add sudo

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* add github actions

* remove cluster-test workflows

* add github actions

* add github actions

* refactor test to docker mode

* refactor test to docker mode

* refactor test to docker mode

* refactor test to docker mode

* remove create schema logic

* remove create schema logic

* remove create schema logic

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* fix runnint cluster test error

* add github actions

* add github actions

* add github actions

* add cluster test timeout

* add cluster start test loop check

* add cluster start test loop check

* optimize cluster start test loop check

(cherry picked from commit baf654cb47)
2022-05-24 23:01:23 +08:00
旺阳
f46a4d7dd8 [Fix][Docker] Support Multi-Architecture Docker Images (#10052)
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
(cherry picked from commit 537c206c2d)
2022-05-23 14:43:26 +08:00
songjianet
7a7807e520
[Feature] Remove old ui. (#9909)
* [Feature] Remove old ui.

* [Docs] Add license file for screenfull.
2022-05-06 14:04:35 +08:00
Jiajie Zhong
d188f49b0e
[doc] Dir development should not ref other dirs content (#9812)
For now, development dir deploy in other place instead
of docs dir https://dolphinscheduler.apache.org/en-us/development/development-environment-setup.html
so could not ref markdown doc from other docs dir
2022-04-27 20:02:34 +08:00
xiangzihao
d18cf184c6
[CI] Improve CI (#9436)
* test

* test

* test

* test

* test

* test

* test

* test

* test

* improve ci

* improve ci

* improve ci
2022-04-12 14:43:15 +08:00
kezhenxu94
14d71d1462
[UI] Migrate NPM to PNPM in CI builds (#9431) 2022-04-11 14:16:23 +08:00
Jiajie Zhong
c828809b46
[doc] Change get help from dev mail list to slack (#9377)
* Change all get help from dev mailing list to slack, because
  we find out mailing list have many users ask for subscribe
  and they maybe subscribe by accident.
* remove join dev mailing list in faq.md because we already
  have it in https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
2022-04-08 10:56:35 +08:00
xiangzihao
80ea8049e0
[CI] try to fix ci (#9366)
* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci

* try to fix ci
2022-04-06 21:57:52 +08:00
xiangzihao
40b73f7962
[Improve][CI] improve ci checking (#9325) 2022-04-06 09:33:41 +08:00
Jiajie Zhong
9f84dbbda0
[doc] Migrate dev doc from website repository (#9291)
* [doc] Migrate dev doc from website repository

* Correct release language

* Fix license issue

* Add all images from website

* Delete unused images

* Add CI

* Fix ci

* fix ci

* Remove no need file

* Add latest commit from website

* correct img_utils.py script

* Remove unused imgs
2022-04-02 10:43:34 +08:00
Jiajie Zhong
30746d9762
[python] Add integrated test to python gateway server (#8966)
* [python] Add integrated test to python gateway server

* Build java code and create standalone server image in GA
* Add component start docker in python
* Run example to make sure it work to it

close: #8035

* Fix build docker image working directory

* Fix working directory
2022-03-18 16:24:40 +08:00
xiangzihao
eefe7896c8
[Feature-8850][E2E] Restore project center e2e test cases in ui-next (#8909) 2022-03-15 22:07:30 +08:00
kezhenxu94
14bb722f6f
[CI] Require E2E to passed before merging (#8891) 2022-03-15 14:35:21 +08:00
Jiajie Zhong
b04a535744
[ci] Add timeout to each job avoiding zombie process (#8886) 2022-03-14 22:34:44 +08:00
kezhenxu94
d3cbf8eeaf
[UI-NEXT] Fix wrong packaging locations (#8851) 2022-03-14 15:26:15 +08:00
xiangzihao
2335453482
[Feature-8844][E2E] Restore datasource center e2e test cases in ui-next (#8845) 2022-03-13 12:33:19 +08:00
Jiajie Zhong
90f9aed11d
[python] Add tox for out of box test (#8837)
* [python] Add tox for out of box test

* Add `tox` for easy and complete test
* Correct package data attribute
* Add more dirs in flake8 ignore

* Change GA from raw to tox

* Fix env setter

* Temp skip sanity

* Rm not exists job

* Add code test

* fix

* rm all need

* fix

* fix

* fix

* fix

* fix

* fix

* Migrate pip upgrede to tox setting

* Change install_commands to commands

* fix config of install command

* Add GA env to do that

* Fix env error

* Finial commit

* remove duplicate

* Change doc

* Change dependence of ci

* Change dependence of ci
2022-03-12 21:27:36 +08:00
xiangzihao
49b1a073dd
[Feature-8823][E2E] Restore resource center e2e test cases in ui-next (#8840) 2022-03-12 20:49:46 +08:00
xiangzihao
20dd0c12ce
[E2E] Restore security center e2e test cases in ui-next (#8815) 2022-03-10 21:49:04 +08:00
kezhenxu94
d2a360480a
[CI][UI NEXT] Package new UI into release tar and remove old UI from it (#8670)
* [CI][UI NEXT] Package new UI into release tar and remove old UI from it

* Update e2e.yml
2022-03-07 11:20:24 +08:00
kezhenxu94
60a00490c6
[CI][UI Next][V1.0.0-Alpha] Add frontend UI to CI checks and fix the linting problems found (#8666) 2022-03-02 20:01:13 +08:00
wind
fc9a31f813
[E2E] add switch task case (#8180)
* fix switch bug

* add switch e2e test

* add switch workflow

* pre task option

* selectPreTask id

* update

* switch task e2e

* naming optimization

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-10 10:00:50 +08:00
xiangzihao
43bd640aad
[E2E] Add hive datasource e2e test and fix s3 docker image latest tag (#8276) 2022-02-06 14:22:38 +08:00
QuakeWang
63f7efa993
[E2E] Add new test of token management (#8161) 2022-01-30 12:34:50 +08:00
xiangzihao
4b58df9aa8
[E2E] add function manage test case (#8250) 2022-01-30 09:01:19 +08:00
Jiajie Zhong
f3d663a7ea
[python] Clean deps and prepare release (#8210)
* Change package name
* Migrate requirement*.txt to setup.py
* Add extra required for dev
* Add doc RELEASE and DEVELOP
* Correct description
2022-01-28 10:12:00 +08:00
xiangzihao
d6690fa89c
[E2E] add udf manage test (#8203) 2022-01-27 16:17:32 +08:00
Jiajie Zhong
35578efd08
[python] Initiate document for pydolphinscheduler (#8005) 2022-01-26 12:37:57 +08:00
xiangzihao
e7f480bdca
[E2E][CI] Add DataSource E2E test (#8085) 2022-01-19 17:29:02 +08:00
Jiajie Zhong
a061cf136c
[e2e] Add new test of security environment (#8048)
* [e2e] Add new test of security environment

ref: #7281

* Fix frontend bug

* Fix order of UT

* Fix delete and edit bottom class

* Try to fix

* Try to fix
2022-01-15 09:13:04 +08:00
xiangzihao
1c0ad53a35
[CI][E2E] Optimize CI E2E test performance (#7974) 2022-01-14 10:31:37 +08:00
xiangzihao
5be8e0531c
[E2E] [Resource] add file manage case (#7906) 2022-01-12 14:29:03 +08:00
wind
3edc58cbdd
[E2E] Add queue e2e test (#7941) 2022-01-12 10:31:17 +08:00
janeHe13
0b38d0e826
[E2E]add worker group manage e2e case (#7879)
* add worker group manage e2e case

* add worker group manage e2e case

* Update dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/WorkerGroupE2ETest.java

Co-authored-by: janeHe13 <438044805@qq.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-01-07 21:24:03 +08:00
Jiajie Zhong
ab89e4335e
[ci] Remove workflow paths-ignore attribute (#7807)
It will cause our CI always waitting due to
we requests E2E contexts in .asf.yaml but
we set some ignore path on workflow e2e.
2022-01-05 14:24:12 +08:00
kezhenxu94
269a62418f
Add E2E case for user manage (#7652) 2021-12-28 14:02:06 +08:00
kezhenxu94
120c1755a1
Required E2E to pass before PR can be merged (#7612) 2021-12-24 17:02:42 +08:00
kezhenxu94
b350dc70c9
Build and publish snapshot Docker images for testing (#7454) 2021-12-17 00:26:09 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
kezhenxu94
37ba1eb5ad
Add E2E tests for some core features (#7025) 2021-11-30 13:07:55 +08:00
Jiajie Zhong
7c5c7ec907
[ci][python] Add isort to sort out import (#6871)
* Add isort config file to fix conflict with black
* Add some doc about isort
2021-11-17 11:48:52 +08:00
Jiajie Zhong
54933b33e3
[ci][python] Add coverage check in CI (#6861)
* [ci] Add coverage check in CI

* Coverage add dependent

* Install pydolphinscheduler before run coverage

* Up test coverage to 87% and down threshold to 85%

* Fix code style

* Add doc about coverage
2021-11-17 09:46:40 +08:00
Jiajie Zhong
c110488269
[ci] Remove GA path rule in build and unittest (#6797)
CI could not be run due to we add 9113817b to asf
configure and same file change will not trigger
build or unit test github actions run. This patch
try to skip github action path ignore rule temporarily

Maybe we should find more effective way to spearate py
, js, java CI later
2021-11-11 17:32:57 +08:00