Commit Graph

52 Commits

Author SHA1 Message Date
songjianet
f78847d47f
[Build][UI] Updated dependencies. (#10813)
* [Build][UI] Updated dependencies.

* [Build][UI] Updated dependencies.

* Fix E2E

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-07-07 18:18:48 +08:00
guodong
2c227ab384
[Feature-10363][server] S3 Resource center supports bucket customization (#10433)
* [Feature-10363][server] S3 Resource center supports bucket customization

* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils

* [Feature-10363][server] S3 Resource center supports bucket customization

* Modify code review comments
2022-06-18 10:20:01 +08:00
qianli2022
ff065d8e5b
[Feature][Improvement] Support multi cluster environments - k8s type (#10096)
* service code

* [Feature][UI] Add front-end for cluster manage

* fix e2e

* remove comment on cluster controller

* doc

* img

* setting e2e.yaml

* test

* rerun e2e

* fix bug from comment

* Update index.tsx

use Nspace instead of css.

* Update index.tsx

Remove the style.

* Delete index.module.scss

Remove the useless file.

Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>
2022-06-15 13:39:20 +08:00
PJ Fanning
f90f0f8490
Upgrade log4j in E2E (#10359) 2022-06-09 16:09:46 +08:00
xiangzihao
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup (#10373) 2022-06-08 19:42:05 +08:00
WangJPLeo
e57c2d1663
[Feature] Should not add refresh logic in the e2e case. (#10331)
* Should not add refresh logic in the e2e case.

* When the returned data from paging is empty, it should be an empty collection.

* resource query fix.

* e2e rerun

* timeZone ut fix

* udf resource check fix

* code deduplication

* rerun
2022-06-02 09:48:32 +08:00
WangJPLeo
e3e39cbdea
[Fix] Privilege Reconstruction Data Query Fix & Resource Creation Post Oper… (#10313)
* Privilege Reconstruction Data Query Fix & Resource Creation Post Operations.

* Resource Retry

* e2e retry

* create project log add

* project list query log add

* clean project log

* delete delay

* delete delay

* remove post handle

* project e2e

* browser refresh

* browser refresh

* e2e fix

* e2e browser refresh

* rowk flow e2e fix

* mapper deduplication

* udf e2e

* e2e
2022-06-01 20:29:10 +08:00
WangJPLeo
7ac8ea493c
[Feature][Permission] Project management module authority control refactoring, authority abstract interface setting. (#10261)
* Create process definition with task group and task group priority cannot be save into db

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* [Feature][API] Permission control of project

* The result of the assertion cannot be changed. The only difference is the judgment condition.

* test recovery.

* controller test fix.

* ProjectE2ETest fix.

* WebElement import.

* Prioritize problem solving.

* Judging the length of the set fix

Co-authored-by: houshitao <shitaohou@163.com>
Co-authored-by: hstdream <33045461+hstdream@users.noreply.github.com>
2022-05-29 16:14:16 +08:00
xiangzihao
718e786c24
Fix flaky filemanage_e2e_test (#10068) 2022-05-17 14:51:43 +08:00
Jiajie Zhong
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server (#9726)
* [common] Make dolphinscheduler_env.sh work

* Change dist tarball `dolphinscheduler_env.sh` location
  from `bin/` to `conf/`, which users could finish their
  change configuration operation in one single directory.
  and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
  when we start our sever instead of adding both
  `$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
songjianet
0d0674df1f
[Fix][UI Next][V1.0.0-Alpha] Fix the cannot update tenant individually on user info site. (#9237) 2022-03-28 17:55:15 +08:00
nobolity
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
labbomb
da2e85e9ec
[BUG][UI Next]fix the problem about Batch delete not working (#9080)
* fix the problem about Batch delete not working while click the border of the button

* try to fix e2e

Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
2022-03-24 14:43:02 +08:00
xiangzihao
197accc601
[Feature][e2e] Suggest e2e test adapt M1 chip (#9077) 2022-03-22 14:11:50 +08:00
xiangzihao
e0e7701341
[E2E] [CI] Fix flaky e2e test cause by #9036 (#9053) 2022-03-21 17:30:33 +08:00
Devosend
d91711b322
[Fix][UI Next][V1.0.0-Alpha] Menu error (#9036)
* modify the sidebar menu control mode

* modify side menu

* store last path

* fix security auth error

* fix resource e2e

* fix resource manage e2e bug
2022-03-21 15:19:27 +08:00
kezhenxu94
7cf79da5d9
Reduce the size of distribution tar (#9019) 2022-03-20 16:42:06 +08:00
xiangzihao
82394ba81d
[Feature-9021][E2E] Recover sub_process e2e test in ui-next (#9023) 2022-03-20 13:11:48 +08:00
xiangzihao
eefe7896c8
[Feature-8850][E2E] Restore project center e2e test cases in ui-next (#8909) 2022-03-15 22:07:30 +08:00
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
xiangzihao
2335453482
[Feature-8844][E2E] Restore datasource center e2e test cases in ui-next (#8845) 2022-03-13 12:33:19 +08:00
xiangzihao
49b1a073dd
[Feature-8823][E2E] Restore resource center e2e test cases in ui-next (#8840) 2022-03-12 20:49:46 +08:00
xiangzihao
20dd0c12ce
[E2E] Restore security center e2e test cases in ui-next (#8815) 2022-03-10 21:49:04 +08:00
kezhenxu94
d2a360480a
[CI][UI NEXT] Package new UI into release tar and remove old UI from it (#8670)
* [CI][UI NEXT] Package new UI into release tar and remove old UI from it

* Update e2e.yml
2022-03-07 11:20:24 +08:00
qianli2022
151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 (#8252)
* add db operator for namespace

* add k8s client

* add k8s controller

* add namespace ui

* add licenses

* add kubernetes model known-dependencies

* add new licenses

* fix SonarCloud Code Analysis bug

* remove license for standard protocol

* fix variable MemoryStr to memoryStr

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-02-15 10:21:15 +08:00
wind
fc9a31f813
[E2E] add switch task case (#8180)
* fix switch bug

* add switch e2e test

* add switch workflow

* pre task option

* selectPreTask id

* update

* switch task e2e

* naming optimization

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-10 10:00:50 +08:00
QuakeWang
fd38ef129b
[E2E] Add the update test for tenant (#8282) 2022-02-08 09:25:00 +08:00
xiangzihao
43bd640aad
[E2E] Add hive datasource e2e test and fix s3 docker image latest tag (#8276) 2022-02-06 14:22:38 +08:00
QuakeWang
63f7efa993
[E2E] Add new test of token management (#8161) 2022-01-30 12:34:50 +08:00
xiangzihao
4b58df9aa8
[E2E] add function manage test case (#8250) 2022-01-30 09:01:19 +08:00
xiangzihao
1ce5b5d82d
fix_flaky_worker_group_e2e_test (#8237) 2022-01-28 14:15:33 +08:00
xiangzihao
6094927dda
[E2E] Fix flaky udf manage e2e test (#8232)
* fix_file_manage_e2e_error

* fix error

* fix error

* fix error
2022-01-28 11:49:55 +08:00
wind
756ea1181e
[Improvement][E2E] use testcontainers host (#8224) 2022-01-27 19:15:05 +08:00
xiangzihao
d6690fa89c
[E2E] add udf manage test (#8203) 2022-01-27 16:17:32 +08:00
xiangzihao
7b22c16a85
[E2E]Fix file manage e2e error (#8214) 2022-01-27 14:59:22 +08:00
kezhenxu94
db953a229c
[E2E] Clean up the E2E with Java API to avoid platform sensitive (#8197) 2022-01-26 14:33:11 +08:00
xiangzihao
30b8b4a93e
[E2E][CI] Add renaming file management E2E tests (#8160) 2022-01-26 09:45:21 +08:00
xiangzihao
e7f480bdca
[E2E][CI] Add DataSource E2E test (#8085) 2022-01-19 17:29:02 +08:00
xiangzihao
b705a08794
[Feature-7993][E2E] Replace HDFS image with MinIO S3 image (#8066) 2022-01-17 19:32:47 +08:00
Jiajie Zhong
a061cf136c
[e2e] Add new test of security environment (#8048)
* [e2e] Add new test of security environment

ref: #7281

* Fix frontend bug

* Fix order of UT

* Fix delete and edit bottom class

* Try to fix

* Try to fix
2022-01-15 09:13:04 +08:00
xiangzihao
5be8e0531c
[E2E] [Resource] add file manage case (#7906) 2022-01-12 14:29:03 +08:00
wind
3edc58cbdd
[E2E] Add queue e2e test (#7941) 2022-01-12 10:31:17 +08:00
janeHe13
1ff339f37b
[E2E] Modify the tenant delete of the e2e case (#7921)
* Update UserE2ETest.java
2022-01-10 17:06:32 +08:00
janeHe13
0b38d0e826
[E2E]add worker group manage e2e case (#7879)
* add worker group manage e2e case

* add worker group manage e2e case

* Update dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/WorkerGroupE2ETest.java

Co-authored-by: janeHe13 <438044805@qq.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-01-07 21:24:03 +08:00
Jiajie Zhong
079adc9eca
[ci] Fix flaky test in e2e user (#7847)
* [ci] Fix flaky test in e2e user

fix: #7656

* Change class name

* Change get delete bottom function

* Correct function
2022-01-06 17:29:06 +08:00
kezhenxu94
269a62418f
Add E2E case for user manage (#7652) 2021-12-28 14:02:06 +08:00
kezhenxu94
03a71297dd
Unify the element selector rules for E2E (#7409)
* Unify the element selector rules for E2E

* Fix typo in docker swarm
2021-12-22 14:53:39 +08:00
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
kezhenxu94
37ba1eb5ad
Add E2E tests for some core features (#7025) 2021-11-30 13:07:55 +08:00
kezhenxu94
ef0cb46661
Fix shell tools for database schema cannot run (#7003) 2021-11-26 11:10:38 +08:00