currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify code review comments
* [Fix] Fix the JSONUtils tool class time zone problem #10282
* [Fix] Fix the JSONUtils tool class time zone problem #10282
* Update JSONUtils.java
remove unnessnary log
* Update JSONUtilsTest.java
add timezone import
Co-authored-by: xiangzihao <460888207@qq.com>
* Add a status code for password verification length to improve user interaction
* Modify the length and null value, verify them separately and return
* Modify the length and null value, verify them separately and return
* remove the outer not empty validate and Modify judgment conditions
* add the outer not empty validate and remove inter empty
Co-authored-by: syyang <730122+syyang99@user.noreply.gitee.com>
* Added disk available capacity monitoring
* resolve conflict
Added disk available capacity monitoring
* resolve conflict
Added disk available capacity monitoring
* resolve conflict
Added disk available capacity monitoring
* Resource userName is null
* [Fix-10082]Resource userName is null and udf function manager add a username
* [Fix-10082]Resource userName is null and udf function manager add a username
* [Fix-10082]Resource userName is null and udf function manager add a username
* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.
* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.
* [Feature][API] Modify the permissions of project management, security center, data source center and data quality module.
Co-authored-by: houshitao <shitaohou@163.com>
* Create process definition with task group and task group priority cannot be save into db
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* [Feature][API] Permission control of project
* The result of the assertion cannot be changed. The only difference is the judgment condition.
* test recovery.
* controller test fix.
* ProjectE2ETest fix.
* WebElement import.
* Prioritize problem solving.
* Judging the length of the set fix
Co-authored-by: houshitao <shitaohou@163.com>
Co-authored-by: hstdream <33045461+hstdream@users.noreply.github.com>
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.
At present, considering that it is a distributed user, it should not be allowed to create users in linux
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Add test method
The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.
At present, considering that it is a distributed user, it should not be allowed to create users in linux
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Add parameter description to configuration.md
Add test method
The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.
At present, considering that it is a distributed user, it should not be allowed to create users in linux
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Add parameter description to configuration.md
Add test method
The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.
At present, considering that it is a distributed user, it should not be allowed to create users in linux
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
* [Fix-10181] Fix the logic of judging that the tenant does not exist
Add parameter description to configuration.md
Add test method
The configuration item adds 'tenant-distributed-user' in worker application.yaml to make it suitable for distributed users. If it is false, the original logic remains unchanged.
At present, considering that it is a distributed user, it should not be allowed to create users in linux
Use the linux command as id to get the user information that exists in /etc/passwd file and the cached sssd user.
for example:
id test
1. exist in /etc/passwd file or ldap : uid=1030(test) gid=1030(test) groups=1030(test)
2. no exist in /etc/passwd file and ldap: id: test: no such user
Temporarily unable to test the system for windows and mac
Co-authored-by: ouyangl <ouyangl@tebon.com.cn>
* [fix] Enhance name pre checker in resource center
Add file name and directory checker to avoid directory traversal
* add some missing change and change docs
* change var name in directoryTraversal
* Fix ci
* [Fix-10080]When the created tenant name is too long, an error message will be prompted
* [Fix-10080]When the created tenant name is too long, an error message will be prompted
* [Fix-10080]When the created tenant name is too long, an error message will be prompted
Co-authored-by: houshitao <shitaohou@163.com>
* [fix-9856]Failed to upload the file because the full name was too long, but get misleading message
* Update Status.java
Co-authored-by: litiliu <litiliu@cisco.com>
Co-authored-by: lidongdai <dailidong66@gmail.com>
* [Feature][UI Next] Add front-end for Apache Zeppelin task plugin (#9597)
* Fix minor bugs of zeppelin server endpoint in common.properties
* Fix several bugs in zeppelin task plugin front-end
* add noteId and paragraphId into ITaskParams
* Fix bug in tasks/use-zeppelin.ts
* [common] Make dolphinscheduler_env.sh work
* Change dist tarball `dolphinscheduler_env.sh` location
from `bin/` to `conf/`, which users could finish their
change configuration operation in one single directory.
and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
when we start our sever instead of adding both
`$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
* The resource download method is selected according to the configuration and the service startup verification is added.
* common check CI fix
* Startup check changed to running check
* code smell
* Coordinate resources to increase test coverage.
* Split resource download method.
* Unit Test Coverage
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
* k8s auth
* remove log
* fix test
* use constants
* use constants K8S_LOCAL_TEST_CLUSTER
* simple auth get
* change test
* add namespace authorize in user page
* prettier code
* change test data
Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
* add new properties to Alert.java and do minor changes to comments
* fix Integer to int
* fix Integer to int
* fix sql files
* fix not null properties to default null
* rebase dev
* change task state to dispatch if dispatch success
* update task host when dispatch
* add dispatch task event
* test
* check tenant after enable auto create
* handle dispatch state
Co-authored-by: caishunfeng <534328519@qq.com>
* feat(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config
* fix(resource manager): update the dependency
* fix(resource manager): extend s3 to the storage of ds
fix the constant of hadooputils
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager):
merge the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl
* fix(resource manager): extend s3 to the storage of ds
merge the resourceServiceTest
* fix(resource manager): test cancel the test method
createTenant verifyTenant
* fix(resource manager): merge the code follow the check-result of sonar
* fix(resource manager): extend s3 to the storage of ds
fit the spell question
* fix(resource manager): extend s3 to the storage of ds
revert the common.properties
* fix(resource manager): extend s3 to the storage of ds
update the storageConfig with None
* fix(resource manager): extend s3 to the storage of ds
fix the judge of resourceType
* fix(resource manager): extend s3 to the storage of ds
undo the compile-mysql
* fix(resource manager): extend s3 to the storage of ds
delete hadoop aws
* fix(resource manager): extend s3 to the storage of ds
update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties
* fix(resource manager): extend s3 to the storage of ds
update the aws-region
* fix(resource manager): extend s3 to the storage of ds
fix the storageconfig init
* fix(resource manager): update e2e docker-compose
update e2e docker-compose
* fix(resource manager): extend s3 to the storage of ds
revent the e2e common.proprites
print the resource type in propertyUtil
* fix(resource manager): extend s3 to the storage of ds
1.println the properties
* fix(resource manager): println the s3 info
* fix(resource manager): extend s3 to the storage of ds
delete the info and upgrade the s3 info to e2e
* fix(resource manager): extend s3 to the storage of ds
add the bucket init
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): extend s3 to the storage of ds
upgrade the s3 endpoint
* fix(resource manager): withPathStyleAccessEnabled(true)
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): upgrade the s3client builder
* fix(resource manager): correct the s3 point to s3client
* fix(resource manager): update the constant BUCKET_NAME
* fix(resource manager): e2e s3 endpoint -> s3:9000
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* style(resource manager): add info to createBucket
* style(resource manager): debug the log
* ci(resource manager): test
test s3
* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql
* fix(resource manager): update the h2 sql
* fix(resource manager): solve to delete the tenant
* style(resource manager): merge the style end delete the unuse s3 config
* fix(resource manager): extend s3 to the storage of ds
UPDATE the rename resources when s3
* fix(resource manager): extend s3 to the storage of ds
1.fix the code style of QuartzImpl
* fix(resource manager): extend s3 to the storage of ds
1.impoort restore_type to CommonUtils
* fix(resource manager): update the work thread
* fix(resource manager): update the baseTaskProcessor
* fix(resource manager): upgrade dolphinscheduler-standalone-server.xml
* fix(resource manager): add user Info to dolphinscheduler_h2.sql
* fix(resource manager): merge the resourceType to NONE
* style(upgrade the log level to info):
* fix(resource manager): sysnc the h2.sql
* fix(resource manager): update the merge the user tenant
* fix(resource manager): merge the resourcesServiceImpl
* fix(resource manager):
when the storage is s3 ,that the directory can't be renamed
* fix(resource manager): in s3 ,the directory cannot be renamed
* fix(resource manager): delete the deleteRenameDirectory in E2E
* fix(resource manager): check the style and recoverd the test
* fix(resource manager): delete the log.print(LoginUser)
* feat(data-quality) change data-quality task main jar path
* feat(data-quality) change data-quality task main jar path
Co-authored-by: zixi0825 <sunzhaohe@linklogis.com>
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
* [Improve]Use hibernate validation framework
Improve email verification method
Support domain name verification
* Email is at least a second-level domain name
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
* threadpool optimization
* threadpool params
* rebase dev
* ut check fix
* add return
* rebase dev
* event loop
Co-authored-by: caishunfeng <534328519@qq.com>
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987: @Param -> @Name
* to #6987: fix Sonar
* to #6987: fix Sonar
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
* [DS-5229][fix] server port custom config
This closes#5229
* [DS-5229][feat] Implement server port custom config
This closes#5229
* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862)
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md
* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852)
Co-authored-by: caishunfeng <534328519@qq.com>
* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes#5229
Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
