* fix bug: send task result to master until success.
* fix bug: send task result to master until success.
* add sleep
* set null if send result failed
* set null if send result failed
* change max length of username to 39
Co-authored-by: baoliang <baoliang@analysys.com.cn>
* The value method of enum is a deep copy. If there are many parameters, it is not recommended to use the value method. Specifically see jmh results
* beautiful misunderstanding
Co-authored-by: dailidong <dailidong66@gmail.com>
* job history status url when application number threshold is reached(default 10000,maybe it was set to 1000)
* job history status url when application number threshold is reached(default 10000,maybe it was set to 1000)
* job history status url when application number threshold is reached(default 10000,maybe it was set to 1000)
Co-authored-by: yuhaibin@lizhi.fm <35716fc5847f6d154cf556296453ca91>
Co-authored-by: dailidong <dailidong66@gmail.com>
* When update the data source,the host and port show error
* update DataSourceServiceTest
* update DataSourceServiceTest
* update DataSourceServiceTest
* fix bug: some bugs about worker group from 1.2.0 to 1.3.0
* fix bug: scheduler from 1.2.0 to 1.3.0 do not work
* fix bug: scheduler from 1.2.0 to 1.3.0 do not work
* fix ut bug
* fix bug: Process shows success,when the task of the process is killed in the termina
* fix bug #2634: Process shows success,when the task of the process is killed in the terminal
Co-authored-by: baoliang <baoliang@analysys.com.cn>
* fix worker group config no effect
* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2
* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0
# Conflicts:
# dolphinscheduler-server/src/main/resources/config/install_config.conf
* Update datasource.properties
* script variable has "processDefinitionId" is error #2664
* blank in eamil and left font align #2648
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* Fix some code specification issues
* Pre-compile regular expressions to improve efficiency and solve concurrency problems
* Solve the nullpoint problem that Object.equals may bring
* Modify if else structure, add {} to improve code readability
Co-Authored-By: daili <daili@users.noreply.github.com>
* Update ResourceTreeVisitor.java
re-run test process
Co-authored-by: daili <daili@users.noreply.github.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
* update README about DolphinScheduler
* Update issue templates
* update
* regularize api pom xml
update rpc maven compile to 1.8
* regularize api pom xml
* change commons.lang3.StringUtils to common.utils.StringUtils
* update pom.xml
* update
* correct equals method
* jasper-runtime is needed when api server start
* jasper-runtime jar is needed when api server start
* combine logback config of master/worker/alert/api server to one logback.xml
* remove tomcat runtime jar
* add UT
* add license
* remove jasper-runtime jar, not need anymore
* sovle jar conflict， remove servlet-api 2.5 jar
* remove servlet 2.5
* add embedded database h2 and update embedded zookeeper
* delete logs
* add license
* change log level to INFO
* simplify and optimize config
* add development state
* simplify common config
Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
* update README about DolphinScheduler
* Update issue templates
* update
* regularize api pom xml
update rpc maven compile to 1.8
* regularize api pom xml
* change commons.lang3.StringUtils to common.utils.StringUtils
* update pom.xml
* update
* correct equals method
* jasper-runtime is needed when api server start
* jasper-runtime jar is needed when api server start
* combine logback config of master/worker/alert/api server to one logback.xml
* remove tomcat runtime jar
* add UT
* add license
* remove jasper-runtime jar, not need anymore
* sovle jar conflict， remove servlet-api 2.5 jar
* remove servlet 2.5
* add embedded database h2 and update embedded zookeeper
* delete logs
* add license
* change log level to INFO
* simplify and optimize config
* add development state
Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
* fix#2442 when the resource name contains '$',need translate it to '\$'
* fix#2442 when the resource name contains '$',need translate it to '/$'
* verify whether the filename start with '/'
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
* dispatch task fail will set task status failed
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，task status statistics and process status statistics bug fix (#2357)
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357)
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357)
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357)
2，worker group bug fix
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* init full_name in dml of version 1.2.2
* redefine request parameter resourceIds
* redefine request parameter resourceIds
* Actually grant resource file if choose the directory
* To cancel authorized resource need check whether it is used by the process definition which is online
* If release the process definition online,It is necessary to check whether resource is valid
* update ResourceServiceTest and ResourceMapperTest
* add batchUpdateResourceTest
* add getHdfsFileNameTest and getHdfsResourceFileNameTest
* update ResourceServiceTest and ResourceMapperTest
* extract getResourceProcessMap to ResourceProcessDefinitonUtils
Co-authored-by: dailidong <dailidong66@gmail.com>
* Adapting partial code(file name start with O) to the sonar cloud rule
* resolve conflict with dev branch
Co-authored-by: dailidong <dailidong66@gmail.com>
* add create resource directory
* add create resource directory
* update the resource test
* add upgrade sql in version 1.2.2
* Adding request parameter id to update queryResourceListPaging
* set isDirectory value is false default
* add full name to update updateResource
* remove request parameter isDirectory to update createResource method
* update queryResourceListPaging with change get to post
* update updateResource method with remove fullName
* File management list modification (#1976)
* add resource component
* add resource tree visitor
* return json string
* update queryResourceList
* upload file need fullName
* add method rootNode
* Shell task resources and authorization resources (#1989)
* File management list modification
* Shell task resources and authorization resources
* download resource when execute task
* download resource when execute task
* update authorization type
* download resource when execute task
* Spark task resource changes (#1990)
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* download resource when execute task
* update udf function service
* add resource type in ResourceComponent
* UDF resource tree and change DAG style (#2019)
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* UDF resource tree and change DAG style
* add deleteIds method in ResourceMapper and ResourceMapperTest
* Add comments on class and method
* add queryResourceByName method in controller
* update verify-name with change name to full name
* update queryResource with add parameter pid
* update queryResource with add parameter pid
* add resource ids in process definition and delete resource need judge whether it is used by any process definition
* Breadcrumb development (#2033)
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* UDF resource tree and change DAG style
* Breadcrumb development
* Breadcrumb development
* Resource tree bug fix (#2040)
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* UDF resource tree and change DAG style
* Breadcrumb development
* Breadcrumb development
* Resource tree bug fix
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* Fix github action rerun failed (#2067)
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* Change crumb position
* Change crumb position (#2068)
* build resource process definition map
* UDF changed to multiple choice
* UDF changed to multiple choice (#2077)
* Change crumb position
* UDF changed to multiple choice
* build resource process definition map (#2076)
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* update resource name also need update all the children full name (#2096)
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* Limit customization file content to no more than 3000 lines
* Limit customization file content to no more than 3000 lines(#2128)
* Limit customization file content to no more than 3000 lines(#2128) (#2140)
* Change crumb position
* UDF changed to multiple choice
* Limit customization file content to no more than 3000 lines
* Limit customization file content to no more than 3000 lines(#2128)
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList (#2192)
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* Modify the main jar package
* Modify the main jar package (#2200)
* Change crumb position
* UDF changed to multiple choice
* Limit customization file content to no more than 3000 lines
* Limit customization file content to no more than 3000 lines(#2128)
* Modify the main jar package
* add resource filter in order to get filtered resource
* add comments of resource filter
* update list children by resource
* choose main jar with resource tree (#2220)
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* add resource filter in order to get filtered resource
* add comments of resource filter
* update list children by resource
* Return null if query resource list is empty
* update queryResource method change parameter pid to id
* getResouDelete checksum and modify parameter namerceId
* revert .env
* remove parameter pid
* Delete request interface
* go back to the last page
* jar interface call
* Fix issue #2234 and #2228
* change resource name with full name
* Fix issue #2234 and #2228 (#2246)
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* add resource filter in order to get filtered resource
* add comments of resource filter
* update list children by resource
* Return null if query resource list is empty
* update queryResource method change parameter pid to id
* revert .env
* remove parameter pid
* Fix issue #2234 and #2228
* change resource name with full name
* Fix list query value error
* remove unauth-file with authorize-resource-tree
* Repair data cannot be echoed
* Repair data cannot be echoed
* execute mr and spark task need query resource name before
* Authorized resource interface replacement
* Authorized resource interface replacement
* Filter UDF resources
* Change parameters
* need query all authorized directory children when create task
* Change normalize.scss import method and animation.scss license modification
* Delete file list update processing
* It's fixed that resource not deleted in hdfs when delete it.
* add tooltips
* add tooltips (#2310)
* Echo workflow name and modify udf management name
* [new feature]add resource tree function
* revert front code in order to be same as dev branch
* revert front code in order to be same as dev branch
* revert common.properties and application.properties
* add super method
* update flink parameter test
* update flink parameter and unit test
* update resource service test
* If resource list is empty,need init it
* update flink parameter test
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
* add release notes in 1.2.1 (#1654)
* fix Monitor bug (#1656)
* fix Monitor bug
* fix zk monitor bug
* fix api url (#1673)
* get root path from zookeeper config
* Fixed DAG zoom in and zoom out nodes separated from arrows #1679 (#1685)
* fix api url
* Fixed DAG zoom in and zoom out nodes separated from arrows #1679
* check license and update (#1725)
* Fixed tasks_queue and tasks_kill did not exist in zookeeper #1696 (#1734)
Co-authored-by: elonlo <gamelifedong@gmail.com>
* upgrade jackson from 2.9.8 to 2.10.1 (#1767)
* add out put log when master/worker server start (#1769)
* merge hadoop.properties into common.properties
* merge hadoop,zookeeper.properties into common.properties
remove combined.properties/master.properties/worker.properties
* change db user/pwd to test/test
* rename .dolphinscheduler_en.sh to dolphinscheduler_env.sh
remove some unused in install.sh
* add out put log when master/worker server start...
* add start log when servers start
* add check download resource permission in order to fix issues 1770 (#1788)
* merge dev branch front-end code (#1786)
* fix#1775 bug，delete process definition when process instance is running (#1790)
* fix#1775 bug，delete process definition when process instance is running
* revert CONTRIBUTING.md
* fix udfs assignment and task instance page jump #1789 (#1791)
* merge dev branch front-end code
* fix udfs assignment
* Fix task instance page jump
* fix udfs assignment and task instance page jump #1789
* update method checkDownloadPermission in order to fix issues 1770 (#1794)
* add check download resource permission in order to fix issues 1770
* update method checkDownloadPermission in order to fix issues 1770
* update method listUnauthorizedResource in order to fix issues 1770
* update method listUnauthorizedResource in order to fix issues 1770 (#1797)
* if login user is admin,it will has permission to query all udf function (#1799)
* Fixed space and icon display issues before and after the input box (#1798)
* merge dev branch front-end code
* fix udfs assignment
* Fix task instance page jump
* fix udfs assignment and task instance page jump #1789
* Fixed space and icon display issues before and after the input box
* add license
* add license
* donot submit task to queue if sub process (#1793)
* dockerfile modify (#1800)
* fix#1775 bug，delete process definition when process instance is running
* revert CONTRIBUTING.md
* dockerfile modify
* dockerfile modify
* dockerfile modify
* Fix data echo, style and popup cannot be closed (#1801)
* merge dev branch front-end code
* fix udfs assignment
* Fix task instance page jump
* fix udfs assignment and task instance page jump #1789
* Fixed space and icon display issues before and after the input box
* add license
* add license
* Fix data echo, style and popup cannot be closed
* Fix page number loading issue and dag not getting value(#1810) (#1815)
* merge dev branch front-end code
* fix udfs assignment
* Fix task instance page jump
* fix udfs assignment and task instance page jump #1789
* Fixed space and icon display issues before and after the input box
* add license
* add license
* Fix data echo, style and popup cannot be closed
* Fix page number loading issue and dag not getting value
* [Fix issue #1770]check udf and data source in order to fix issue 1770 (#1817)
* if login user is admin,it will has permission to query all udf function
* check udf and data source in order to fix issue 1770
* check udf and data source in order to fix issue 1770
* check udf and data source in order to fix issue 1770
* check udf and data source in order to fix issue 1770
* check udf and data source in order to fix issue 1770
* revert common.properties
* update the test method name
* remove useless code and import in unit test
* refactor has permission and check permission
* #1813 remove "_001" after the master/server register path in zookeepe (#1824)
* donot submit task to queue if sub process
* [feature] #1813 remove "_001" after the master/server register path in zookeeper (#1820)
* change master/worker register path.
* remove "_" from register path.
* remove install.sh server.servlet.context-path replace (#1823)
* fix#1775 bug，delete process definition when process instance is running
* revert CONTRIBUTING.md
* dockerfile modify
* dockerfile modify
* dockerfile modify
* remove install.sh server.servlet.context-path replace
* Fix DAG add dependent project value exception # 1816 and keep requesting task interface (#1827)
* merge dev branch front-end code
* fix udfs assignment
* Fix task instance page jump
* fix udfs assignment and task instance page jump #1789
* Fixed space and icon display issues before and after the input box
* add license
* add license
* Fix data echo, style and popup cannot be closed
* Fix page number loading issue and dag not getting value
* Fix DAG add dependent project value exception # 1816 and keep requesting task interface
* rpm package add UI (#1846)
* [fix#1828] when the executor of process instance is not the owner of udf resouce, the path of the read resource file is incorrect (#1847)
* fix issue 1828:get the udf resource path error when create udf function
* update grantResources
* first verify whether udf resource is bound by udf function
* update grantResources
* update testListAuthorizedUdfFunc
* update getUserInfo in order to run success
* add LoggerServer RPC PORT modified. #1848 (#1849)
* fix#1775 bug，delete process definition when process instance is running
* revert CONTRIBUTING.md
* dockerfile modify
* dockerfile modify
* dockerfile modify
* remove install.sh server.servlet.context-path replace
* add LoggerServer RPC PORT modified. #1848
* LoggerService Logger RPC PORT get Error modify . #1848 (#1857)
* fix#1775 bug，delete process definition when process instance is running
* revert CONTRIBUTING.md
* dockerfile modify
* dockerfile modify
* dockerfile modify
* remove install.sh server.servlet.context-path replace
* add LoggerServer RPC PORT modified. #1848
* LoggerService Logger RPC PORT get Error modify . #1848
* Fix UDF function list delete data without refresh and page data display widened #1851 (#1852)
* merge dev branch front-end code
* fix udfs assignment
* Fix task instance page jump
* fix udfs assignment and task instance page jump #1789
* Fixed space and icon display issues before and after the input box
* add license
* add license
* Fix data echo, style and popup cannot be closed
* Fix page number loading issue and dag not getting value
* Fix DAG add dependent project value exception # 1816 and keep requesting task interface
* Fix UDF function list delete data without refresh and page data display widened
* [Fix#1828]check whether has permission to download udf file or delete udf function (#1858)
* fix issue 1828:get the udf resource path error when create udf function
* update grantResources
* first verify whether udf resource is bound by udf function
* update grantResources
* update testListAuthorizedUdfFunc
* update getUserInfo in order to run success
* check whether has permission to download udf file or delete udf file
* update listAuthorizedResourceById in ResourceMapper.xml
* add log, don't swallow exception info (#1877)
* Added hints in Resource Center (#1891)
* update jackson version from 2.9.8 to 2.10.1 (#1892)
* update release notes and modify the plugin of rat (#1899)
* modify how to build (#1902)
* update release notes and modify the plugin of rat
* modify how to build
* [maven-release-plugin] prepare release 1.2.1
* [maven-release-plugin] prepare for next development iteration
* get authorized udf resources need exclude all created by self (#1905)
* get authorized udf resources need exclude all created by self
* reset pom version to 1.2.1-SNAPSHOT
* Update ReleaseNotes.md
add detail info
* Update ReleaseNotes.md
* update notice year form 2019 to 2019-2020 (#1907)
* [maven-release-plugin] prepare release 1.2.1
* [maven-release-plugin] prepare for next development iteration
* 当手动重跑历史活着补数的时候对shell和sql脚本中的[YYYYmmddd...]变量赋值与传递的日期，而不是当前日期
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: bao liang <29528966+lenboo@users.noreply.github.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: elonlo <gamelifedong@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: zhangchunyang <18910529250@163.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: 谷雨 <lixun@guyudeMacBook-Pro.local>
* Cache HadoopUtils instance with 7 days expire time
* solve sonar issue
* add kerberos expire time config
* move KERBEROS_EXPIRE_TIME to Constants.java
* Adapting partial code(file name start with F) to the sonar cloud rule
* add more unit test
* add License
* add includes configuration to maven-surefire-plugin
* fix getResourceFilesList incorrect logic
* access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability
Current implementation relies on `sun.misc.JavaIOFileDescriptorAccess`
which is only accessible on oraclejdk8.
Basically the demand is getting & setting `handle` field of
`FileDescriptor`, so we can directly do that with reflection.
Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.
* harden initialization of ProcessImplForWin32
* ignore ShellTaskTest#testHandleForWindows outside Windows
* Support worker server to run bat script
1. Reimplement ProcessImpl.java, ProcessEnvironment.java and ProcessBuilder.java for Windows
2. Modify shell task code for windows
3. Add ASF License
* Add Unit Test
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
