mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git synced 2024-11-29 18:58:05 +08:00

Migrated repository

Go to file

xiangzihao 627c76b75a Some checks failed API-Test / API-Test-Path-Filter (push) Has been cancelled Details Backend / Backend-Path-Filter (push) Has been cancelled Details CodeQL / Analyze (push) Has been cancelled Details E2E-K8S / E2E-K8S-Path-Filter (push) Has been cancelled Details E2E / E2E-Path-Filter (push) Has been cancelled Details publish-docker / build (push) Has been cancelled Details publish-helm-chart / build (push) Has been cancelled Details Test / Unit-Test-Path-Filter (push) Has been cancelled Details API-Test / API-Test-Build (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.ExecutorAPITest name:ExecutorAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.LdapLoginAPITest name:LdapLoginAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.ProjectAPITest name:ProjectAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.SchedulerAPITest name:SchedulerAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.TenantAPITest name:TenantAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.WorkerGroupAPITest name:WorkerGroupAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.WorkflowDefinitionAPITest name:WorkflowDefinitionAPITest]) (push) Has been cancelled Details API-Test / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.api.test.cases.WorkflowInstanceAPITest name:WorkflowInstanceAPITest]) (push) Has been cancelled Details API-Test / API-Test-Result (push) Has been cancelled Details Backend / Backend-Build (11) (push) Has been cancelled Details Backend / Backend-Build (8) (push) Has been cancelled Details Backend / ${{ matrix.case.name }} (map[name:cluster-test-mysql-with-mysql-registry script:.github/workflows/cluster-test/mysql_with_mysql_registry/start-job.sh]) (push) Has been cancelled Details Backend / ${{ matrix.case.name }} (map[name:cluster-test-mysql-with-zookeeper-registry script:.github/workflows/cluster-test/mysql_with_zookeeper_registry/start-job.sh]) (push) Has been cancelled Details Backend / ${{ matrix.case.name }} (map[name:cluster-test-postgresql-with-postgresql-registry script:.github/workflows/cluster-test/postgresql_with_postgresql_registry/start-job.sh]) (push) Has been cancelled Details Backend / ${{ matrix.case.name }} (map[name:cluster-test-postgresql-zookeeper-registry script:.github/workflows/cluster-test/postgresql_with_zookeeper_registry/start-job.sh]) (push) Has been cancelled Details Backend / ${{ matrix.case.name }}-${{ matrix.version }} (map[name:schema-check-with-mysql script:.github/workflows/schema-check/mysql/start-job.sh], 3.1.9) (push) Has been cancelled Details Backend / ${{ matrix.case.name }}-${{ matrix.version }} (map[name:schema-check-with-mysql script:.github/workflows/schema-check/mysql/start-job.sh], 3.2.0) (push) Has been cancelled Details Backend / ${{ matrix.case.name }}-${{ matrix.version }} (map[name:schema-check-with-postgresql script:.github/workflows/schema-check/postgresql/start-job.sh], 3.1.9) (push) Has been cancelled Details Backend / ${{ matrix.case.name }}-${{ matrix.version }} (map[name:schema-check-with-postgresql script:.github/workflows/schema-check/postgresql/start-job.sh], 3.2.0) (push) Has been cancelled Details Backend / Build (push) Has been cancelled Details E2E-K8S / E2E-K8S-Execute (push) Has been cancelled Details E2E-K8S / E2E-K8S-Result (push) Has been cancelled Details E2E / E2E-Build (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.ClickhouseDataSourceE2ETest name:ClickhouseDataSourceE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.ClusterE2ETest name:ClusterE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.EnvironmentE2ETest name:EnvironmentE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.FileManageE2ETest name:FileManageE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.HiveDataSourceE2ETest name:HiveDataSourceE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.MysqlDataSourceE2ETest name:MysqlDataSourceE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.PostgresDataSourceE2ETest name:PostgresDataSourceE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.ProjectE2ETest name:ProjectE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.QueueE2ETest name:QueueE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.SqlServerDataSourceE2ETest name:SqlServerDataSourceE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.TenantE2ETest name:TenantE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.TokenE2ETest name:TokenE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.UserE2ETest name:UserE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.WorkerGroupE2ETest name:WorkerGroupE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.WorkflowE2ETest name:WorkflowE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.WorkflowHttpTaskE2ETest name:WorkflowHttpTaskE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.WorkflowJavaTaskE2ETest name:WorkflowJavaTaskE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.tasks.PythonTaskE2ETest name:PythonTaskE2ETest]) (push) Has been cancelled Details E2E / ${{ matrix.case.name }} (map[class:org.apache.dolphinscheduler.e2e.cases.tasks.ShellTaskE2ETest name:ShellTaskE2ETest]) (push) Has been cancelled Details E2E / E2E (push) Has been cancelled Details Test / Unit-Test (11) (push) Has been cancelled Details Test / Unit-Test (8) (push) Has been cancelled Details Test / Unit Test (push) Has been cancelled Details [Improvement-16106] Add admin-user-filter in LDAP (#16826 ) * improvement 16106 * fix ut * fix comment * fix spotless		2024-11-22 17:39:18 +08:00
.github	[Improvement-16106] Add admin-user-filter in LDAP (#16826 )	2024-11-22 17:39:18 +08:00
.idea	[Chore] Add issue link tempate for IDEA (#16158 )	2024-06-18 21:02:15 +08:00
.mvn/wrapper	[Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557 )	2022-02-27 19:54:23 +08:00
config	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
deploy	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
docs	[Improvement-16106] Add admin-user-filter in LDAP (#16826 )	2024-11-22 17:39:18 +08:00
dolphinscheduler-alert	[Improvement-16758][Deploy] Record the fatal error Log (#16829 )	2024-11-21 19:54:36 +08:00
dolphinscheduler-api	[Improvement-16106] Add admin-user-filter in LDAP (#16826 )	2024-11-22 17:39:18 +08:00
dolphinscheduler-api-test	[Improvement-16106] Add admin-user-filter in LDAP (#16826 )	2024-11-22 17:39:18 +08:00
dolphinscheduler-authentication	[DSIP-42] Add dolphinscheduler-aws-authentication module (#16043 )	2024-05-24 14:15:03 +08:00
dolphinscheduler-bom	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-common	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-dao	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-dao-plugin	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
dolphinscheduler-datasource-plugin	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-dist	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-e2e	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-eventbus	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
dolphinscheduler-extract	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
dolphinscheduler-master	[Improvement-16758][Deploy] Record the fatal error Log (#16829 )	2024-11-21 19:54:36 +08:00
dolphinscheduler-meter	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
dolphinscheduler-microbench	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
dolphinscheduler-registry	[DSIP-82][Master/Worker] Use FAILOVER_FINISH_NODES to avoid duplicate workflow/task when failover (#16821 )	2024-11-20 08:04:11 +00:00
dolphinscheduler-scheduler-plugin	[DSIP-68] Unify the table structure and table fields in backend and frontend ui sections (#16544 )	2024-09-04 21:22:34 +08:00
dolphinscheduler-service	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-spi	[DSIP-38] Add plugin directory in binary package (#16613 )	2024-09-23 09:59:34 +08:00
dolphinscheduler-standalone-server	[Improvement-16106] Add admin-user-filter in LDAP (#16826 )	2024-11-22 17:39:18 +08:00
dolphinscheduler-storage-plugin	[Feature-16801][UI] Add copy path of resource file in task definition (#16803 )	2024-11-18 17:12:18 +08:00
dolphinscheduler-task-plugin	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-tools	[DSIP-38] Add plugin directory in binary package (#16613 )	2024-09-23 09:59:34 +08:00
dolphinscheduler-ui	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
dolphinscheduler-worker	[Improvement-16758][Deploy] Record the fatal error Log (#16829 )	2024-11-21 19:54:36 +08:00
dolphinscheduler-yarn-aop	Rename dolphinscheduler-aop to dolphinscheduler-yarn-aop (#16095 )	2024-05-31 17:12:22 +08:00
images	[Doc] fix CNCF logo (#15634 )	2024-02-27 14:42:07 +08:00
licenses	[Fix][Common] rewrite code generate,fix bit shift (#6914 )	2021-11-19 16:04:05 +08:00
script	[DSIP-38] Support upload/download plugins from maven repo (#16654 )	2024-09-26 14:39:14 +08:00
style	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
tools	[Improvement-16507] Remove spring-boot-starter-cache (#16593 )	2024-10-21 17:14:41 +08:00
.asf.yaml	[DSIP-38] Add plugin directory in binary package (#16613 )	2024-09-23 09:59:34 +08:00
.dlc.json	[Feature][DataSource] add oceanbase datasource (#13675 )	2023-03-07 10:12:58 +08:00
.flake8	[release] version 3.0.3 (#13131 )	2022-12-08 16:57:17 +08:00
.gitattributes	Create .gitattributes	2019-03-29 23:51:32 +08:00
.gitignore	[DSIP-68] Unify the table structure and table fields in backend and frontend ui sections (#16544 )	2024-09-04 21:22:34 +08:00
.gitmodules	[Chore] Fix push docker ci error (#16418 )	2024-08-04 10:13:08 +08:00
.licenserc.yaml	[DSIP-38] Support upload/download plugins from maven repo (#16654 )	2024-09-26 14:39:14 +08:00
.pre-commit-config.yaml	[chore] Separate Python API into another repository (#12779 )	2022-11-09 21:20:15 +08:00
CONTRIBUTING.md	doc: Change readme and standalone docker quick start (#14002 )	2023-04-25 09:35:49 +08:00
doap.rdf	Add doap.rdf (#14883 )	2023-09-09 22:40:36 +08:00
LICENSE	[DSIP-75] Support auto add license header by spotless (#16643 )	2024-09-22 16:27:29 +08:00
lombok.config	add lombok config (#11212 )	2022-08-02 11:46:58 +08:00
mvnw	[Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557 )	2022-02-27 19:54:23 +08:00
mvnw.cmd	[Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557 )	2022-02-27 19:54:23 +08:00
NOTICE	[Fix-16314][NOTICE] Correct year in NOTICE file (#16316 )	2024-07-15 18:01:45 +08:00
pom.xml	[DSIP-78][Data Quality] Remove data quality module (#16794 )	2024-11-20 09:20:41 +08:00
README_zh_CN.md	[Chore] Remove unsupported feature in README.md (#16549 )	2024-08-30 13:52:48 +08:00
README.md	[Chore] Remove unsupported feature in README.md (#16549 )	2024-08-30 13:52:48 +08:00

README.md

Apache Dolphinscheduler

About

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code. It is also provided powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available out of the box

The key features for DolphinScheduler are as follows:

Easy to deploy, provide four ways to deploy which including Standalone, Cluster, Docker and Kubernetes.
Easy to use, workflow can be created and managed by four ways, which including Web UI, Python SDK and Open API
Highly reliable and high availability, decentralized architecture with multi-master and multi-worker, native supports horizontal scaling.
High performance, its performance is N times faster than other orchestration platform and it can support tens of millions of tasks per day
Cloud Native, DolphinScheduler supports orchestrating multi-cloud/data center workflow, and supports custom task type
Versioning both workflow and workflow instance(including tasks)
Various state control of workflow and task, support pause/stop/recover them in any time
Multi-tenancy support
Others like backfill support(Web UI native), permission control including project and data source

QuickStart

For quick experience
- Want to start with standalone
- Want to start with Docker
For Kubernetes
- Start with Kubernetes
For Terraform
- Start with Terraform

User Interface Screenshots

Homepage: Project and workflow overview, including the latest workflow instance and task instance status statistics.
Workflow Definition: Create and manage workflow by drag and drop, easy to build and maintain complex workflow, support bulk of tasks out of box.
Workflow Tree View: Abstract tree structure could clearer understanding of the relationship between tasks
Data source: Manage support multiple external data sources, provide unified data access capabilities for such as MySQL, PostgreSQL, Hive, Trino, etc.
Monitor: View the status of the master, worker and database in real time, including server resource usage and load, do quick health check without logging in to the server.

Suggestions & Bug Reports

Follow this guide to report your suggestions or bugs.

Contributing

The community welcomes everyone to contribute, please refer to this page to find out more: How to contribute, find the good first issue in here if you are new to DolphinScheduler.

Community

Welcome to join the Apache DolphinScheduler community by:

Join the DolphinScheduler Slack to keep in touch with the community
Follow the DolphinScheduler Twitter and get the latest news
Subscribe DolphinScheduler mail list, users@dolphinscheduler.apache.org for user and dev@dolphinscheduler.apache.org for developer

Landscapes

DolphinScheduler enriches the CNCF CLOUD NATIVE Landscape.