dify/api/core/model_runtime
2024-10-03 15:17:28 +08:00
..
callbacks refactor: update Callback to an abstract class (#8868) 2024-09-28 21:41:02 +08:00
docs docs: add english versions for the files customizable_model_scale_out and predefined_model_scale_out (#8871) 2024-09-29 16:16:56 +08:00
entities chore: refurish python code by applying Pylint linter rules (#8322) 2024-09-13 22:42:08 +08:00
errors chore(api/core): apply ruff reformatting (#7624) 2024-09-10 17:00:20 +08:00
model_providers feat: Update AWS Bedrock supported regions (#8992) 2024-10-03 15:17:28 +08:00
schema_validators chore: refurish python code by applying Pylint linter rules (#8322) 2024-09-13 22:42:08 +08:00
utils chore(api/core): apply ruff reformatting (#7624) 2024-09-10 17:00:20 +08:00
__init__.py Model Runtime (#1858) 2024-01-02 23:42:00 +08:00
README_CN.md Update README_CN.md (#2550) 2024-02-23 17:38:03 +08:00
README.md fix: Corrected schema link in model_runtime's README.md (#4757) 2024-05-29 13:03:21 +08:00

Model Runtime

This module provides the interface for invoking and authenticating various models, and offers Dify a unified information and credentials form rule for model providers.

  • On one hand, it decouples models from upstream and downstream processes, facilitating horizontal expansion for developers,
  • On the other hand, it allows for direct display of providers and models in the frontend interface by simply defining them in the backend, eliminating the need to modify frontend logic.

Features

  • Supports capability invocation for 5 types of models

    • LLM - LLM text completion, dialogue, pre-computed tokens capability
    • Text Embedding Model - Text Embedding, pre-computed tokens capability
    • Rerank Model - Segment Rerank capability
    • Speech-to-text Model - Speech to text capability
    • Text-to-speech Model - Text to speech capability
    • Moderation - Moderation capability
  • Model provider display

    image-20231210143654461

    Displays a list of all supported providers, including provider names, icons, supported model types list, predefined model list, configuration method, and credentials form rules, etc. For detailed rule design, see: Schema.

  • Selectable model list display

    image-20231210144229650

    After configuring provider/model credentials, the dropdown (application orchestration interface/default model) allows viewing of the available LLM list. Greyed out items represent predefined model lists from providers without configured credentials, facilitating user review of supported models.

    In addition, this list also returns configurable parameter information and rules for LLM, as shown below:

    image-20231210144814617

    These parameters are all defined in the backend, allowing different settings for various parameters supported by different models, as detailed in: Schema.

  • Provider/model credential authentication

    image-20231210151548521

    image-20231210151628992

    The provider list returns configuration information for the credentials form, which can be authenticated through Runtime's interface. The first image above is a provider credential DEMO, and the second is a model credential DEMO.

Structure

Model Runtime is divided into three layers:

  • The outermost layer is the factory method

    It provides methods for obtaining all providers, all model lists, getting provider instances, and authenticating provider/model credentials.

  • The second layer is the provider layer

    It provides the current provider's model list, model instance obtaining, provider credential authentication, and provider configuration rule information, allowing horizontal expansion to support different providers.

  • The bottom layer is the model layer

    It offers direct invocation of various model types, predefined model configuration information, getting predefined/remote model lists, model credential authentication methods. Different models provide additional special methods, like LLM's pre-computed tokens method, cost information obtaining method, etc., allowing horizontal expansion for different models under the same provider (within supported model types).

Next Steps

  • Add new provider configuration: Link
  • Add new models for existing providers: Link
  • View YAML configuration rules: Link
  • Implement interface methods: Link