You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

service_conf.yaml.template 3.1KB

Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
6 months ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111
  1. ragflow:
  2. host: ${RAGFLOW_HOST:-0.0.0.0}
  3. http_port: 9380
  4. mysql:
  5. name: '${MYSQL_DBNAME:-rag_flow}'
  6. user: '${MYSQL_USER:-root}'
  7. password: '${MYSQL_PASSWORD:-infini_rag_flow}'
  8. host: '${MYSQL_HOST:-mysql}'
  9. port: 3306
  10. max_connections: 900
  11. stale_timeout: 300
  12. max_allowed_packet: ${MYSQL_MAX_PACKET:-1073741824}
  13. minio:
  14. user: '${MINIO_USER:-rag_flow}'
  15. password: '${MINIO_PASSWORD:-infini_rag_flow}'
  16. host: '${MINIO_HOST:-minio}:9000'
  17. es:
  18. hosts: 'http://${ES_HOST:-es01}:9200'
  19. username: '${ES_USER:-elastic}'
  20. password: '${ELASTIC_PASSWORD:-infini_rag_flow}'
  21. os:
  22. hosts: 'http://${OS_HOST:-opensearch01}:9201'
  23. username: '${OS_USER:-admin}'
  24. password: '${OPENSEARCH_PASSWORD:-infini_rag_flow_OS_01}'
  25. infinity:
  26. uri: '${INFINITY_HOST:-infinity}:23817'
  27. db_name: 'default_db'
  28. redis:
  29. db: 1
  30. password: '${REDIS_PASSWORD:-infini_rag_flow}'
  31. host: '${REDIS_HOST:-redis}:6379'
  32. # postgres:
  33. # name: '${POSTGRES_DBNAME:-rag_flow}'
  34. # user: '${POSTGRES_USER:-rag_flow}'
  35. # password: '${POSTGRES_PASSWORD:-infini_rag_flow}'
  36. # host: '${POSTGRES_HOST:-postgres}'
  37. # port: 5432
  38. # max_connections: 100
  39. # stale_timeout: 30
  40. # s3:
  41. # access_key: 'access_key'
  42. # secret_key: 'secret_key'
  43. # region: 'region'
  44. # endpoint_url: 'endpoint_url'
  45. # bucket: 'bucket'
  46. # prefix_path: 'prefix_path'
  47. # signature_version: 'v4'
  48. # addressing_style: 'path'
  49. # oss:
  50. # access_key: '${ACCESS_KEY}'
  51. # secret_key: '${SECRET_KEY}'
  52. # endpoint_url: '${ENDPOINT}'
  53. # region: '${REGION}'
  54. # bucket: '${BUCKET}'
  55. # prefix_path: '${OSS_PREFIX_PATH}'
  56. # azure:
  57. # auth_type: 'sas'
  58. # container_url: 'container_url'
  59. # sas_token: 'sas_token'
  60. # azure:
  61. # auth_type: 'spn'
  62. # account_url: 'account_url'
  63. # client_id: 'client_id'
  64. # secret: 'secret'
  65. # tenant_id: 'tenant_id'
  66. # container_name: 'container_name'
  67. # user_default_llm:
  68. # factory: 'Tongyi-Qianwen'
  69. # api_key: 'sk-xxxxxxxxxxxxx'
  70. # base_url: ''
  71. # default_models:
  72. # chat_model: 'qwen-plus'
  73. # embedding_model: 'BAAI/bge-large-zh-v1.5@BAAI'
  74. # rerank_model: ''
  75. # asr_model: ''
  76. # image2text_model: ''
  77. # oauth:
  78. # oauth2:
  79. # display_name: "OAuth2"
  80. # client_id: "your_client_id"
  81. # client_secret: "your_client_secret"
  82. # authorization_url: "https://your-oauth-provider.com/oauth/authorize"
  83. # token_url: "https://your-oauth-provider.com/oauth/token"
  84. # userinfo_url: "https://your-oauth-provider.com/oauth/userinfo"
  85. # redirect_uri: "https://your-app.com/v1/user/oauth/callback/oauth2"
  86. # oidc:
  87. # display_name: "OIDC"
  88. # client_id: "your_client_id"
  89. # client_secret: "your_client_secret"
  90. # issuer: "https://your-oauth-provider.com/oidc"
  91. # scope: "openid email profile"
  92. # redirect_uri: "https://your-app.com/v1/user/oauth/callback/oidc"
  93. # github:
  94. # type: "github"
  95. # icon: "github"
  96. # display_name: "Github"
  97. # client_id: "your_client_id"
  98. # client_secret: "your_client_secret"
  99. # redirect_uri: "https://your-app.com/v1/user/oauth/callback/github"
  100. # authentication:
  101. # client:
  102. # switch: false
  103. # http_app_key:
  104. # http_secret_key:
  105. # site:
  106. # switch: false
  107. # permission:
  108. # switch: false
  109. # component: false
  110. # dataset: false