You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

service_conf.yaml 2.8KB

Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
6 months ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103
  1. ragflow:
  2. host: 0.0.0.0
  3. http_port: 9380
  4. mysql:
  5. name: 'rag_flow'
  6. user: 'root'
  7. password: 'infini_rag_flow'
  8. host: 'localhost'
  9. port: 5455
  10. max_connections: 900
  11. stale_timeout: 300
  12. max_allowed_packet: 1073741824
  13. minio:
  14. user: 'rag_flow'
  15. password: 'infini_rag_flow'
  16. host: 'localhost:9000'
  17. es:
  18. hosts: 'http://localhost:1200'
  19. username: 'elastic'
  20. password: 'infini_rag_flow'
  21. os:
  22. hosts: 'http://localhost:1201'
  23. username: 'admin'
  24. password: 'infini_rag_flow_OS_01'
  25. infinity:
  26. uri: 'localhost:23817'
  27. db_name: 'default_db'
  28. redis:
  29. db: 1
  30. password: 'infini_rag_flow'
  31. host: 'localhost:6379'
  32. # postgres:
  33. # name: 'rag_flow'
  34. # user: 'rag_flow'
  35. # password: 'infini_rag_flow'
  36. # host: 'postgres'
  37. # port: 5432
  38. # max_connections: 100
  39. # stale_timeout: 30
  40. # s3:
  41. # access_key: 'access_key'
  42. # secret_key: 'secret_key'
  43. # region: 'region'
  44. # oss:
  45. # access_key: 'access_key'
  46. # secret_key: 'secret_key'
  47. # endpoint_url: 'http://oss-cn-hangzhou.aliyuncs.com'
  48. # region: 'cn-hangzhou'
  49. # bucket: 'bucket_name'
  50. # azure:
  51. # auth_type: 'sas'
  52. # container_url: 'container_url'
  53. # sas_token: 'sas_token'
  54. # azure:
  55. # auth_type: 'spn'
  56. # account_url: 'account_url'
  57. # client_id: 'client_id'
  58. # secret: 'secret'
  59. # tenant_id: 'tenant_id'
  60. # container_name: 'container_name'
  61. # The OSS object storage uses the MySQL configuration above by default. If you need to switch to another object storage service, please uncomment and configure the following parameters.
  62. # opendal:
  63. # scheme: 'mysql' # Storage type, such as s3, oss, azure, etc.
  64. # config:
  65. # oss_table: 'opendal_storage'
  66. # user_default_llm:
  67. # factory: 'Tongyi-Qianwen'
  68. # api_key: 'sk-xxxxxxxxxxxxx'
  69. # base_url: ''
  70. # oauth:
  71. # oauth2:
  72. # display_name: "OAuth2"
  73. # client_id: "your_client_id"
  74. # client_secret: "your_client_secret"
  75. # authorization_url: "https://your-oauth-provider.com/oauth/authorize"
  76. # token_url: "https://your-oauth-provider.com/oauth/token"
  77. # userinfo_url: "https://your-oauth-provider.com/oauth/userinfo"
  78. # redirect_uri: "https://your-app.com/v1/user/oauth/callback/oauth2"
  79. # oidc:
  80. # display_name: "OIDC"
  81. # client_id: "your_client_id"
  82. # client_secret: "your_client_secret"
  83. # issuer: "https://your-oauth-provider.com/oidc"
  84. # scope: "openid email profile"
  85. # redirect_uri: "https://your-app.com/v1/user/oauth/callback/oidc"
  86. # github:
  87. # type: "github"
  88. # icon: "github"
  89. # display_name: "Github"
  90. # client_id: "your_client_id"
  91. # client_secret: "your_client_secret"
  92. # redirect_uri: "https://your-app.com/v1/user/oauth/callback/github"
  93. # authentication:
  94. # client:
  95. # switch: false
  96. # http_app_key:
  97. # http_secret_key:
  98. # site:
  99. # switch: false
  100. # permission:
  101. # switch: false
  102. # component: false
  103. # dataset: false