You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

.env 7.3KB

Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
6 months ago
1 year ago
1 year ago
Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
6 months ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
Feat: make document parsing and embedding batch sizes configurable via environment variables (#8266) ### Description This PR introduces two new environment variables, ‎`DOC_BULK_SIZE` and ‎`EMBEDDING_BATCH_SIZE`, to allow flexible tuning of batch sizes for document parsing and embedding vectorization in RAGFlow. By making these parameters configurable, users can optimize performance and resource usage according to their hardware capabilities and workload requirements. ### What problem does this PR solve? Previously, the batch sizes for document parsing and embedding were hardcoded, limiting the ability to adjust throughput and memory consumption. This PR enables users to set these values via environment variables (in ‎`.env`, Helm chart, or directly in the deployment environment), improving flexibility and scalability for both small and large deployments. - ‎`DOC_BULK_SIZE`: Controls how many document chunks are processed in a single batch during document parsing (default: 4). - ‎`EMBEDDING_BATCH_SIZE`: Controls how many text chunks are processed in a single batch during embedding vectorization (default: 16). This change updates the codebase, documentation, and configuration files to reflect the new options. ### Type of change - [ ] Bug Fix (non-breaking change which fixes an issue) - [x] New Feature (non-breaking change which adds functionality) - [x] Documentation Update - [ ] Refactoring - [x] Performance Improvement - [ ] Other (please describe): ### Additional context - Updated ‎`.env`, ‎`helm/values.yaml`, and documentation to describe the new variables. - Modified relevant code paths to use the environment variables instead of hardcoded values. - Users can now tune these parameters to achieve better throughput or reduce memory usage as needed. Before: Default value: <img width="643" alt="image" src="https://github.com/user-attachments/assets/086e1173-18f3-419d-a0f5-68394f63866a" /> After: 10x: <img width="777" alt="image" src="https://github.com/user-attachments/assets/5722bbc0-0bcb-4536-b928-077031e550f1" />
4 months ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193
  1. # The type of doc engine to use.
  2. # Available options:
  3. # - `elasticsearch` (default)
  4. # - `infinity` (https://github.com/infiniflow/infinity)
  5. # - `opensearch` (https://github.com/opensearch-project/OpenSearch)
  6. DOC_ENGINE=${DOC_ENGINE:-elasticsearch}
  7. # ------------------------------
  8. # docker env var for specifying vector db type at startup
  9. # (based on the vector db type, the corresponding docker
  10. # compose profile will be used)
  11. # ------------------------------
  12. COMPOSE_PROFILES=${DOC_ENGINE}
  13. # The version of Elasticsearch.
  14. STACK_VERSION=8.11.3
  15. # The hostname where the Elasticsearch service is exposed
  16. ES_HOST=es01
  17. # The port used to expose the Elasticsearch service to the host machine,
  18. # allowing EXTERNAL access to the service running inside the Docker container.
  19. ES_PORT=1200
  20. # The password for Elasticsearch.
  21. ELASTIC_PASSWORD=infini_rag_flow
  22. # the hostname where OpenSearch service is exposed, set it not the same as elasticsearch
  23. OS_PORT=1201
  24. # The hostname where the OpenSearch service is exposed
  25. OS_HOST=opensearch01
  26. # The password for OpenSearch.
  27. # At least one uppercase letter, one lowercase letter, one digit, and one special character
  28. OPENSEARCH_PASSWORD=infini_rag_flow_OS_01
  29. # The port used to expose the Kibana service to the host machine,
  30. # allowing EXTERNAL access to the service running inside the Docker container.
  31. KIBANA_PORT=6601
  32. KIBANA_USER=rag_flow
  33. KIBANA_PASSWORD=infini_rag_flow
  34. # The maximum amount of the memory, in bytes, that a specific Docker container can use while running.
  35. # Update it according to the available memory in the host machine.
  36. MEM_LIMIT=8073741824
  37. # The hostname where the Infinity service is exposed
  38. INFINITY_HOST=infinity
  39. # Port to expose Infinity API to the host
  40. INFINITY_THRIFT_PORT=23817
  41. INFINITY_HTTP_PORT=23820
  42. INFINITY_PSQL_PORT=5432
  43. # The password for MySQL.
  44. MYSQL_PASSWORD=infini_rag_flow
  45. # The hostname where the MySQL service is exposed
  46. MYSQL_HOST=mysql
  47. # The database of the MySQL service to use
  48. MYSQL_DBNAME=rag_flow
  49. # The port used to expose the MySQL service to the host machine,
  50. # allowing EXTERNAL access to the MySQL database running inside the Docker container.
  51. MYSQL_PORT=5455
  52. # The maximum size of communication packets sent to the MySQL server
  53. MYSQL_MAX_PACKET=1073741824
  54. # The hostname where the MinIO service is exposed
  55. MINIO_HOST=minio
  56. # The port used to expose the MinIO console interface to the host machine,
  57. # allowing EXTERNAL access to the web-based console running inside the Docker container.
  58. MINIO_CONSOLE_PORT=9001
  59. # The port used to expose the MinIO API service to the host machine,
  60. # allowing EXTERNAL access to the MinIO object storage service running inside the Docker container.
  61. MINIO_PORT=9000
  62. # The username for MinIO.
  63. # When updated, you must revise the `minio.user` entry in service_conf.yaml accordingly.
  64. MINIO_USER=rag_flow
  65. # The password for MinIO.
  66. # When updated, you must revise the `minio.password` entry in service_conf.yaml accordingly.
  67. MINIO_PASSWORD=infini_rag_flow
  68. # The hostname where the Redis service is exposed
  69. REDIS_HOST=redis
  70. # The port used to expose the Redis service to the host machine,
  71. # allowing EXTERNAL access to the Redis service running inside the Docker container.
  72. REDIS_PORT=6379
  73. # The password for Redis.
  74. REDIS_PASSWORD=infini_rag_flow
  75. # The port used to expose RAGFlow's HTTP API service to the host machine,
  76. # allowing EXTERNAL access to the service running inside the Docker container.
  77. SVR_HTTP_PORT=9380
  78. # The RAGFlow Docker image to download.
  79. # Defaults to the v0.20.1-slim edition, which is the RAGFlow Docker image without embedding models.
  80. RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1-slim
  81. #
  82. # To download the RAGFlow Docker image with embedding models, uncomment the following line instead:
  83. # RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1
  84. #
  85. # The Docker image of the v0.20.1 edition includes built-in embedding models:
  86. # - BAAI/bge-large-zh-v1.5
  87. # - maidalun1020/bce-embedding-base_v1
  88. #
  89. # If you cannot download the RAGFlow Docker image:
  90. #
  91. # - For the `nightly-slim` edition, uncomment either of the following:
  92. # RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:nightly-slim
  93. # RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:nightly-slim
  94. #
  95. # - For the `nightly` edition, uncomment either of the following:
  96. # RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:nightly
  97. # RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:nightly
  98. # The local time zone.
  99. TIMEZONE='Asia/Shanghai'
  100. # Uncomment the following line if you have limited access to huggingface.co:
  101. # HF_ENDPOINT=https://hf-mirror.com
  102. # Optimizations for MacOS
  103. # Uncomment the following line if your operating system is MacOS:
  104. # MACOS=1
  105. # The maximum file size limit (in bytes) for each upload to your knowledge base or File Management.
  106. # To change the 1GB file size limit, uncomment the line below and update as needed.
  107. # MAX_CONTENT_LENGTH=1073741824
  108. # After updating, ensure `client_max_body_size` in nginx/nginx.conf is updated accordingly.
  109. # Note that neither `MAX_CONTENT_LENGTH` nor `client_max_body_size` sets the maximum size for files uploaded to an agent.
  110. # See https://ragflow.io/docs/dev/begin_component for details.
  111. # Controls how many documents are processed in a single batch.
  112. # Defaults to 4 if DOC_BULK_SIZE is not explicitly set.
  113. DOC_BULK_SIZE=${DOC_BULK_SIZE:-4}
  114. # Defines the number of items to process per batch when generating embeddings.
  115. # Defaults to 16 if EMBEDDING_BATCH_SIZE is not set in the environment.
  116. EMBEDDING_BATCH_SIZE=${EMBEDDING_BATCH_SIZE:-16}
  117. # Log level for the RAGFlow's own and imported packages.
  118. # Available levels:
  119. # - `DEBUG`
  120. # - `INFO` (default)
  121. # - `WARNING`
  122. # - `ERROR`
  123. # For example, the following line changes the log level of `ragflow.es_conn` to `DEBUG`:
  124. # LOG_LEVELS=ragflow.es_conn=DEBUG
  125. # aliyun OSS configuration
  126. # STORAGE_IMPL=OSS
  127. # ACCESS_KEY=xxx
  128. # SECRET_KEY=eee
  129. # ENDPOINT=http://oss-cn-hangzhou.aliyuncs.com
  130. # REGION=cn-hangzhou
  131. # BUCKET=ragflow65536
  132. # A user registration switch:
  133. # - Enable registration: 1
  134. # - Disable registration: 0
  135. REGISTER_ENABLED=1
  136. # Sandbox settings
  137. # Important: To enable sandbox, you must re-declare the compose profiles. See hints at the end of file.
  138. # Double check if you add `sandbox-executor-manager` to your `/etc/hosts`
  139. # Pull the required base images before running:
  140. # docker pull infiniflow/sandbox-base-nodejs:latest
  141. # docker pull infiniflow/sandbox-base-python:latest
  142. # Our default sandbox environments include:
  143. # - Node.js base image: includes axios
  144. # - Python base image: includes requests, numpy, and pandas
  145. # Specify custom executor images below if you're using non-default environments.
  146. # SANDBOX_ENABLED=1
  147. # SANDBOX_HOST=sandbox-executor-manager
  148. # SANDBOX_EXECUTOR_MANAGER_IMAGE=infiniflow/sandbox-executor-manager:latest
  149. # SANDBOX_EXECUTOR_MANAGER_POOL_SIZE=3
  150. # SANDBOX_BASE_PYTHON_IMAGE=infiniflow/sandbox-base-python:latest
  151. # SANDBOX_BASE_NODEJS_IMAGE=infiniflow/sandbox-base-nodejs:latest
  152. # SANDBOX_EXECUTOR_MANAGER_PORT=9385
  153. # SANDBOX_ENABLE_SECCOMP=false
  154. # SANDBOX_MAX_MEMORY=256m # b, k, m, g
  155. # SANDBOX_TIMEOUT=10s # s, m, 1m30s
  156. # Important: To enable sandbox, you must re-declare the compose profiles.
  157. # 1. Comment out the COMPOSE_PROFILES line above.
  158. # 2. Uncomment one of the following based on your chosen document engine:
  159. # - For Elasticsearch:
  160. # COMPOSE_PROFILES=elasticsearch,sandbox
  161. # - For Infinity:
  162. # COMPOSE_PROFILES=infinity,sandbox
  163. # - For OpenSearch:
  164. # COMPOSE_PROFILES=opensearch,sandbox