You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

docker-compose-base.yml 6.1KB

Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
6 months ago
Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
6 months ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227
  1. services:
  2. es01:
  3. container_name: ragflow-es-01
  4. profiles:
  5. - elasticsearch
  6. image: elasticsearch:${STACK_VERSION}
  7. volumes:
  8. - esdata01:/usr/share/elasticsearch/data
  9. ports:
  10. - ${ES_PORT}:9200
  11. env_file: .env
  12. environment:
  13. - node.name=es01
  14. - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
  15. - bootstrap.memory_lock=false
  16. - discovery.type=single-node
  17. - xpack.security.enabled=true
  18. - xpack.security.http.ssl.enabled=false
  19. - xpack.security.transport.ssl.enabled=false
  20. - cluster.routing.allocation.disk.watermark.low=5gb
  21. - cluster.routing.allocation.disk.watermark.high=3gb
  22. - cluster.routing.allocation.disk.watermark.flood_stage=2gb
  23. - TZ=${TIMEZONE}
  24. mem_limit: ${MEM_LIMIT}
  25. ulimits:
  26. memlock:
  27. soft: -1
  28. hard: -1
  29. healthcheck:
  30. test: ["CMD-SHELL", "curl http://localhost:9200"]
  31. interval: 10s
  32. timeout: 10s
  33. retries: 120
  34. networks:
  35. - ragflow
  36. restart: on-failure
  37. opensearch01:
  38. container_name: ragflow-opensearch-01
  39. profiles:
  40. - opensearch
  41. image: hub.icert.top/opensearchproject/opensearch:2.19.1
  42. volumes:
  43. - osdata01:/usr/share/opensearch/data
  44. ports:
  45. - ${OS_PORT}:9201
  46. env_file: .env
  47. environment:
  48. - node.name=opensearch01
  49. - OPENSEARCH_PASSWORD=${OPENSEARCH_PASSWORD}
  50. - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_PASSWORD}
  51. - bootstrap.memory_lock=false
  52. - discovery.type=single-node
  53. - plugins.security.disabled=false
  54. - plugins.security.ssl.http.enabled=false
  55. - plugins.security.ssl.transport.enabled=true
  56. - cluster.routing.allocation.disk.watermark.low=5gb
  57. - cluster.routing.allocation.disk.watermark.high=3gb
  58. - cluster.routing.allocation.disk.watermark.flood_stage=2gb
  59. - TZ=${TIMEZONE}
  60. - http.port=9201
  61. mem_limit: ${MEM_LIMIT}
  62. ulimits:
  63. memlock:
  64. soft: -1
  65. hard: -1
  66. healthcheck:
  67. test: ["CMD-SHELL", "curl http://localhost:9201"]
  68. interval: 10s
  69. timeout: 10s
  70. retries: 120
  71. networks:
  72. - ragflow
  73. restart: on-failure
  74. infinity:
  75. container_name: ragflow-infinity
  76. profiles:
  77. - infinity
  78. image: infiniflow/infinity:v0.6.0-dev5
  79. volumes:
  80. - infinity_data:/var/infinity
  81. - ./infinity_conf.toml:/infinity_conf.toml
  82. command: ["-f", "/infinity_conf.toml"]
  83. ports:
  84. - ${INFINITY_THRIFT_PORT}:23817
  85. - ${INFINITY_HTTP_PORT}:23820
  86. - ${INFINITY_PSQL_PORT}:5432
  87. env_file: .env
  88. environment:
  89. - TZ=${TIMEZONE}
  90. mem_limit: ${MEM_LIMIT}
  91. ulimits:
  92. nofile:
  93. soft: 500000
  94. hard: 500000
  95. networks:
  96. - ragflow
  97. healthcheck:
  98. test: ["CMD", "curl", "http://localhost:23820/admin/node/current"]
  99. interval: 10s
  100. timeout: 10s
  101. retries: 120
  102. restart: on-failure
  103. sandbox-executor-manager:
  104. container_name: ragflow-sandbox-executor-manager
  105. profiles:
  106. - sandbox
  107. image: ${SANDBOX_EXECUTOR_MANAGER_IMAGE-infiniflow/sandbox-executor-manager:latest}
  108. privileged: true
  109. ports:
  110. - ${SANDBOX_EXECUTOR_MANAGER_PORT-9385}:9385
  111. env_file: .env
  112. volumes:
  113. - /var/run/docker.sock:/var/run/docker.sock
  114. networks:
  115. - ragflow
  116. security_opt:
  117. - no-new-privileges:true
  118. environment:
  119. - TZ=${TIMEZONE}
  120. - SANDBOX_EXECUTOR_MANAGER_POOL_SIZE=${SANDBOX_EXECUTOR_MANAGER_POOL_SIZE:-3}
  121. - SANDBOX_BASE_PYTHON_IMAGE=${SANDBOX_BASE_PYTHON_IMAGE:-infiniflow/sandbox-base-python:latest}
  122. - SANDBOX_BASE_NODEJS_IMAGE=${SANDBOX_BASE_NODEJS_IMAGE:-infiniflow/sandbox-base-nodejs:latest}
  123. - SANDBOX_ENABLE_SECCOMP=${SANDBOX_ENABLE_SECCOMP:-false}
  124. - SANDBOX_MAX_MEMORY=${SANDBOX_MAX_MEMORY:-256m}
  125. - SANDBOX_TIMEOUT=${SANDBOX_TIMEOUT:-10s}
  126. healthcheck:
  127. test: ["CMD", "curl", "http://localhost:9385/healthz"]
  128. interval: 10s
  129. timeout: 5s
  130. retries: 5
  131. restart: on-failure
  132. mysql:
  133. # mysql:5.7 linux/arm64 image is unavailable.
  134. image: mysql:8.0.39
  135. container_name: ragflow-mysql
  136. env_file: .env
  137. environment:
  138. - MYSQL_ROOT_PASSWORD=${MYSQL_PASSWORD}
  139. - TZ=${TIMEZONE}
  140. command:
  141. --max_connections=1000
  142. --character-set-server=utf8mb4
  143. --collation-server=utf8mb4_unicode_ci
  144. --default-authentication-plugin=mysql_native_password
  145. --tls_version="TLSv1.2,TLSv1.3"
  146. --init-file /data/application/init.sql
  147. --binlog_expire_logs_seconds=604800
  148. ports:
  149. - ${MYSQL_PORT}:3306
  150. volumes:
  151. - mysql_data:/var/lib/mysql
  152. - ./init.sql:/data/application/init.sql
  153. networks:
  154. - ragflow
  155. healthcheck:
  156. test: ["CMD", "mysqladmin" ,"ping", "-uroot", "-p${MYSQL_PASSWORD}"]
  157. interval: 10s
  158. timeout: 10s
  159. retries: 3
  160. restart: on-failure
  161. minio:
  162. image: quay.io/minio/minio:RELEASE.2025-06-13T11-33-47Z
  163. container_name: ragflow-minio
  164. command: server --console-address ":9001" /data
  165. ports:
  166. - ${MINIO_PORT}:9000
  167. - ${MINIO_CONSOLE_PORT}:9001
  168. env_file: .env
  169. environment:
  170. - MINIO_ROOT_USER=${MINIO_USER}
  171. - MINIO_ROOT_PASSWORD=${MINIO_PASSWORD}
  172. - TZ=${TIMEZONE}
  173. volumes:
  174. - minio_data:/data
  175. networks:
  176. - ragflow
  177. restart: on-failure
  178. healthcheck:
  179. test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
  180. interval: 30s
  181. timeout: 20s
  182. retries: 3
  183. redis:
  184. # swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/valkey/valkey:8
  185. image: valkey/valkey:8
  186. container_name: ragflow-redis
  187. command: redis-server --requirepass ${REDIS_PASSWORD} --maxmemory 128mb --maxmemory-policy allkeys-lru
  188. env_file: .env
  189. ports:
  190. - ${REDIS_PORT}:6379
  191. volumes:
  192. - redis_data:/data
  193. networks:
  194. - ragflow
  195. restart: on-failure
  196. healthcheck:
  197. test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
  198. interval: 5s
  199. timeout: 3s
  200. retries: 3
  201. start_period: 10s
  202. volumes:
  203. esdata01:
  204. driver: local
  205. osdata01:
  206. driver: local
  207. infinity_data:
  208. driver: local
  209. mysql_data:
  210. driver: local
  211. minio_data:
  212. driver: local
  213. redis_data:
  214. driver: local
  215. networks:
  216. ragflow:
  217. driver: bridge