浏览代码

Introduced jemalloc (#5590)

### What problem does this PR solve?

Introduced jemalloc.
Python uses pymalloc (which is an reimplementation of gblibc malloc) to
manage RES. It has pools for small objects to avoid returning memory to
OS aggressively. My experience is: Replacing pymalloc with
[jemalloc](https://github.com/jemalloc/jemalloc) can reduce RES and
speedup task_executor.py.

### Type of change

- [x] Performance Improvement
tags/v0.17.1
Zhichang Yu 8 个月前
父节点
当前提交
afe9269534
没有帐户链接到提交者的电子邮件
共有 3 个文件被更改,包括 5 次插入2 次删除
  1. 1
    0
      Dockerfile
  2. 2
    1
      docker/entrypoint-parser.sh
  3. 2
    1
      docker/entrypoint.sh

+ 1
- 0
Dockerfile 查看文件

@@ -59,6 +59,7 @@ RUN --mount=type=cache,id=ragflow_apt,target=/var/cache/apt,sharing=locked \
apt install -y default-jdk && \
apt install -y libatk-bridge2.0-0 && \
apt install -y libpython3-dev libgtk-4-1 libnss3 xdg-utils libgbm-dev && \
apt install -y libjemalloc-dev && \
apt install -y python3-pip pipx nginx unzip curl wget git vim less

RUN if [ "$NEED_MIRROR" == "1" ]; then \

+ 2
- 1
docker/entrypoint-parser.sh 查看文件

@@ -15,8 +15,9 @@ CONSUMER_NO_BEG=$1
CONSUMER_NO_END=$2

function task_exe(){
JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so
while [ 1 -eq 1 ]; do
$PY rag/svr/task_executor.py $1;
LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $1;
done
}


+ 2
- 1
docker/entrypoint.sh 查看文件

@@ -17,8 +17,9 @@ if [[ -z "$WS" || $WS -lt 1 ]]; then
fi

function task_exe(){
JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so
while [ 1 -eq 1 ];do
$PY rag/svr/task_executor.py $1;
LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $1;
done
}


正在加载...
取消
保存