Browse Source

fix: Fix the problem that concurrent execution limit in task executor fails and causes OOM (issue#7580) (#7700)

### What problem does this PR solve?

## Cause of the bug:
During the execution process, due to improper use of trio
CapacityLimiter, the configuration parameter MAX_CONCURRENT_TASKS is
invalid, causing the executor to take out a large number of tasks from
the Redis queue at one time.

This behavior will cause the task executor to occupy too much memory and
be killed by the OS when a large number of tasks exist at the same time.
As a result, all executing tasks are suspended.

## Fix:
Added the task_manager method to the entry of /rag/svr/task_executor.py
to make CapacityLimiter effective. Deleted the invalid async with
statement.

## Fix result:
After testing, the task executor execution meets expectations, that is:
concurrent execution of up to $MAX_CONCURRENT_TASKS tasks.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
tags/v0.19.0
S0b3Rr 5 months ago
parent
commit
5d21cc3660
No account linked to committer's email address
1 changed files with 6 additions and 3 deletions
  1. 6
    3
      rag/svr/task_executor.py

+ 6
- 3
rag/svr/task_executor.py View File

@@ -713,7 +713,10 @@ def recover_pending_tasks():
redis_lock.release()
stop_event.wait(60)

async def task_manager():
global task_limiter
async with task_limiter:
await handle_task()


async def main():
@@ -742,8 +745,8 @@ async def main():
async with trio.open_nursery() as nursery:
nursery.start_soon(report_status)
while not stop_event.is_set():
async with task_limiter:
nursery.start_soon(handle_task)
nursery.start_soon(task_manager)
await trio.sleep(0.1)
logging.error("BUG!!! You should not reach here!!!")

if __name__ == "__main__":

Loading…
Cancel
Save