Fix:After being idle for a while, new tasks need to be cancel and redo (#958)
### What problem does this PR solve?
After being idle for a while (When Redis Queue exceeds the
SVR_QUEUE_RETENTION(60*60) expiration time), new tasks need to be cancel
and redo.
When use xgroup_create to create a consumer group, set the ID to "$",
meaning that only messages added to the stream after the group is
created will be visible to new consumers. If the application scenario
requires processing messages that already exist in the queue, you might
need to change this ID to "0", so that the new consumer group can read
all messages from the beginning.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Optimize task broker and executor for reduce memory usage and deployment
complexity.
### Type of change
- [x] Performance Improvement
- [x] Refactoring
### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
CKLogic
KevinHuSh
Fakai Zhao