-
- Downloads
There was an error fetching the commit references. Please try again later.
optimize srv broker and executor logic (#630)
### What problem does this PR solve?
Optimize task broker and executor for reduce memory usage and deployment
complexity.
### Type of change
- [x] Performance Improvement
- [x] Refactoring
### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose
---------
Co-authored-by:
Kevin Hu <kevinhu.sh@gmail.com>
Showing
- .gitignore 2 additions, 0 deletions.gitignore
- api/apps/document_app.py 12 additions, 1 deletionapi/apps/document_app.py
- api/db/services/document_service.py 58 additions, 5 deletionsapi/db/services/document_service.py
- api/db/services/task_service.py 71 additions, 26 deletionsapi/db/services/task_service.py
- api/ragflow_server.py 17 additions, 0 deletionsapi/ragflow_server.py
- conf/service_conf.yaml 4 additions, 0 deletionsconf/service_conf.yaml
- docker/.env 2 additions, 0 deletionsdocker/.env
- docker/README.md 1 addition, 1 deletiondocker/README.md
- docker/docker-compose-base.yml 13 additions, 21 deletionsdocker/docker-compose-base.yml
- docker/entrypoint.sh 4 additions, 19 deletionsdocker/entrypoint.sh
- docker/service_conf.yaml 5 additions, 1 deletiondocker/service_conf.yaml
- docs/conversation_api.md 1 addition, 1 deletiondocs/conversation_api.md
- rag/llm/embedding_model.py 1 addition, 2 deletionsrag/llm/embedding_model.py
- rag/nlp/query.py 1 addition, 2 deletionsrag/nlp/query.py
- rag/settings.py 6 additions, 0 deletionsrag/settings.py
- rag/svr/task_broker.py 0 additions, 189 deletionsrag/svr/task_broker.py
- rag/svr/task_executor.py 22 additions, 29 deletionsrag/svr/task_executor.py
- rag/utils/redis_conn.py 67 additions, 2 deletionsrag/utils/redis_conn.py
- requirements.txt 1 addition, 1 deletionrequirements.txt
- requirements_dev.txt 126 additions, 0 deletionsrequirements_dev.txt
Loading
Please register or sign in to comment