Feat: repair corrupted PDF files on upload automatically (#7693)
### What problem does this PR solve?
Try the best to repair corrupted PDF files on upload automatically.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix `filed_map` was incorrectly persisted. #7412
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
[BREAKING CHANGE] GET to POST: enhance document list capability (#7349)
### What problem does this PR solve?
Enhance capability of `list_docs`.
Breaking change: change method from `GET` to `POST`.
### Type of change
- [x] Refactoring
- [x] Enhancement with breaking change
Fix: when remove document do not delete the file in storage if the source is not knowledge base (#7005)
### What problem does this PR solve?
https://github.com/infiniflow/ragflow/issues/6905
When deleting a document will check before removing it from storage
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Fix: HTTP API Updates Read-Only Dataset Fields During Modification #5923 (#5937)
### What problem does this PR solve?
Fixes #5923
Fixes the readonly variables from payload at
/datasets/<dataset_id>
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
Now if user tries to modify readonly values then it will show " The
input parameters are invalid. "
invalid_keys = {"id", "embd_id", "chunk_num", "doc_num", "parser_id",
"create_date", "create_time", "created_by",
"status","token_num","update_date","update_time"}
if any(key in req for key in invalid_keys):
return get_error_data_result(message="The input parameters are
invalid.")
i have include those readonly keys in invalid_keys
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Raghav <2020csb1115@iitrpr.ac.in>
### What problem does this PR solve?
Try to reuse existing chunks. Close #3793
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Added static check at PR CI
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Move settings initialization after module init phase (#3438)
### What problem does this PR solve?
1. Module init won't connect database any more.
2. Config in settings need to be used with settings.CONFIG_NAME
### Type of change
- [x] Refactoring
Signed-off-by: jinhai <haijin.chn@gmail.com>
### What problem does this PR solve?
Integration with Infinity
- Replaced ELASTICSEARCH with dataStoreConn
- Renamed deleteByQuery with delete
- Renamed bulk to upsertBulk
- getHighlight, getAggregation
- Fix KGSearch.search
- Moved Dealer.sql_retrieval to es_conn.py
### Type of change
- [x] Refactoring
### What problem does this PR solve?
1. fix white screen issue when chat response
2. thumbnail bug when document not support
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: chongchuanbing <chongchuanbing@gmail.com>
fix: Large document thumbnail display failed (#2763)
### What problem does this PR solve?
In MySQL, when the thumbnail base64 of a document is relatively large,
the display of the document's thumbnail fails.
Now, I put the document thumbnail into MiniIO storage.
### Type of change
- [✓] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: chongchuanbing <chongchuanbing@gmail.com>
Storage: Rename all the variables about get file to storage from minio. (#2497)
https://github.com/infiniflow/ragflow/issues/2496
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Storage: Support the s3, azure blob as the object storage of ragflow. (#2278)
### What problem does this PR solve?
issue: https://github.com/infiniflow/ragflow/issues/2277
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Add Authorization checks
#2203
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Feiue <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
Format file format from Windows/dos to Unix (#1949)
### What problem does this PR solve?
Related source file is in Windows/DOS format, they are format to Unix
format.
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Supports obtaining PDF documents from web pages (#1107)
### What problem does this PR solve?
Knowledge base management supports crawling information from web pages
and generating PDF documents
### Type of change
- [x] New Feature (Support document from web pages)
let file in knowledgebases visible in file manager (#714)
### What problem does this PR solve?
Let file in knowledgebases visible in file manager.
#162
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Yongteng Lei
Stephen Hu
Zhichang Yu
Raghav Patidar
Omar Leonardo Sanchez Granados
Kevin Hu
Jin Hai
chongchuanbing
Fachuan Bai
LiuHua
H
Fakai Zhao