Delete Corresponding Minio Bucket When Deleting a Knowledge Base (#7841)
### What problem does this PR solve?
Delete Corresponding Minio Bucket When Deleting a Knowledge Base
[issue #4113 ](https://github.com/infiniflow/ragflow/issues/4113)
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Add more robust fallbacks for citations
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
change default models to buildin models
https://github.com/infiniflow/ragflow/issues/7774
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Refa: HTTP API list datasets / test cases / docs (#7720)
### What problem does this PR solve?
This PR introduces Pydantic-based validation for the list datasets HTTP
API, improving code clarity and robustness. Key changes include:
Pydantic Validation
Error Handling
Test Updates
Documentation Updates
### Type of change
- [x] Documentation Update
- [x] Refactoring
Feat: Add OAuth `state` parameter for CSRF protection (#7709)
### What problem does this PR solve?
Add OAuth `state` parameter for CSRF protection:
- Updated `get_authorization_url()` to accept an optional state
parameter
- Generated a unique state value during OAuth login and stored in
session
- Verified state parameter in callback to ensure request legitimacy
This PR follows OAuth 2.0 security best practices by ensuring that the
authorization request originates from the same user who initiated the
flow.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Fix:When you create a new API module named xxxa_api, the access route will become xxx instead of xxxa. For example, when I create a new API module named 'data_api', the access route will become 'dat' instead of 'data (#7325)
### What problem does this PR solve?
Fix:When you create a new API module named xxxa_api, the access route
will become xxx instead of xxxa. For example, when I create a new API
module named 'data_api', the access route will become 'dat' instead of
'data'
Fix:Fixed the issue where the new knowledge base would not be renamed
when there was a knowledge base with the same name
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: tangyu <1@1.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Refa: more fallbacks for bad citation format (#7710)
### What problem does this PR solve?
More fallbacks for bad citation format
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Feat: repair corrupted PDF files on upload automatically (#7693)
### What problem does this PR solve?
Try the best to repair corrupted PDF files on upload automatically.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
When creating an assistant, no dataset is specified, a different default system promt is used (#7690)
### What problem does this PR solve?
- Updated the dialog settings function to add a default prompt
configuration for no dataset.
- The prompt configuration will be determined based on the presence of
`kb_ids` in the request.
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (Non-breaking change, adding functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: wenju.li <wenju.li@deepctr.cn>
### What problem does this PR solve?
Hello, when I input a very long line in the chat input box, it will fail
with following error:
```
2025-05-17 16:11:26,004 ERROR 182558 value too long for type character varying(255)
Traceback (most recent call last):
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 3291, in execute_sql
cursor.execute(sql, params or ())
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(255)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/home/sfc/Projects/ragflow/api/apps/conversation_app.py", line 68, in set_conversation
ConversationService.save(**conv)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 3128, in inner
return fn(*args, **kwargs)
File "/var/home/sfc/Projects/ragflow/api/db/services/common_service.py", line 145, in save
return cls.save_n(**kwargs)
File "/var/home/sfc/Projects/ragflow/api/db/services/common_service.py", line 139, in save_n
sample_obj = cls.model(**kwargs).save(force_insert=True)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 6923, in save
pk = self.insert(**field_dict).execute()
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 2011, in inner
return method(self, database, *args, **kwargs)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 2082, in execute
return self._execute(database)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 2887, in _execute
return super(Insert, self)._execute(database)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 2598, in _execute
cursor = self.execute_returning(database)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 2605, in execute_returning
cursor = database.execute(self)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 3299, in execute
return self.execute_sql(sql, params)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 3289, in execute_sql
with __exception_wrapper__:
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 3059, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 192, in reraise
raise value.with_traceback(tb)
File "/var/home/sfc/Projects/ragflow/.venv/lib/python3.10/site-packages/peewee.py", line 3291, in execute_sql
cursor.execute(sql, params or ())
peewee.DataError: value too long for type character varying(255)
```
This PR fix it by truncate the `name` field in the `set_conversation`
method in the `conversation_app.py`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Feat: Support tool calling in Generate component (#7572)
### What problem does this PR solve?
Hello, our use case requires LLM agent to invoke some tools, so I made a
simple implementation here.
This PR does two things:
1. A simple plugin mechanism based on `pluginlib`:
This mechanism lives in the `plugin` directory. It will only load
plugins from `plugin/embedded_plugins` for now.
A sample plugin `bad_calculator.py` is placed in
`plugin/embedded_plugins/llm_tools`, it accepts two numbers `a` and `b`,
then give a wrong result `a + b + 100`.
In the future, it can load plugins from external location with little
code change.
Plugins are divided into different types. The only plugin type supported
in this PR is `llm_tools`, which must implement the `LLMToolPlugin`
class in the `plugin/llm_tool_plugin.py`.
More plugin types can be added in the future.
2. A tool selector in the `Generate` component:
Added a tool selector to select one or more tools for LLM:
And with the `bad_calculator` tool, it results this with the `qwen-max`
model:
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Docs: Improve oauth configuration documentation and examples (#7675)
### What problem does this PR solve?
Improve oauth configuration documentation and examples.
- Related pull requests:
- #7379
- #7553
- #7587
- Related issues:
- #3495
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
Launch sandbox from docker-compose.
#4977
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
---------
Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com>
### What problem does this PR solve?
Close #7655
Based on the codes atthe api_app, I think the reference is one-to-one
with the message
`
def fillin_conv(ans):
nonlocal conv, message_id
if not conv.reference:
conv.reference.append(ans["reference"])
else:
conv.reference[-1] = ans["reference"]
conv.message[-1] = {"role": "assistant", "content": ans["answer"], "id":
message_id}
ans["id"] = message_id
`
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Refa: HTTP API delete dataset / test cases / docs (#7657)
### What problem does this PR solve?
This PR introduces Pydantic-based validation for the delete dataset HTTP
API, improving code clarity and robustness. Key changes include:
1. Pydantic Validation
2. Error Handling
3. Test Updates
4. Documentation Updates
### Type of change
- [x] Documentation Update
- [x] Refactoring
Refa: Deprecate `/github_callback` in favor of `/oauth/callback/<channel>` for GitHub OAuth integration (#7587)
### What problem does this PR solve?
Deprecate `/github_callback` route in favor of
`/oauth/callback/<channel>` for GitHub OAuth integration:
- Added GitHub OAuth support in the authentication module
- Introduced `GithubOAuthClient` with methods to fetch and normalize
user info
- Updated `CLIENT_TYPES` to include GitHub OAuth client
- Deprecated `/github_callback` route and suggested using the generic
`/oauth/callback/<channel>` route
---
- Related pull requests:
- #7379
- #7553
### Usage
- [Create a GitHub OAuth
App](https://github.com/settings/applications/new) to obtain the
`client_id` and `client_secret`, configure the authorization callback
url: `https://your-app.com/v1/user/oauth/callback/github`
- Edit `service_conf.yaml.template`:
```yaml
# ...
oauth:
github:
type: "github"
icon: "github"
display_name: "Github"
client_id: "your_client_id"
client_secret: "your_client_secret"
redirect_uri: "https://your-app.com/v1/user/oauth/callback/github"
# ...
```
### Type of change
- [x] Documentation Update
- [x] Refactoring (non-breaking change)
Feat: When Delete Chunk Will Also Delete Chunk Related Image (#7656)
### What problem does this PR solve?
When Delete Chunk Will Also Delete Chunk Related Image
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Fix(api): correct default value handling in dataset parser config (#7589)
### What problem does this PR solve?
Fix HTTP API Create/Update dataset parser config default value error
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Feat: Add http api to create, update, or delete agents. (#7515)
### What problem does this PR solve?
Hello, we are using ragflow as a backend service, so we need to manage
agents from our own frontend. So adding these http APIs to manage
agents.
The code logic is copied and modified from the `rm` and `save` methods
in `api/apps/canvas_app.py`.
btw, I found that the `save` method in `canvas_app.py` actually allows
to modify an agent to an existing title, so I kept the behavior in the
http api. I'm not sure if this is intentional.
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Refa: HTTP API update dataset / test cases / docs (#7564)
### What problem does this PR solve?
This PR introduces Pydantic-based validation for the update dataset HTTP
API, improving code clarity and robustness. Key changes include:
1. Pydantic Validation
2. Error Handling
3. Test Updates
4. Documentation Updates
5. fix bug: #5915
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
- [x] Refactoring
Perf: Increase database connection pool size (#7559)
### What problem does this PR solve?
1. The MySQL instance is configured with max_connections=1000,
but our connection pool was limited to max_connections: 100.
This mismatch caused connection pool exhaustion during performance
testing.
2. Increase stale_timeout to resolve #6548
### Type of change
- [x] Performance Improvement
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Feat: KB detail supports document total size (#7546)
### What problem does this PR solve?
Kb detail supports return document total size now.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
https://github.com/infiniflow/ragflow/issues/7519
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Feat: Add `/login/channels` route and improve auth logic for frontend third-party login integration (#7521)
### What problem does this PR solve?
Add `/login/channels` route and improve auth logic to support frontend
integration with third-party login providers:
- Add `/login/channels` route to provide authentication channel list
with `display_name` and `icon`
- Optimize user info parsing logic by prioritizing `avatar_url` and
falling back to `picture`
- Simplify OIDC token validation by removing unnecessary `kid` checks
- Ensure `client_id` is safely cast to string during `audience`
validation
- Fix typo
---
- Related pull request: #7379
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
Fix:When sharing the knowledge base of multiple tenants with one person, when this person queries the knowledge base of both tenants, they will only query the question of the first person's knowledge base (#7500)
Fix:When sharing the knowledge base of multiple tenants with one person,
when this person queries the knowledge base of both tenants, they will
only query the question of the first person's knowledge base
Co-authored-by: 杜有强 <duyq@internal.ths.com.cn>
Feat: Improve 'user_canvan_version' delete and 'document' delete performance (#6553)
### What problem does this PR solve?
1. Add delete_by_ids method
2. Add get_doc_ids_by_doc_names
3. Improve user_canvan_version's logic (avoid O(n) db IO)
4. Improve document delete logic (avoid O(n) db IO)
### Type of change
- [x] Performance Improvement
fix: After the file is deleted, it still remains in the bucket. (#7482)
### What problem does this PR solve?
Fix: After deleting the file from the file management menu, it was not
removed from the MinIO bucket.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: liuzhenghua-jk <liuzhenghua-jk@360shuke.com>
### What problem does this PR solve?
Fix `filed_map` was incorrectly persisted. #7412
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Fix: change create dataset htto api delimiter default value to r'\n' (#7434)
### What problem does this PR solve?
change create dataset delimiter default value to r'\n'
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Fix: Tried to fix the fid mis match under some cases (#7426)
### What problem does this PR solve?
https://github.com/infiniflow/ragflow/issues/7407
Based on this context, I think there should be some reasons that let
some LLMs have a mismatch (add the wrong "@xxx"),
So I think when use fid can not fetch llm then tried to just use name
should can fetch it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Remove unnecessary parameter restrictions in dataset creation API
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Fix: whole knowledge graph lost after removing any document in the knowledge base (#7151)
### What problem does this PR solve?
When you removed any document in a knowledge base using knowledge graph,
the graph's `removed_kwd` is set to "Y".
However, in the function `graphrag.utils.get_gaph`, `rebuild_graph`
method is passed and directly return `None` while `removed_kwd=Y`,
making residual part of the graph abandoned (but old entity data still
exist in db).
Besides, infinity instance actually pass deleting graph components'
`source_id` when removing document. It may cause wrong graph after
rebuild.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Refa: http API create dataset and test cases (#7393)
### What problem does this PR solve?
This PR introduces Pydantic-based validation for the create dataset HTTP
API, improving code clarity and robustness. Key changes include:
1. Pydantic Validation
2. Error Handling
3. Test Updates
4. Documentation
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
- [x] Refactoring
Feat: Add support for OAuth2 and OpenID Connect (OIDC) authentication (#7379)
### What problem does this PR solve?
Add support for OAuth2 and OpenID Connect (OIDC) authentication,
allowing OAuth/OIDC authentication using the specified routes:
- `/login/<channel>`: Initiates the OAuth flow for the specified channel
- `/oauth/callback/<channel>`: Handles the OAuth callback after
successful authentication
The callback URL should be configured in your OAuth provider as:
```
https://your-app.com/oauth/callback/<channel>
```
For detailed instructions on configuring **service_conf.yaml.template**,
see: `./api/apps/auth/README.md#usage`.
- Related issues
#3495
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
Fix:Update chat assistant with an empty dataset (#7354)
### What problem does this PR solve?
When updating a chat assistant using API,if the dataset attached by the
current chat assistant is not empty，setting dataset to
null("dataset_ids":[]) will cause update failure：'dataset_ids' can't be
empty
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
In the generate_confirmation_token method, a spelling error was found
with 'tenent_id'. The correct spelling should be 'tenant_id'.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: shengliang xiao <shengliangxiao2024@gmail.com>
Kevin Hu
Hao Zhang
Yongteng Lei
liu an
Chaoxi Weng
donblack01
liwenju0
Song Fuchang
Stephen Hu
so95
zhanglei
duramisu
liuzhenghua
alkscr
Qi
xiaosl-cell