[Next week]✨ Knowledge Base Summary Development #1364
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # backend/pyproject.toml
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
SimengBian
reviewed
Oct 13, 2025
| @@ -0,0 +1,116 @@ | |||
| # Async Knowledge Summary Prompt Templates (Chinese) | |||
| # 异步知识库总结提示词模板（中文版） | |||
There was a problem hiding this comment.
建议注释统一用英文就可以，无需提供中文注释
SimengBian
reviewed
Oct 13, 2025
| @@ -0,0 +1,116 @@ | |||
| # Async Knowledge Summary Prompt Templates (Chinese) | |||
SimengBian
reviewed
Oct 13, 2025
| @@ -0,0 +1,115 @@ | |||
| # Async Knowledge Summary Prompt Templates (English) | |||
| # 异步知识库总结提示词模板（英文版） | |||
There was a problem hiding this comment.
同上，只需要英文注释就可以，另外，有些注释没有必要（例如下面这行注释），可以删掉
SimengBian
reviewed
Oct 13, 2025
|
|
||
| # Summary Generation Prompt | ||
| SUMMARY_GENERATION_PROMPT: |- | ||
| ### You are a【Knowledge Summary Expert】responsible for generating concise and accurate knowledge summaries. |
There was a problem hiding this comment.
建议英文提示词中不要用【这样的中文符号，不太友好，可以替换为[
SimengBian
reviewed
Oct 13, 2025
| raise Exception("Failed to get embedding model") | ||
|
|
||
| # Async summary generation stream | ||
| async def generate_summary_stream(): |
SimengBian
reviewed
Oct 13, 2025
| documents = {} | ||
|
|
||
| for chunk in chunks: | ||
| doc_id = chunk.get('filename', chunk.get('source_doc', 'unknown')) |
There was a problem hiding this comment.
建议使用or写法，更清晰：doc_id = chunk.get('filename') or chunk.get('source_doc') or 'unknown'
SimengBian
reviewed
Oct 13, 2025
|
|
||
| # Group chunks by document cluster | ||
| for chunk in chunks: | ||
| doc_id = chunk.get('filename', chunk.get('source_doc', 'unknown')) |
SimengBian
reviewed
Oct 13, 2025
| doc_vectors = await async_vectorize_batch(doc_texts, embedding_model, batch_size=20) | ||
|
|
||
| # Document-level clustering | ||
| from utils.async_knowledge_summary_utils import DocumentClusterer |
There was a problem hiding this comment.
不建议在代码中间位置进行import，建议整理，包括后面的numpy
SimengBian
reviewed
Oct 13, 2025
| chunk_clusterer = ChunkClusterer(similarity_threshold=0.70, min_cluster_size=1) | ||
| chunk_cluster_result = chunk_clusterer.cluster_chunks_with_document_clusters(chunk_vectors, chunks, chunks_by_doc_cluster) | ||
|
|
||
| n_clusters = chunk_cluster_result['n_clusters'] |
There was a problem hiding this comment.
是否要考虑chunk_cluster_result为None的情况？n_clusters = chunk_cluster_result.get('n_clusters', 0)
WMC001
reviewed
Oct 14, 2025
| # Escape quotes | ||
| yield f"data: {{\"status\": \"success\", \"message\": \"\\\"\"}}\n\n" | ||
| else: | ||
| yield f"data: {{\"status\": \"success\", \"message\": \"{char}\"}}\n\n" |
There was a problem hiding this comment.
这种处理方式太粗暴了，使用json.dumps应该能很好的处理这种json格式问题
WMC001
reviewed
Oct 14, 2025
| except Exception as e: | ||
| logger.error(f"Error in async summary generation: {e}", exc_info=True) | ||
| error_msg = str(e).replace('"', '\\"').replace('\n', '\\n') | ||
| yield f"data: {{\"status\": \"error\", \"message\": \"{error_msg}\"}}\n\n" |
There was a problem hiding this comment.
这种流式的message建议统一整改使用json.dumps
WMC001
reviewed
Oct 14, 2025
|
|
||
| response = await self.chat_async( | ||
| messages=messages, | ||
| max_tokens=max_length, # 减少token数量，强制简洁 |
WMC001
reviewed
Oct 14, 2025
| """Calculate confidence score""" | ||
| size_score = min(chunk_cluster['size'] / 10.0, 1.0) | ||
| similarity_score = chunk_cluster.get('avg_similarity', 0.0) | ||
| confidence = 0.4 * size_score + 0.6 * similarity_score |
There was a problem hiding this comment.
这种直接使用数值计算的函数建议增加注释，说明使用这种计算方式的原因，以提升代码可读性
WMC001
reviewed
Oct 14, 2025
|
|
||
| def _fallback_keyword_extraction(self, text: str) -> List[str]: | ||
| """Fallback keyword extraction""" | ||
| import re |
WMC001
reviewed
Oct 14, 2025
| import re | ||
|
|
||
| words = re.findall(r'[\u4e00-\u9fa5]+', text) | ||
| stop_words = {'的', '了', '和', '是', '在', '有', '个', '等', '与', '及'} |
WMC001
reviewed
Oct 14, 2025
| logger.warning("LLM global integration failed, using fallback strategy") | ||
| return "\n\n".join(cluster_summaries) | ||
|
|
||
| def _clean_markdown_symbols(self, text: str) -> str: |
porkpink
reviewed
Oct 14, 2025
| import re | ||
|
|
||
| words = re.findall(r'[\u4e00-\u9fa5]+', text) | ||
| stop_words = {'的', '了', '和', '是', '在', '有', '个', '等', '与', '及'} |
WMC001
reviewed
Oct 14, 2025
| formatted_text = '\n\n'.join(formatted_lines) | ||
|
|
||
| # Ensure each point has proper spacing and clear structure | ||
| import re |
porkpink
reviewed
Oct 14, 2025
| import re | ||
| # Clean up any extra spaces and ensure consistent formatting | ||
| formatted_text = re.sub(r'\n\s*\n\s*\n', '\n\n', formatted_text) # Remove excessive line breaks | ||
| formatted_text = re.sub(r'([一二三四五六七八九十]+、[^一-十\n]+)(?=\n|$)', r'\1', formatted_text) # Ensure proper ending |
WMC001
reviewed
Oct 14, 2025
| logger.error(f"Error getting stats for index {index_name}: {str(e)}") | ||
| all_stats[index_name] = {"error": str(e)} | ||
|
|
||
| return all_stats |
There was a problem hiding this comment.
这个文件看上去没有改动，是不是修改了制表符导致git识别到修改，建议恢复
WMC001
reviewed
Oct 14, 2025
| self.assertIsInstance(result, StreamingResponse) | ||
| mock_get_docs.assert_called_once() | ||
| mock_calculate_weights.assert_called_once() | ||
| mock_get_model_by_model_id.assert_called_once_with(1, "test_tenant") |
单测覆盖率不足，请补足
|
WMC001
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✨ Knowledge Base Summary Development
一个基于文档级聚类和层级总结的知识库智能总结方法
可以正确分点，完成内容正确划分，目前是内容的总结，也可以加上一些分析。
2.智能分块聚类：基于Token预估，动态决定是否需要对chunk进行二级聚类
主要函数:
│ └── async_knowledge_summary_utils.py # 核心工具类 (1000+ 行)
│ ├── AsyncLLMClient # 异步LLM客户端
│ ├── DocumentClusterer # 文档聚类器
│ ├── ChunkClusterer # Chunk聚类器
│ ├── KnowledgeIntegrator # 知识整合器
│ └── async_vectorize_batch() # 异步向量化
│
├── services/
│ └── elasticsearch_service.py # RAG流程编排
│ ├── summary_index_name() # 主流程入口
│ ├── _reconstruct_documents() # 文档重建
│ └── _organize_chunks_by_clusters() # Chunk重组