Merge pull request #33 from xerrors/dev

Dev 修复图谱bug
This commit is contained in:
Wenjie Zhang 2025-02-28 02:48:42 +08:00 committed by GitHub
commit 56444740b9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
30 changed files with 378 additions and 359 deletions

View File

@ -10,7 +10,6 @@
</div> </div>
## 概述 ## 概述
基于大模型 RAG 知识库与知识图谱的问答平台。Llamaindex + VueJS + Flask + Neo4j。大模型适配 OpenAI、国内主流大模型平台的模型调用、本地 vllm 部署。只需要配置对应服务平台的 `API_KEY` 即可使用。 基于大模型 RAG 知识库与知识图谱的问答平台。Llamaindex + VueJS + Flask + Neo4j。大模型适配 OpenAI、国内主流大模型平台的模型调用、本地 vllm 部署。只需要配置对应服务平台的 `API_KEY` 即可使用。
@ -18,11 +17,12 @@
![image](https://github.com/user-attachments/assets/75010511-4ac5-4924-8268-fea9a589839c) ![image](https://github.com/user-attachments/assets/75010511-4ac5-4924-8268-fea9a589839c)
办清单 办清单
- [ ] Ollma Embedding 支持Open-like Embedding 支持) - [ ] Ollma Embedding 支持Open-like Embedding 支持)
- [x] 知识图谱索引支持自定义 Embedding 模型 - [ ] embedding 过程添加进度显示
- [x] DeepSeek-R1 支持 - [X] 知识图谱索引支持自定义 Embedding 模型
- [X] DeepSeek-R1 支持
## 更新日志 ## 更新日志
@ -39,7 +39,6 @@
## 快速上手 ## 快速上手
在启动之前,提供 API 服务商的 API_KEY并放置在 `src/.env` 文件中。默认使用的是硅基流动。因此务必需要配置 `SILICONFLOW_API_KEY=<SILICONFLOW_API_KEY>`。其余模型的配置可以参考 [src/static/models.yaml](src/static/models.yaml) 中的 env。 在启动之前,提供 API 服务商的 API_KEY并放置在 `src/.env` 文件中。默认使用的是硅基流动。因此务必需要配置 `SILICONFLOW_API_KEY=<SILICONFLOW_API_KEY>`。其余模型的配置可以参考 [src/static/models.yaml](src/static/models.yaml) 中的 env。
@ -103,15 +102,15 @@ docker compose -f docker/docker-compose.yml --env-file src/.env up --build
模型仅支持通过API调用的模型如果是需要运行本地模型则建议使用 vllm 转成 API 服务之后使用。使用前请在 `.env` 配置 APIKEY 后使用,配置项目参考:[src/static/models.yaml](src/static/models.yaml) 模型仅支持通过API调用的模型如果是需要运行本地模型则建议使用 vllm 转成 API 服务之后使用。使用前请在 `.env` 配置 APIKEY 后使用,配置项目参考:[src/static/models.yaml](src/static/models.yaml)
| 模型供应商 | 默认模型 | 配置项目 | | 模型供应商 | 默认模型 | 配置项目 |
| :-------------------- | :---------------------------------------- | :--------------------------------------------- | | :--------------------- | :---------------------------------- | :--------------------------------------------- |
| `siliconflow` (默认) | `Qwen/Qwen2.5-7B-Instruct` (免费) | `SILICONFLOW_API_KEY` | | `siliconflow` (默认) | `Qwen/Qwen2.5-7B-Instruct` (免费) | `SILICONFLOW_API_KEY` |
| `openai` | `gpt-4o` | `OPENAI_API_KEY` | | `openai` | `gpt-4o` | `OPENAI_API_KEY` |
| `deepseek` | `deepseek-chat` | `DEEPSEEK_API_KEY` | | `deepseek` | `deepseek-chat` | `DEEPSEEK_API_KEY` |
| `arc`(豆包方舟) | `doubao-1-5-pro-32k-250115` | `ARK_API_KEY` | | `arc`(豆包方舟) | `doubao-1-5-pro-32k-250115` | `ARK_API_KEY` |
| `zhipu`(智谱清言) | `glm-4-flash` | `ZHIPUAI_API_KEY` | | `zhipu`(智谱清言) | `glm-4-flash` | `ZHIPUAI_API_KEY` |
| `dashscope`(阿里) | `qwen-max-latest` | `DASHSCOPE_API_KEY` | | `dashscope`(阿里) | `qwen-max-latest` | `DASHSCOPE_API_KEY` |
| `qianfan`(百度) | `ernie_speed` | `QIANFAN_ACCESS_KEY`, `QIANFAN_SECRET_KEY` | | `qianfan`(百度) | `ernie_speed` | `QIANFAN_ACCESS_KEY`, `QIANFAN_SECRET_KEY` |
此外,如果想要添加供应商的模型,确认知识 OpenAI 调用的方法之后,只需要在 [src/static/models.yaml](src/static/models.yaml) 中添加对应的模型配置即可。配置示例如下: 此外,如果想要添加供应商的模型,确认知识 OpenAI 调用的方法之后,只需要在 [src/static/models.yaml](src/static/models.yaml) 中添加对应的模型配置即可。配置示例如下:
@ -140,14 +139,12 @@ docker compose -f docker/docker-compose.yml --env-file src/.env up --build
> [!Warning] > [!Warning]
> 需要注意,由于知识库和图数据库的构建都依赖于向量模型,如果中途更改向量模型,会导致知识库不可用。此外,知识图谱的向量索引的建立默认使用 embedding-3 构建,因此检索的时候必须使用 embedding-3现阶段还不支持修改 > 需要注意,由于知识库和图数据库的构建都依赖于向量模型,如果中途更改向量模型,会导致知识库不可用。此外,知识图谱的向量索引的建立默认使用 embedding-3 构建,因此检索的时候必须使用 embedding-3现阶段还不支持修改
对于**语言模型**,并不支持直接运行本地语言模型,请使用 vllm 或者 ollama 转成 API 服务之后使用。 对于**语言模型**,并不支持直接运行本地语言模型,请使用 vllm 或者 ollama 转成 API 服务之后使用。
对于**向量模型**和**重排序模型**,选择以 `local` 前缀开头的模型,可以不做修改会自动下载模型,如果下载过程中出现问题,请参考 [HF-Mirror](https://hf-mirror.com/) 配置相关内容。(但请注意,如果是 Docker 运行,模型仅会缓存到 Docker 里面) 对于**向量模型**和**重排序模型**,选择以 `local` 前缀开头的模型,可以不做修改会自动下载模型,如果下载过程中出现问题,请参考 [HF-Mirror](https://hf-mirror.com/) 配置相关内容。(但请注意,如果是 Docker 运行,模型仅会缓存到 Docker 里面)
如果想要使用本地已经下载好的模型,可以在网页的 settings 里面做映射。或者修改 `saves/config/base.yaml` 来配置映射关系。但请记得,本地模型的路径要在 docker-compose 的文件中映射 volumes。 如果想要使用本地已经下载好的模型,可以在网页的 settings 里面做映射。或者修改 `saves/config/base.yaml` 来配置映射关系。但请记得,本地模型的路径要在 docker-compose 的文件中映射 volumes。
## 知识库支持 ## 知识库支持
本项目支持多种格式的知识库目前支持的格式有pdf、txt、md。文件上传后会首先转换为纯文本然后使用向量模型将文本转换为向量然后存储到向量数据库中。这个过程可能会花费不少时间。 本项目支持多种格式的知识库目前支持的格式有pdf、txt、md。文件上传后会首先转换为纯文本然后使用向量模型将文本转换为向量然后存储到向量数据库中。这个过程可能会花费不少时间。

View File

@ -6,8 +6,8 @@ WORKDIR /app
COPY ./web/package*.json ./ COPY ./web/package*.json ./
# 安装依赖 # 安装依赖
RUN npm install --verbose --force # RUN npm install --verbose --force
# RUN npm install --registry http://mirrors.cloud.tencent.com/npm/ --verbose --force RUN npm install --registry http://mirrors.cloud.tencent.com/npm/ --verbose --force
# 复制源代码 # 复制源代码
COPY ./web . COPY ./web .
@ -22,8 +22,8 @@ FROM node:latest AS build-stage
WORKDIR /app WORKDIR /app
COPY ./web/package*.json ./ COPY ./web/package*.json ./
RUN npm install --force # RUN npm install --force
# RUN npm install --registry https://registry.npmmirror.com --force RUN npm install --registry https://registry.npmmirror.com --force
COPY ./web . COPY ./web .
RUN npm run build RUN npm run build

View File

@ -2,11 +2,9 @@ import os
import json import json
import yaml import yaml
from pathlib import Path from pathlib import Path
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
logger = setup_logger("Config") with open(Path("src/static/models.yaml"), 'r', encoding='utf-8') as f:
with open(Path("src/static/models.yaml"), "r") as f:
_models = yaml.safe_load(f) _models = yaml.safe_load(f)
MODEL_NAMES = _models["MODEL_NAMES"] MODEL_NAMES = _models["MODEL_NAMES"]
@ -53,7 +51,6 @@ class Config(SimpleConfig):
self.add_item("enable_reranker", default=False, des="是否开启重排序") self.add_item("enable_reranker", default=False, des="是否开启重排序")
self.add_item("enable_knowledge_base", default=False, des="是否开启知识库") self.add_item("enable_knowledge_base", default=False, des="是否开启知识库")
self.add_item("enable_knowledge_graph", default=False, des="是否开启知识图谱") self.add_item("enable_knowledge_graph", default=False, des="是否开启知识图谱")
self.add_item("enable_search_engine", default=False, des="是否开启搜索引擎")
self.add_item("enable_web_search", default=False, des="是否开启网页搜索(需配置 TAVILY_API_KEY") self.add_item("enable_web_search", default=False, des="是否开启网页搜索(需配置 TAVILY_API_KEY")
# 模型配置 # 模型配置
## 注意这里是模型名,而不是具体的模型路径,默认使用 HuggingFace 的路径 ## 注意这里是模型名,而不是具体的模型路径,默认使用 HuggingFace 的路径
@ -90,6 +87,9 @@ class Config(SimpleConfig):
def handle_self(self): def handle_self(self):
self.model_names = MODEL_NAMES self.model_names = MODEL_NAMES
self.embed_model_names = EMBED_MODEL_INFO
self.reranker_names = RERANKER_LIST
model_provider_info = self.model_names.get(self.model_provider, {}) model_provider_info = self.model_names.get(self.model_provider, {})
self.model_dir = os.environ.get("MODEL_DIR", "") self.model_dir = os.environ.get("MODEL_DIR", "")

View File

@ -2,11 +2,9 @@ import os
import json import json
import time import time
from src.plugins import pdf2txt from src.plugins import pdf2txt
from src.utils import hashstr, setup_logger, is_text_pdf from src.utils import hashstr, logger, is_text_pdf
from src.models.embedding import get_embedding_model from src.models.embedding import get_embedding_model
logger = setup_logger("DataBaseManager")
class DataBaseManager: class DataBaseManager:
@ -21,7 +19,6 @@ class DataBaseManager:
if self.config.enable_knowledge_graph: if self.config.enable_knowledge_graph:
from src.core.graphbase import GraphDatabase from src.core.graphbase import GraphDatabase
self.graph_base = GraphDatabase(self.config, self.embed_model) self.graph_base = GraphDatabase(self.config, self.embed_model)
self.graph_base.start()
else: else:
self.graph_base = None self.graph_base = None

View File

@ -9,9 +9,7 @@ import warnings
from src.plugins import pdf2txt from src.plugins import pdf2txt
from src.plugins.oneke import OneKE from src.plugins.oneke import OneKE
from src.utils import setup_logger from src.utils import logger
logger = setup_logger("server-graphbase")
warnings.filterwarnings("ignore", category=UserWarning) warnings.filterwarnings("ignore", category=UserWarning)
@ -28,6 +26,16 @@ class GraphDatabase:
assert embed_model, "embed_model=None" assert embed_model, "embed_model=None"
self.embed_model = embed_model self.embed_model = embed_model
self.embed_model_name = None self.embed_model_name = None
self.work_dir = os.path.join(config.save_dir, "knowledge_graph", kgdb_name)
os.makedirs(self.work_dir, exist_ok=True)
# 尝试加载已保存的图数据库信息
if self.load_graph_info():
logger.info(f"已加载图数据库信息:{self.kgdb_name}")
else:
logger.info(f"未找到已保存的图数据库信息,将创建新的配置")
self.start()
def start(self): def start(self):
uri = os.environ.get("NEO4J_URI", "bolt://localhost:7687") uri = os.environ.get("NEO4J_URI", "bolt://localhost:7687")
@ -38,11 +46,12 @@ class GraphDatabase:
self.driver = GD.driver(f"{uri}/{self.kgdb_name}", auth=(username, password)) self.driver = GD.driver(f"{uri}/{self.kgdb_name}", auth=(username, password))
self.status = "open" self.status = "open"
logger.info(f"Connected to Neo4j at {uri}/{self.kgdb_name}, {self.get_database_info()}") logger.info(f"Connected to Neo4j at {uri}/{self.kgdb_name}, {self.get_database_info()}")
# 连接成功后保存图数据库信息
self.save_graph_info()
except Exception as e: except Exception as e:
logger.error(f"Failed to connect to Neo4j: {e}, {uri}, {self.kgdb_name}, {username}, {password}") logger.error(f"Failed to connect to Neo4j: {e}, {uri}, {self.kgdb_name}, {username}, {password}")
self.config.enable_knowledge_graph = False self.config.enable_knowledge_graph = False
def close(self): def close(self):
"""关闭数据库连接""" """关闭数据库连接"""
self.driver.close() self.driver.close()
@ -119,33 +128,39 @@ class GraphDatabase:
with self.driver.session() as session: with self.driver.session() as session:
session.execute_write(create, triples) session.execute_write(create, triples)
def pdf_file_add_entity(self, file_path, output_path, kgdb_name='neo4j'): # def pdf_file_add_entity(self, file_path, output_path, kgdb_name='neo4j'):
self.use_database(kgdb_name) # 切换到指定数据库 # self.use_database(kgdb_name) # 切换到指定数据库
text_path = pdf2txt(file_path) # text_path = pdf2txt(file_path)
global UIE_MODEL # global UIE_MODEL
if UIE_MODEL is None: # if UIE_MODEL is None:
UIE_MODEL = OneKE() # UIE_MODEL = OneKE()
triples_path = UIE_MODEL.processing_text_to_kg(text_path, output_path) # triples_path = UIE_MODEL.processing_text_to_kg(text_path, output_path)
self.jsonl_file_add_entity(triples_path) # self.jsonl_file_add_entity(triples_path)
return kgdb_name # return kgdb_name
def txt_add_vector_entity(self, triples, kgdb_name='neo4j'): def txt_add_vector_entity(self, triples, kgdb_name='neo4j'):
"""添加实体三元组""" """添加实体三元组"""
self.use_database(kgdb_name) self.use_database(kgdb_name)
def _index_exists(tx, index_name): def _index_exists(tx, index_name):
"""检查索引是否存在"""
result = tx.run("SHOW INDEXES") result = tx.run("SHOW INDEXES")
for record in result: for record in result:
if record["name"] == index_name: if record["name"] == index_name:
return True return True
return False return False
def _create_graph(tx, data): def _create_graph(tx, data):
"""添加一个三元组"""
for entry in data: for entry in data:
tx.run(""" tx.run("""
MERGE (h:Entity {name: $h}) MERGE (h:Entity {name: $h})
MERGE (t:Entity {name: $t}) MERGE (t:Entity {name: $t})
MERGE (h)-[r:RELATION {type: $r}]->(t) MERGE (h)-[r:RELATION {type: $r}]->(t)
""", h=entry['h'], t=entry['t'], r=entry['r']) """, h=entry['h'], t=entry['t'], r=entry['r'])
def _create_vector_index(tx, dim): def _create_vector_index(tx, dim):
"""创建向量索引"""
# NOTE 这里是否是会重复构建索引?
index_name = "entityEmbeddings" index_name = "entityEmbeddings"
if not _index_exists(tx, index_name): if not _index_exists(tx, index_name):
tx.run(f""" tx.run(f"""
@ -157,24 +172,34 @@ class GraphDatabase:
}} }}; }} }};
""") """)
# 判断模型名称是否匹配
from src.config import EMBED_MODEL_INFO from src.config import EMBED_MODEL_INFO
embed_info = EMBED_MODEL_INFO[self.config.embed_model] cur_embed_info = EMBED_MODEL_INFO[self.config.embed_model]
with self.driver.session() as session: self.embed_model_name = self.embed_model_name or cur_embed_info.get('name')
session.execute_write(_create_graph, triples) assert self.embed_model_name == cur_embed_info.get('name') or self.embed_model_name is None, \
session.execute_write(_create_vector_index, embed_info.get('dimension')) f"embed_model_name={self.embed_model_name}, {cur_embed_info.get('name')=}"
for i, entry in enumerate(triples):
logger.info(f"Adding entity {i+1}/{len(triples)}")
embedding_h = self.get_embedding(entry['h'])
session.execute_write(self.set_embedding, entry['h'], embedding_h)
with self.driver.session() as session:
logger.info(f"Adding entity to {kgdb_name}")
session.execute_write(_create_graph, triples)
logger.info(f"Creating vector index for {kgdb_name} with {self.config.embed_model}")
session.execute_write(_create_vector_index, cur_embed_info['dimension'])
# NOTE 这里需要异步处理
for i, entry in enumerate(triples):
logger.debug(f"Adding entity {i+1}/{len(triples)}")
embedding_h = self.get_embedding(entry['h'])
embedding_t = self.get_embedding(entry['t']) embedding_t = self.get_embedding(entry['t'])
session.execute_write(self.set_embedding, entry['h'], embedding_h)
session.execute_write(self.set_embedding, entry['t'], embedding_t) session.execute_write(self.set_embedding, entry['t'], embedding_t)
# 数据添加完成后保存图信息
self.save_graph_info()
def jsonl_file_add_entity(self, file_path, kgdb_name='neo4j'): def jsonl_file_add_entity(self, file_path, kgdb_name='neo4j'):
self.status = "processing" self.status = "processing"
kgdb_name = kgdb_name or 'neo4j' kgdb_name = kgdb_name or 'neo4j'
self.embed_model_name = self.embed_model_name or self.config.embed_model
self.use_database(kgdb_name) # 切换到指定数据库 self.use_database(kgdb_name) # 切换到指定数据库
logger.info(f"Start adding entity to {kgdb_name} with {file_path}")
def read_triples(file_path): def read_triples(file_path):
with open(file_path, 'r', encoding='utf-8') as file: with open(file_path, 'r', encoding='utf-8') as file:
@ -186,6 +211,8 @@ class GraphDatabase:
self.txt_add_vector_entity(triples, kgdb_name) self.txt_add_vector_entity(triples, kgdb_name)
self.status = "open" self.status = "open"
# 更新并保存图数据库信息
self.save_graph_info()
return kgdb_name return kgdb_name
def delete_entity(self, entity_name=None, kgdb_name="neo4j"): def delete_entity(self, entity_name=None, kgdb_name="neo4j"):
@ -213,27 +240,14 @@ class GraphDatabase:
def query_node(self, entity_name, hops=2, **kwargs): def query_node(self, entity_name, hops=2, **kwargs):
# TODO 添加判断节点数量为 0 停止检索 # TODO 添加判断节点数量为 0 停止检索
logger.debug(f"Query graph node {entity_name} with {hops=}")
if kwargs.get("exact_match"): if kwargs.get("exact_match"):
raise NotImplemented("not implement for `exact_match`") raise NotImplemented("not implement for `exact_match`")
else: else:
return self.query_by_vector(entity_name=entity_name, **kwargs) return self.query_by_vector(entity_name=entity_name, **kwargs)
def query_by_vector(self, entity_name, threshold=0.9, kgdb_name='neo4j', hops=2, num_of_res=5): def query_by_vector(self, entity_name, threshold=0.9, kgdb_name='neo4j', hops=2, num_of_res=5):
results = self.query_by_vector_tep(entity_name=entity_name)
# 筛选出分数高于阈值的实体
qualified_entities = [result[0] for result in results[:num_of_res] if result[1] > threshold]
# 对每个合格的实体进行查询
all_query_results = []
for entity in qualified_entities:
query_result = self.query_specific_entity(entity_name=entity, hops=hops, kgdb_name=kgdb_name)
all_query_results.extend(query_result)
return all_query_results
def query_by_vector_tep(self, entity_name, kgdb_name='neo4j'):
"""向量查询"""
self.use_database(kgdb_name) self.use_database(kgdb_name)
def query(tx, text): def query(tx, text):
embedding = self.get_embedding(text) embedding = self.get_embedding(text)
@ -245,15 +259,28 @@ class GraphDatabase:
return result.values() return result.values()
with self.driver.session() as session: with self.driver.session() as session:
return session.execute_read(query, entity_name) results = session.execute_read(query, entity_name)
# 筛选出分数高于阈值的实体
qualified_entities = [result[0] for result in results[:num_of_res] if result[1] > threshold]
logger.debug(f"Graph Query Entities: {entity_name}, {qualified_entities=}")
# 对每个合格的实体进行查询
all_query_results = []
for entity in qualified_entities:
query_result = self.query_specific_entity(entity_name=entity, hops=hops, kgdb_name=kgdb_name)
all_query_results.extend(query_result)
return all_query_results
def query_specific_entity(self, entity_name, kgdb_name='neo4j', hops=2): def query_specific_entity(self, entity_name, kgdb_name='neo4j', hops=2):
"""查询指定实体三元组信息""" """查询指定实体三元组信息(无向关系)"""
self.use_database(kgdb_name) self.use_database(kgdb_name)
def query(tx, entity_name, hops): def query(tx, entity_name, hops):
result = tx.run(f""" result = tx.run(f"""
MATCH (n {{name: $entity_name}})-[r*1..{hops}]->(m) MATCH (n {{name: $entity_name}})-[r*1..{hops}]-(m)
RETURN n.name AS node_name, r, m.name AS neighbor_name RETURN n, r, m
""", entity_name=entity_name) """, entity_name=entity_name)
return result.values() return result.values()
@ -316,11 +343,9 @@ class GraphDatabase:
return session.execute_read(query, node_name, hops) return session.execute_read(query, node_name, hops)
def get_embedding(self, text): def get_embedding(self, text):
inputs = [text]
with torch.no_grad(): with torch.no_grad():
outputs = self.embed_model.encode(inputs) outputs = self.embed_model.encode([text])[0]
embeddings = outputs[0] # 假设取平均作为文本的嵌入向量 return outputs
return embeddings
def set_embedding(self, tx, entity_name, embedding): def set_embedding(self, tx, entity_name, embedding):
tx.run(""" tx.run("""
@ -328,160 +353,71 @@ class GraphDatabase:
CALL db.create.setNodeVectorProperty(e, 'embedding', $embedding) CALL db.create.setNodeVectorProperty(e, 'embedding', $embedding)
""", name=entity_name, embedding=embedding) """, name=entity_name, embedding=embedding)
# def format_query_results(self, results): def save_graph_info(self):
# formatted_results = [] """
# for row in results: 将图数据库的基本信息保存到工作目录中的JSON文件
# n, rs, m = row 保存的信息包括数据库名称状态嵌入模型名称等
# entity_a = n['name'] """
# entity_b = m['name'] try:
# for rel in rs: # 获取数据库信息
# relationship = rel.type db_info = None
# formatted_results.append(f"实体 {entity_a} 和 实体 {entity_b} 的关系是 {relationship}") if self.status == "open" and self.driver:
# return formatted_results try:
db_info = self.get_database_info(self.kgdb_name)
except Exception as e:
logger.warning(f"无法获取数据库信息:{e}")
# 构建要保存的信息字典
graph_info = {
"kgdb_name": self.kgdb_name,
"status": self.status,
"embed_model_name": self.embed_model_name,
"last_updated": None, # 这里可以添加时间戳
"database_info": db_info
}
# 添加时间戳
from datetime import datetime
graph_info["last_updated"] = datetime.now().isoformat()
# 保存到JSON文件
info_file_path = os.path.join(self.work_dir, "graph_info.json")
with open(info_file_path, 'w', encoding='utf-8') as f:
json.dump(graph_info, f, ensure_ascii=False, indent=2)
logger.info(f"图数据库信息已保存到:{info_file_path}")
return True
except Exception as e:
logger.error(f"保存图数据库信息失败:{e}")
return False
def load_graph_info(self):
"""
从工作目录中的JSON文件加载图数据库的基本信息
返回True表示加载成功False表示加载失败
"""
try:
info_file_path = os.path.join(self.work_dir, "graph_info.json")
if not os.path.exists(info_file_path):
logger.warning(f"图数据库信息文件不存在:{info_file_path}")
return False
with open(info_file_path, 'r', encoding='utf-8') as f:
graph_info = json.load(f)
# 更新对象属性
if graph_info.get("embed_model_name"):
self.embed_model_name = graph_info["embed_model_name"]
# 如果需要,可以加载更多信息
# 注意这里不更新self.kgdb_name因为它是在初始化时设置的
logger.info(f"已加载图数据库信息,最后更新时间:{graph_info.get('last_updated')}")
return True
except Exception as e:
logger.error(f"加载图数据库信息失败:{e}")
return False
if __name__ == "__main__": if __name__ == "__main__":
config = None pass
kgdb_name = "neo4j"
class EmbeddingModel(FlagModel):
def __init__(self, config, **kwargs):
model_name_or_path = "/data2024/yyyl/model/BAAI/bge-large-zh-v1.5/"
super().__init__(model_name_or_path, use_fp16=False, **kwargs)
model = EmbeddingModel(config)
# 初始化知识图谱数据库
kgdb = GraphDatabase(config, model)
# 创建新的数据库
# kgdb.create_graph_database("db2")
# 返回指定数据库信息
# info = kgdb.get_database_info(kgdb_name)
# print(info)
# triples = [
# {
# "h": "CCC",
# "t": "EE",
# "r": "同学"
# },
# {
# "h": "EE",
# "t": "RR",
# "r": "同事"
# }
# ]
# kgdb.query_by_vector("z")
# def format_query_results(results):
# formatted_results = {"nodes": [], "edges": []}
# # 用于存储所有唯一的节点信息
# node_dict = {}
# for item in results:
# # 确保item[1]是一个非空的列表
# if isinstance(item[1], list) and len(item[1]) > 0:
# relationship = item[1][0]
# rel_id = relationship.element_id
# nodes = relationship.nodes
# if len(nodes) == 2:
# node1, node2 = nodes
# # 提取源节点和目标节点信息
# node1_id = node1.element_id
# node2_id = node2.element_id
# node1_name = item[0] # 假设节点名称和列表中的第一个元素相同
# node2_name = item[2] if len(item) > 2 else 'unknown'
# # 记录节点信息
# if node1_id not in node_dict:
# node_dict[node1_id] = {"id": node1_id, "name": node1_name}
# if node2_id not in node_dict:
# node_dict[node2_id] = {"id": node2_id, "name": node2_name}
# # 确定关系类型
# relationship_type = relationship._properties.get('type', 'unknown')
# if relationship_type == 'unknown':
# relationship_type = relationship.type
# # 记录边的信息
# formatted_results["edges"].append({
# "id": rel_id,
# "type": relationship_type,
# "source_id": node1_id,
# "target_id": node2_id,
# "source_name": node1_name,
# "target_name": node2_name
# })
# # 将唯一的节点信息添加到结果中
# formatted_results["nodes"] = list(node_dict.values())
# return formatted_results
# entities = ['jqy', '维c']
# results = []
# for entitie in entities:
# result = kgdb.query_by_vector(entitie)
# if result != []:
# results.extend(result)
# print(format_query_results(results))
# kgdb.txt_add_vector_entity(triples, model)
# print("Extend the Graph data base")
# kgdb.jsonl_file_add_entity("/data2024/yyyl/ProjectAthena/tep.jsonl", kgdb_name)
# print("Extend the Graph data base")
# triples_path = "output.jsonl"
# def read_triples(file_path):
# with open(file_path, 'r', encoding='utf-8') as file:
# for line in file:
# item = json.loads(line.strip())
# yield [item]
# for trio in read_triples(triples_path):
# kgdb.txt_add_entity(trio)
# 通过文件添加三元组数据
# kgdb.file_add_entity("/data2024/yyyl/ProjectAthena/test.pdf", "/data2024/yyyl/ProjectAthena/output.jsonl", kgdb_name)
# print("Extend the Graph data base")
# 删除数据库信息
# kgdb.delete_entity()
# print("Clear the Graph data base")
# 查询所有节点和关系
# results = kgdb.query_all_nodes_and_relationships(kgdb_name)
# print(results)
# 查询特定实体及其关系
# results = kgdb.query_specific_entity("kllll", kgdb_name)
# print(results)
# 查询特定实体及其关系
# results = kgdb.query_by_vector_tep("z", model, tokenizer)
# print(results)
# 查询特定关系类型的所有节点
# results = kgdb.query_by_relationship_type("作用或食用效果", kgdb_name)
# print(results)
# 模糊查询
# results = kgdb.query_entity_like("三七", kgdb_name)
# print(results)
# 查询节点信息
# results = kgdb.query_entity_like("三七提取物", kgdb_name)
# print(results)
# 关闭数据库连接
kgdb.close()

View File

@ -1,8 +1,7 @@
import os import os
from pymilvus import MilvusClient, MilvusException from pymilvus import MilvusClient, MilvusException
from src.utils import setup_logger, hashstr from src.utils import logger, hashstr
logger = setup_logger("KnowledgeBase")
class KnowledgeBase: class KnowledgeBase:
@ -68,7 +67,7 @@ class KnowledgeBase:
logger.error(f"Collection {collection_name} not found, create it") logger.error(f"Collection {collection_name} not found, create it")
# self.add_collection(collection_name) # self.add_collection(collection_name)
vectors = self.embed_model.encode(docs) vectors = self.embed_model.batch_encode(docs)
data = [{ data = [{
"id": int(random.random() * 1e12), "id": int(random.random() * 1e12),

View File

@ -1,7 +1,5 @@
from src.models.rerank_model import get_reranker from src.models.rerank_model import get_reranker
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
logger = setup_logger("server-common")
class Retriever: class Retriever:
@ -19,7 +17,6 @@ class Retriever:
self.web_searcher = WebSearcher() self.web_searcher = WebSearcher()
def retrieval(self, query, history, meta): def retrieval(self, query, history, meta):
refs = {"query": query, "history": history, "meta": meta} refs = {"query": query, "history": history, "meta": meta}
refs["model_name"] = self.config.model_name refs["model_name"] = self.config.model_name
refs["entities"] = self.reco_entities(query, history, refs) refs["entities"] = self.reco_entities(query, history, refs)
@ -66,8 +63,8 @@ class Retriever:
def query_classification(self, query): def query_classification(self, query):
"""判断是否需要查询 """判断是否需要查询
- 对于完全基于用户给定信息的任务称之为足够sufficient不需要检索 - 对于完全基于用户给定信息的任务称之为"足够""sufficient"不需要检索
- 否则称之为不足insufficient可能需要检索 - 否则称之为"不足""insufficient"可能需要检索
""" """
raise NotImplementedError raise NotImplementedError
@ -107,12 +104,14 @@ class Retriever:
distance_threshold = meta.get("distanceThreshold", 0) distance_threshold = meta.get("distanceThreshold", 0)
top_k = meta.get("topK", 5) top_k = meta.get("topK", 5)
# 检索
all_kb_res = self.dbm.knowledge_base.search(rw_query, db_name, limit=max_query_count) all_kb_res = self.dbm.knowledge_base.search(rw_query, db_name, limit=max_query_count)
for r in all_kb_res: for r in all_kb_res:
r["file"] = kb.id2file(r["entity"]["file_id"]) r["file"] = kb.id2file(r["entity"]["file_id"])
kb_res = [r for r in all_kb_res if r["distance"] > distance_threshold] kb_res = [r for r in all_kb_res if r["distance"] > distance_threshold]
# 重排序
if self.config.enable_reranker and len(kb_res) > 0: if self.config.enable_reranker and len(kb_res) > 0:
texts = [r["entity"]["text"] for r in kb_res] texts = [r["entity"]["text"] for r in kb_res]
rerank_scores = self.reranker.compute_score([rw_query, texts], normalize=True) rerank_scores = self.reranker.compute_score([rw_query, texts], normalize=True)
@ -176,7 +175,7 @@ class Retriever:
return entities return entities
def _extract_relationship_info(self, relationship, source_name, target_name): def _extract_relationship_info(self, relationship, source_name=None, target_name=None, node_dict=None):
""" """
提取关系信息并返回格式化的节点和边信息 提取关系信息并返回格式化的节点和边信息
""" """
@ -189,6 +188,9 @@ class Retriever:
source_id = source.element_id source_id = source.element_id
target_id = target.element_id target_id = target.element_id
source_name = node_dict[source_id]["name"] if source_name is None else source_name
target_name = node_dict[target_id]["name"] if target_name is None else target_name
relationship_type = relationship._properties.get("type", "unknown") relationship_type = relationship._properties.get("type", "unknown")
if relationship_type == "unknown": if relationship_type == "unknown":
relationship_type = relationship.type relationship_type = relationship.type
@ -230,24 +232,33 @@ class Retriever:
return formatted_results return formatted_results
def format_query_results(self, results): def format_query_results(self, results):
logger.debug(f"Graph Query Results: {results}")
formatted_results = {"nodes": [], "edges": []} formatted_results = {"nodes": [], "edges": []}
node_dict = {} node_dict = {}
for item in results: for item in results:
if not isinstance(item[1], list) or not item[1]: # 检查数据格式
if len(item) < 2 or not isinstance(item[1], list):
continue continue
relationship = item[1][0] node_dict[item[0].element_id] = dict(id=item[0].element_id, name=item[0]._properties.get("name", "Unknown"))
source_name = item[0] node_dict[item[2].element_id] = dict(id=item[2].element_id, name=item[2]._properties.get("name", "Unknown"))
target_name = item[2] if len(item) > 2 else "unknown"
node_info, edge_info = self._extract_relationship_info(relationship, source_name, target_name) # 处理关系列表中的每个关系
if node_info is None or edge_info is None: for i, relationship in enumerate(item[1]):
continue try:
# 提取关系信息
node_info, edge_info = self._extract_relationship_info(relationship, node_dict=node_dict)
if node_info is None or edge_info is None:
continue
node_dict.update({node["id"]: node for node in node_info}) # 添加边
formatted_results["edges"].append(edge_info) formatted_results["edges"].append(edge_info)
except Exception as e:
logger.error(f"处理关系时出错: {e}, 关系: {relationship}")
continue
# 将节点字典转换为列表
formatted_results["nodes"] = list(node_dict.values()) formatted_results["nodes"] = list(node_dict.values())
return formatted_results return formatted_results

View File

@ -3,9 +3,7 @@ from src.core import DataBaseManager
from src.core.retriever import Retriever from src.core.retriever import Retriever
from src.models import select_model from src.models import select_model
from src.config import Config from src.config import Config
from src.utils import setup_logger from src.utils import logger
logger = setup_logger("Startup")
class Startup: class Startup:

View File

@ -1,11 +1,13 @@
import uvicorn import uvicorn
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv("src/.env")
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from src.routers import router from src.routers import router
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
load_dotenv()
app = FastAPI() app = FastAPI()
app.include_router(router) app.include_router(router)
@ -20,9 +22,6 @@ app.add_middleware(
) )
logger = setup_logger("server:main")
if __name__ == "__main__": if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=5000, threads=10, workers=10) uvicorn.run(app, host="0.0.0.0", port=5000, threads=10, workers=10)

View File

@ -18,6 +18,7 @@ def select_model(config, model_provider=None, model_name=None):
"siliconflow", "siliconflow",
"zhipu", "zhipu",
"lingyiwanwu", "lingyiwanwu",
"together.ai",
]: ]:
return OpenAIBase( return OpenAIBase(
api_key=os.getenv(model_info["env"][0]), api_key=os.getenv(model_info["env"][0]),

View File

@ -1,9 +1,6 @@
import os import os
from openai import OpenAI from openai import OpenAI
from src.utils.logging_config import setup_logger from src.utils import logger
logger = setup_logger(__name__)
class OpenAIBase(): class OpenAIBase():
def __init__(self, api_key, base_url, model_name): def __init__(self, api_key, base_url, model_name):

View File

@ -4,12 +4,9 @@ import requests
from FlagEmbedding import FlagModel from FlagEmbedding import FlagModel
from src.config import EMBED_MODEL_INFO from src.config import EMBED_MODEL_INFO
from src.utils.logging_config import setup_logger from src.utils import hashstr, logger
from src.utils import hashstr
logger = setup_logger("EmbeddingModel")
class LocalEmbeddingModel(FlagModel): class LocalEmbeddingModel(FlagModel):
def __init__(self, config, **kwargs): def __init__(self, config, **kwargs):
info = EMBED_MODEL_INFO[config.embed_model] info = EMBED_MODEL_INFO[config.embed_model]
@ -31,6 +28,7 @@ class RemoteEmbeddingModel:
embed_state = {} embed_state = {}
def batch_encode(self, messages, batch_size=20): def batch_encode(self, messages, batch_size=20):
logger.info(f"Batch encoding {len(messages)} messages")
data = [] data = []
if len(messages) > batch_size: if len(messages) > batch_size:

View File

@ -4,9 +4,7 @@ import numpy as np
from typing import List, Union, Dict from typing import List, Union, Dict
from src.models.embedding import RemoteEmbeddingModel from src.models.embedding import RemoteEmbeddingModel
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
logger = setup_logger("OllamaEmbedding")
class OllamaEmbedding(RemoteEmbeddingModel): class OllamaEmbedding(RemoteEmbeddingModel):
""" """

View File

@ -5,10 +5,7 @@ import numpy as np
from FlagEmbedding import FlagReranker from FlagEmbedding import FlagReranker
from src.config import RERANKER_LIST from src.config import RERANKER_LIST
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
logger = setup_logger("RerankModel")
class LocalReranker(FlagReranker): class LocalReranker(FlagReranker):

View File

@ -11,8 +11,7 @@ from transformers import (
BitsAndBytesConfig BitsAndBytesConfig
) )
from src.utils import setup_logger from src.utils import logger
logger = setup_logger("OneKE")
dotenv.load_dotenv() dotenv.load_dotenv()

View File

@ -5,11 +5,11 @@ base = APIRouter()
from fastapi import FastAPI, HTTPException from fastapi import FastAPI, HTTPException
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from fastapi import Request, Body from fastapi import Request, Body
from src.core import HistoryManager
from src.utils.logging_config import setup_logger
from src.core.startup import startup
logger = setup_logger("server-base") from src.core import HistoryManager
from src.core.startup import startup
from src.utils import logger
@base.get("/") @base.get("/")
async def route_index(): async def route_index():
@ -39,6 +39,6 @@ def get_log():
last_lines = deque(f, maxlen=1000) last_lines = deque(f, maxlen=1000)
log = ''.join(last_lines) log = ''.join(last_lines)
return {"log": log} return {"log": log, "message": "success", "log_file": LOG_FILE}

View File

@ -5,14 +5,12 @@ from fastapi.responses import StreamingResponse, Response
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from src.core import HistoryManager from src.core import HistoryManager
from src.core.startup import startup from src.core.startup import startup
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
chat = APIRouter(prefix="/chat") chat = APIRouter(prefix="/chat")
logger = setup_logger("server-chat")
# 创建线程池 # 创建线程池
executor = ThreadPoolExecutor() executor = ThreadPoolExecutor()
refs_pool = {}
@chat.get("/") @chat.get("/")
async def chat_get(): async def chat_get():
@ -26,6 +24,7 @@ def chat_post(
cur_res_id: str = Body(...)): cur_res_id: str = Body(...)):
history_manager = HistoryManager(history) history_manager = HistoryManager(history)
logger.debug(f"Received query: {query} with meta: {meta}")
def make_chunk(content=None, **kwargs): def make_chunk(content=None, **kwargs):
return json.dumps({ return json.dumps({
@ -112,10 +111,4 @@ async def call(query: str = Body(...), meta: dict = Body(None)):
response = await predict_async(query) response = await predict_async(query)
logger.debug({"query": query, "response": response.content}) logger.debug({"query": query, "response": response.content})
return {"response": response.content} return {"response": response.content}
@chat.get("/refs")
def get_refs(cur_res_id: str):
global refs_pool
refs = refs_pool.pop(cur_res_id, None)
return {"refs": refs}

View File

@ -2,12 +2,11 @@ import os
from typing import List, Optional from typing import List, Optional
from fastapi import APIRouter, File, UploadFile, HTTPException, Depends, Body from fastapi import APIRouter, File, UploadFile, HTTPException, Depends, Body
from src.utils import setup_logger, hashstr from src.utils import logger, hashstr
from src.core.startup import startup from src.core.startup import startup
data = APIRouter(prefix="/data") data = APIRouter(prefix="/data")
logger = setup_logger("server-database")
@data.get("/") @data.get("/")
def get_databases(): def get_databases():
@ -100,7 +99,6 @@ async def get_graph_info():
@data.get("/graph/node") @data.get("/graph/node")
async def get_graph_node(entity_name: str): async def get_graph_node(entity_name: str):
logger.debug(f"Get graph node {entity_name}")
result = startup.dbm.graph_base.query_node(entity_name=entity_name) result = startup.dbm.graph_base.query_node(entity_name=entity_name)
return {"result": startup.retriever.format_query_results(result), "message": "success"} return {"result": startup.retriever.format_query_results(result), "message": "success"}
@ -113,7 +111,7 @@ async def get_graph_nodes(kgdb_name: str, num: int):
result = startup.dbm.graph_base.get_sample_nodes(kgdb_name, num) result = startup.dbm.graph_base.get_sample_nodes(kgdb_name, num)
return {"result": startup.retriever.format_general_results(result), "message": "success"} return {"result": startup.retriever.format_general_results(result), "message": "success"}
@data.post("/graph/add") @data.post("/graph/add-by-jsonl")
async def add_graph_entity(file_path: str = Body(...), kgdb_name: Optional[str] = Body(None)): async def add_graph_entity(file_path: str = Body(...), kgdb_name: Optional[str] = Body(None)):
if not startup.config.enable_knowledge_graph: if not startup.config.enable_knowledge_graph:
raise HTTPException(status_code=400, detail="Knowledge graph is not enabled") raise HTTPException(status_code=400, detail="Knowledge graph is not enabled")

View File

@ -3,11 +3,10 @@ from fastapi import APIRouter, Body
from pydantic import BaseModel from pydantic import BaseModel
from typing import List, Dict, Any, Optional from typing import List, Dict, Any, Optional
from src.utils import setup_logger from src.utils import logger
tool = APIRouter(prefix="/tool") tool = APIRouter(prefix="/tool")
logger = setup_logger("server-tools")
class Tool(BaseModel): class Tool(BaseModel):
name: str name: str

View File

@ -50,6 +50,18 @@ MODEL_NAMES:
- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B - deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
- Qwen/Qwen2.5-72B-Instruct - Qwen/Qwen2.5-72B-Instruct
- Qwen/Qwen2.5-7B-Instruct - Qwen/Qwen2.5-7B-Instruct
together.ai:
name: Together.ai
url: https://api.together.ai/models
base_url: https://api.together.xyz/v1/
default: meta-llama/Llama-3.3-70B-Instruct-Turbo-Free
env:
- TOGETHER_API_KEY
models:
- meta-llama/Llama-3.3-70B-Instruct-Turbo
- meta-llama/Llama-3.3-70B-Instruct-Turbo-Free
- deepseek-ai/DeepSeek-R1-Distill-Llama-70B-free
qianfan: qianfan:
name: 百度千帆 (QianFan) name: 百度千帆 (QianFan)
url: https://open.bigmodel.cn/dev/api url: https://open.bigmodel.cn/dev/api

View File

@ -1,6 +1,6 @@
import time import time
import random import random
from src.utils.logging_config import setup_logger, logger from src.utils.logging_config import logger
def is_text_pdf(pdf_path): def is_text_pdf(pdf_path):
import fitz import fitz

View File

@ -1,9 +1,7 @@
import os import os
from typing import List, Dict from typing import List, Dict
from tavily import TavilyClient from tavily import TavilyClient
from src.utils.logging_config import setup_logger from src.utils.logging_config import logger
logger = setup_logger("web-search")
class WebSearcher: class WebSearcher:
def __init__(self): def __init__(self):

View File

@ -0,0 +1,5 @@
{"h": "贾代善", "t": "贾源", "r": "子"}
{"h": "娄氏", "t": "贾源", "r": "重孙媳妇"}
{"h": "贾母", "t": "贾代善", "r": "妻"}
{"h": "老姨奶奶", "t": "贾代善", "r": "妾"}
{"h": "贾敏", "t": "贾代善", "r": "女"}

View File

@ -1,6 +1,6 @@
<template> <template>
<div class="chat" ref="chatContainer"> <div class="chat" ref="chatContainer">
<div class="header"> <div class="chat-header">
<div class="header__left"> <div class="header__left">
<div <div
v-if="!state.isSidebarOpen" v-if="!state.isSidebarOpen"
@ -154,7 +154,7 @@
</template> </template>
<script setup> <script setup>
import { reactive, ref, onMounted, toRefs, nextTick, computed, watch } from 'vue' import { reactive, ref, onMounted, toRefs, nextTick, onUnmounted, watch } from 'vue'
import { import {
SendOutlined, SendOutlined,
MenuOutlined, MenuOutlined,
@ -197,7 +197,11 @@ const configStore = useConfigStore()
const { conv, state } = toRefs(props) const { conv, state } = toRefs(props)
const chatContainer = ref(null) const chatContainer = ref(null)
const isStreaming = ref(false) const isStreaming = ref(false)
const userIsScrolling = ref(false);
const shouldAutoScroll = ref(true);
const panel = ref(null) const panel = ref(null)
const modelCard = ref(null) const modelCard = ref(null)
const examples = ref([ const examples = ref([
@ -295,10 +299,23 @@ const renameTitle = () => {
} }
} }
const handleUserScroll = () => {
// 100
const isNearBottom = chatContainer.value.scrollHeight - chatContainer.value.scrollTop - chatContainer.value.clientHeight < 100;
//
userIsScrolling.value = !isNearBottom;
//
shouldAutoScroll.value = isNearBottom;
};
const scrollToBottom = () => { const scrollToBottom = () => {
setTimeout(() => { if (shouldAutoScroll.value) {
chatContainer.value.scrollTop = chatContainer.value.scrollHeight - chatContainer.value.clientHeight setTimeout(() => {
}, 10) chatContainer.value.scrollTop = chatContainer.value.scrollHeight - chatContainer.value.clientHeight;
}, 10);
}
} }
const generateRandomHash = (length) => { const generateRandomHash = (length) => {
@ -502,22 +519,6 @@ const fetchChatResponse = (user_input, cur_res_id) => {
}); });
} }
const fetchRefs = (cur_res_id) => {
return new Promise((resolve, reject) => {
fetch(`/api/chat/refs?cur_res_id=${cur_res_id}`, {
method: "GET",
headers: {
"Content-Type": "application/json"
},
}).then(response => response.json())
.then(data => {
resolve(data.refs)
})
.catch((error) => {
reject(error)
})
})
}
// sendMessage // sendMessage
const sendMessage = () => { const sendMessage = () => {
@ -531,6 +532,8 @@ const sendMessage = () => {
isStreaming.value = true; isStreaming.value = true;
appendUserMessage(user_input); appendUserMessage(user_input);
appendAiMessage("", null); appendAiMessage("", null);
forceScrollToBottom();
const cur_res_id = conv.value.messages[conv.value.messages.length - 1].id; const cur_res_id = conv.value.messages[conv.value.messages.length - 1].id;
conv.value.inputText = ''; conv.value.inputText = '';
meta.db_name = dbName; meta.db_name = dbName;
@ -553,15 +556,15 @@ const retryMessage = (id) => {
sendMessage(); sendMessage();
} }
const autoSend = (msg) => {
conv.value.inputText = msg
sendMessage()
}
// //
onMounted(() => { onMounted(() => {
scrollToBottom() scrollToBottom()
loadDatabases() loadDatabases()
chatContainer.value.addEventListener('scroll', handleUserScroll);
//
const storedMeta = localStorage.getItem('meta'); const storedMeta = localStorage.getItem('meta');
if (storedMeta) { if (storedMeta) {
const parsedMeta = JSON.parse(storedMeta); const parsedMeta = JSON.parse(storedMeta);
@ -569,6 +572,20 @@ onMounted(() => {
} }
}); });
onUnmounted(() => {
if (chatContainer.value) {
chatContainer.value.removeEventListener('scroll', handleUserScroll);
}
});
//
const forceScrollToBottom = () => {
shouldAutoScroll.value = true;
setTimeout(() => {
chatContainer.value.scrollTop = chatContainer.value.scrollHeight - chatContainer.value.clientHeight;
}, 10);
};
// meta // meta
watch( watch(
() => meta, () => meta,
@ -593,7 +610,7 @@ watch(
flex: 5 5 200px; flex: 5 5 200px;
overflow-y: scroll; overflow-y: scroll;
.header { .chat-header {
user-select: none; user-select: none;
position: sticky; position: sticky;
top: 0; top: 0;
@ -728,7 +745,7 @@ watch(
.chat-box { .chat-box {
width: 100%; width: 100%;
max-width: 900px; max-width: 800px;
margin: 0 auto; margin: 0 auto;
flex-grow: 1; flex-grow: 1;
padding: 1rem 2rem; padding: 1rem 2rem;
@ -819,7 +836,7 @@ watch(
flex-direction: column; flex-direction: column;
width: 100%; width: 100%;
height: auto; height: auto;
max-width: 900px; max-width: 800px;
margin: 0 auto; margin: 0 auto;
padding: 0.25rem 0.5rem; padding: 0.25rem 0.5rem;
border: 2px solid var(--gray-200); border: 2px solid var(--gray-200);
@ -1008,14 +1025,14 @@ watch(
height: calc(100vh - 60px); height: calc(100vh - 60px);
} }
.chat-container .chat .header { .chat-container .chat .chat-header {
background: var(--main-light-4); background: var(--main-light-4);
.header__left, .header__right { .header__left, .header__right {
gap: 20px; gap: 24px;
} }
.nav-btn { .nav-btn {
font-size: 1.5rem; font-size: 1.3rem;
padding: 0; padding: 0;
&:hover { &:hover {
@ -1057,19 +1074,29 @@ watch(
</style> </style>
<style lang="less"> <style lang="less">
.message-box pre {
border-radius: 8px;
font-size: 14px;
border: 1px solid var(--main-light-3);
padding: 1rem;
&:has(code.hljs) {
padding: 0;
}
}
.message-md { .message-md {
color: var(--gray-900);
max-width: 100%; max-width: 100%;
pre {
border-radius: 8px;
font-size: 14px;
border: 1px solid var(--main-light-3);
padding: 1rem;
&:has(code.hljs) {
padding: 0;
}
code.hljs {
background-color: var(--gray-100);
}
}
strong {
color: var(--gray-800);
}
h1, h2, h3, h4, h5, h6 { h1, h2, h3, h4, h5, h6 {
font-size: 1rem; font-size: 1rem;
} }
@ -1083,6 +1110,15 @@ watch(
ol, ul { ol, ul {
padding-left: 1rem; padding-left: 1rem;
} }
hr {
margin-bottom: 1rem;
}
a {
color: var(--main-800);
margin: auto 2px;
}
} }
</style> </style>

View File

@ -337,8 +337,12 @@ div.header, #app-router-view {
div.header { div.header {
display: none; display: none;
} }
}
.debug-panel {
bottom: 10rem;
}
}
.app-layout div.header-mobile { .app-layout div.header-mobile {
display: flex; display: flex;
flex-direction: row; flex-direction: row;
@ -348,6 +352,7 @@ div.header, #app-router-view {
align-items: center; align-items: center;
flex: 0 0 60px; flex: 0 0 60px;
border-right: none; border-right: none;
height: 40px;
.nav-item { .nav-item {
text-decoration: none; text-decoration: none;

View File

@ -256,10 +256,11 @@ onMounted(() => {
} }
@media (max-width: 520px) { @media (max-width: 520px) {
.conversation { .conversations {
position: absolute; position: absolute;
z-index: 101; z-index: 101;
width: 300px; width: 300px;
height: 100%;
border-radius: 0 16px 16px 0; border-radius: 0 16px 16px 0;
box-shadow: 0 0 10px 1px rgba(0, 0, 0, 0.05); box-shadow: 0 0 10px 1px rgba(0, 0, 0, 0.05);
} }

View File

@ -59,7 +59,9 @@
<template v-if="column.key === 'filename'"> <template v-if="column.key === 'filename'">
<a-button class="main-btn" type="link" @click="openFileDetail(record)">{{ text }}</a-button> <a-button class="main-btn" type="link" @click="openFileDetail(record)">{{ text }}</a-button>
</template> </template>
<template v-else-if="column.key === 'type'"><span :class="text">{{ text.toUpperCase() }}</span></template> <template v-else-if="column.key === 'type'">
<span :class="['span-type', text]">{{ text.toUpperCase() }}</span>
</template>
<template v-else-if="column.key === 'status' && text === 'done'"> <template v-else-if="column.key === 'status' && text === 'done'">
<CheckCircleFilled style="color: #41A317;"/> <CheckCircleFilled style="color: #41A317;"/>
</template> </template>
@ -813,7 +815,7 @@ onMounted(() => {
padding: 0; padding: 0;
} }
.pdf, .txt, .md { .span-type {
color: white; color: white;
padding: 2px 4px; padding: 2px 4px;
border-radius: 4px; border-radius: 4px;
@ -821,6 +823,7 @@ onMounted(() => {
font-weight: bold; font-weight: bold;
opacity: 0.8; opacity: 0.8;
user-select: none; user-select: none;
background: #005F77;
} }
.pdf { .pdf {
@ -831,6 +834,16 @@ onMounted(() => {
background: #068033; background: #068033;
} }
.docx, .doc {
background: #2C59B7;
}
.md {
background: #020817;
}
button.main-btn { button.main-btn {
font-weight: bold; font-weight: bold;
font-size: 14px; font-size: 14px;

View File

@ -50,12 +50,13 @@
@ok="addDocumentByFile" @ok="addDocumentByFile"
@cancel="() => state.showModal = false" @cancel="() => state.showModal = false"
ok-text="添加到图数据库" cancel-text="取消" ok-text="添加到图数据库" cancel-text="取消"
:ok-button-props="{ disabled: disabled }"
:confirm-loading="state.precessing"> :confirm-loading="state.precessing">
<div v-if="graphInfo?.embed_model_name"> <div v-if="graphInfo?.embed_model_name">
<p>当前图数据库向量模型{{ graphInfo?.embed_model_name }}</p> <p>当前图数据库向量模型{{ graphInfo?.embed_model_name }}</p>
<p>当前所选择的向量模型是 {{ configStore.config.embed_model }}</p> <p>当前所选择的向量模型是 {{ cur_embed_model }}</p>
</div> </div>
<p v-else>第一次创建之后将无法修改向量模型当前向量模型 {{ configStore.config.embed_model }}</p> <p v-else>第一次创建之后将无法修改向量模型当前向量模型 {{ cur_embed_model }}</p>
<div class="upload"> <div class="upload">
<a-upload-dragger <a-upload-dragger
class="upload-dragger" class="upload-dragger"
@ -63,7 +64,7 @@
name="file" name="file"
:fileList="fileList" :fileList="fileList"
:max-count="1" :max-count="1"
:disabled="state.precessing || (graphInfo?.embed_model_name && graphInfo?.embed_model_name !== configStore.config.embed_model)" :disabled="disabled"
action="/api/data/upload" action="/api/data/upload"
@change="handleFileUpload" @change="handleFileUpload"
@drop="handleDrop" @drop="handleDrop"
@ -87,6 +88,12 @@ import { UploadOutlined } from '@ant-design/icons-vue';
import HeaderComponent from '@/components/HeaderComponent.vue'; import HeaderComponent from '@/components/HeaderComponent.vue';
const configStore = useConfigStore() const configStore = useConfigStore()
const cur_embed_model = computed(() => configStore.config.embed_model_names[configStore.config.embed_model].name)
const disabled = computed(() => {
if (state.precessing) return true
if (graphInfo?.value?.embed_model_name !== cur_embed_model.value) return true
return false
})
let graphInstance let graphInstance
const graphInfo = ref(null) const graphInfo = ref(null)
@ -152,7 +159,7 @@ const getGraphData = () => {
const addDocumentByFile = () => { const addDocumentByFile = () => {
state.precessing = true state.precessing = true
const files = fileList.value.filter(file => file.status === 'done').map(file => file.response.file_path) const files = fileList.value.filter(file => file.status === 'done').map(file => file.response.file_path)
fetch('/api/data/graph/add', { fetch('/api/data/graph/add-by-jsonl', {
method: 'POST', method: 'POST',
headers: { headers: {
"Content-Type": "application/json" // Content-Type "Content-Type": "application/json" // Content-Type
@ -178,7 +185,7 @@ const loadSampleNodes = () => {
.then((res) => { .then((res) => {
if (res.ok) { if (res.ok) {
return res.json(); return res.json();
} else if (!configStore.config.enable_knowledge_graph) { } else if (configStore?.config && !configStore?.config.enable_knowledge_graph) {
throw new Error('请前往设置页面配置启用知识图谱') throw new Error('请前往设置页面配置启用知识图谱')
} else { } else {
throw new Error("加载失败"); throw new Error("加载失败");

View File

@ -14,15 +14,15 @@
</template> </template>
</HeaderComponent> </HeaderComponent>
<div class="setting-container layout-container"> <div class="setting-container layout-container">
<div class="sider"> <div class="sider" v-if="state.windowWidth > 520">
<a-button type="text" :class="{ activesec: state.section === 'base'}" @click="state.section='base'" :icon="h(SettingOutlined)"> 基本设置 </a-button> <a-button type="text" :class="{ activesec: state.section === 'base'}" @click="state.section='base'" :icon="h(SettingOutlined)"> 基本设置 </a-button>
<a-button type="text" :class="{ activesec: state.section === 'model'}" @click="state.section='model'" :icon="h(CodeOutlined)"> 模型配置 </a-button> <a-button type="text" :class="{ activesec: state.section === 'model'}" @click="state.section='model'" :icon="h(CodeOutlined)"> 模型配置 </a-button>
<a-button type="text" :class="{ activesec: state.section === 'path'}" @click="state.section='path'" :icon="h(FolderOutlined)"> 路径配置 </a-button> <a-button type="text" :class="{ activesec: state.section === 'path'}" @click="state.section='path'" :icon="h(FolderOutlined)"> 路径配置 </a-button>
</div> </div>
<div class="setting" v-if="state.section === 'base'"> <div class="setting" v-if="state.windowWidth <= 520 || state.section === 'base'">
<h3>基础模型配置</h3> <h3>基础模型配置</h3>
<div class="section"> <div class="section">
<div class="card"> <div class="card card-select">
<span class="label">{{ items?.embed_model.des }}</span> <span class="label">{{ items?.embed_model.des }}</span>
<a-select style="width: 300px" <a-select style="width: 300px"
:value="configStore.config?.embed_model" :value="configStore.config?.embed_model"
@ -34,7 +34,7 @@
</a-select-option> </a-select-option>
</a-select> </a-select>
</div> </div>
<div class="card"> <div class="card card-select">
<span class="label">{{ items?.reranker.des }}</span> <span class="label">{{ items?.reranker.des }}</span>
<a-select style="width: 300px" <a-select style="width: 300px"
:value="configStore.config?.reranker" :value="configStore.config?.reranker"
@ -81,7 +81,7 @@
</div> </div>
<h3>检索配置</h3> <h3>检索配置</h3>
<div class="section"> <div class="section">
<div class="card"> <div class="card card-select">
<span class="label">{{ items?.use_rewrite_query.des }}</span> <span class="label">{{ items?.use_rewrite_query.des }}</span>
<a-select style="width: 200px" <a-select style="width: 200px"
:value="configStore.config?.use_rewrite_query" :value="configStore.config?.use_rewrite_query"
@ -95,7 +95,7 @@
</div> </div>
</div> </div>
</div> </div>
<div class="setting" v-if="state.section === 'model'"> <div class="setting" v-if="state.windowWidth <= 520 || state.section === 'model'">
<h3>模型配置</h3> <h3>模型配置</h3>
<p>请在 <code>src/.env</code> 文件中配置对应的 APIKEY</p> <p>请在 <code>src/.env</code> 文件中配置对应的 APIKEY</p>
<div class="model-provider-card"> <div class="model-provider-card">
@ -184,7 +184,7 @@
</div> </div>
</div> </div>
</div> </div>
<div class="setting" v-if="state.section ==='path'"> <div class="setting" v-if="state.windowWidth <= 520 || state.section ==='path'">
<h3>本地模型配置</h3> <h3>本地模型配置</h3>
<p>如果是 Docker 启动务必确保在环境变量中设置了 MODEL_DIR或者设置了 volumes 映射</p> <p>如果是 Docker 启动务必确保在环境变量中设置了 MODEL_DIR或者设置了 volumes 映射</p>
<TableConfigComponent <TableConfigComponent
@ -198,7 +198,7 @@
<script setup> <script setup>
import { message } from 'ant-design-vue'; import { message } from 'ant-design-vue';
import { computed, reactive, ref, h, watch } from 'vue' import { computed, reactive, ref, h, watch, onMounted, onUnmounted } from 'vue'
import { useConfigStore } from '@/stores/config'; import { useConfigStore } from '@/stores/config';
import { import {
ReloadOutlined, ReloadOutlined,
@ -231,16 +231,17 @@ const customModel = reactive({
}) })
const state = reactive({ const state = reactive({
loading: false, loading: false,
section: 'base' section: 'base',
windowWidth: window?.innerWidth || 0
}) })
// modelStatus key // modelStatus key
const modelKeys = computed(() => { const modelKeys = computed(() => {
return Object.keys(modelStatus.value).filter(key => modelStatus.value[key]) return Object.keys(modelStatus.value || {}).filter(key => modelStatus.value?.[key])
}) })
const notModelKeys = computed(() => { const notModelKeys = computed(() => {
return Object.keys(modelStatus.value).filter(key => !modelStatus.value[key]) return Object.keys(modelStatus.value || {}).filter(key => !modelStatus.value?.[key])
}) })
const generateRandomHash = (length) => { const generateRandomHash = (length) => {
@ -360,6 +361,19 @@ const handleCancelCustomModel = () => {
customModel.visible = false customModel.visible = false
} }
const updateWindowWidth = () => {
state.windowWidth = window?.innerWidth || 0
}
onMounted(() => {
updateWindowWidth()
window.addEventListener('resize', updateWindowWidth)
})
onUnmounted(() => {
window.removeEventListener('resize', updateWindowWidth)
})
const sendRestart = () => { const sendRestart = () => {
console.log('Restarting...') console.log('Restarting...')
message.loading({ content: '重新加载模型中', key: "restart", duration: 0 }); message.loading({ content: '重新加载模型中', key: "restart", duration: 0 });
@ -618,7 +632,18 @@ const sendRestart = () => {
} }
} }
}
@media (max-width: 520px) {
.setting-container {
flex-direction: column;
}
.card.card-select {
gap: 0.75rem;
align-items: flex-start;
flex-direction: column;
}
} }
</style> </style>

View File

@ -14,7 +14,7 @@ export default defineConfig(({ mode }) => {
server: { server: {
proxy: { proxy: {
'^/api': { '^/api': {
target: env.VITE_API_URL || 'http://localhost:5000', target: env.VITE_API_URL || 'http://localhost:5050',
changeOrigin: true, changeOrigin: true,
rewrite: (path) => path.replace(/^\/api/, '') rewrite: (path) => path.replace(/^\/api/, '')
} }