datai/docs/archive/prompts/017-file-storage-and-extract.md

649 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Prompt - 文件存储和解压处理实现
## 输入引用
引用相关的 docs 文档链接:
- [REQ-010-7.md](../requirements/REQ-010-7.md) - 文件存储和解压处理需求文档
- [0016-file-storage-and-extract.md](../decisions/adr/0016-file-storage-and-extract.md) - 文件存储和解压处理架构决策
- [REQ-010-6.md](../requirements/REQ-010-6.md) - 元数据拉取核心功能需求文档
- [file/index.md](../api-docs/file/index.md) - 文件模块 API 文档索引(唯一真源)
## Context Maps
强制列出本次 Prompt 依赖的 Canvas 文件:
- [Authentication.canvas](../../Authentication.canvas) - 项目架构视觉化展示
- **相关节点**: [集成核心](node_integration_core) - 提供与Salesforce的各种连接方式
## 目标
实现文件存储和解压处理功能,包括:
1. 实现 Zip 文件存储功能,支持本地文件系统和 OSS 存储
2. 实现文件解压功能,支持大文件处理和流式解压
3. 实现文件路径管理,支持路径规范化和安全性验证
4. 实现存储空间管理,支持存储空间监控和告警
5. 实现文件清理功能,支持过期文件自动清理和手动清理
## 输出格式
### 代码结构
```
datai-salesforce-metadata/
├── src/main/java/com/datai/salesforce/metadata/file/
│ ├── service/
│ │ ├── IFileStorageService.java # 文件存储服务接口
│ │ ├── LocalStorageService.java # 本地文件系统存储实现
│ │ ├── OssStorageService.java # OSS 存储实现
│ │ ├── IFileExtractService.java # 文件解压服务接口
│ │ ├── FileExtractServiceImpl.java # 文件解压服务实现
│ │ ├── IPathValidator.java # 路径验证接口
│ │ ├── PathValidator.java # 路径验证实现
│ │ ├── IStorageMonitorService.java # 存储空间监控服务接口
│ │ ├── StorageMonitorServiceImpl.java # 存储空间监控服务实现
│ │ ├── IFileCleanupService.java # 文件清理服务接口
│ │ └── FileCleanupServiceImpl.java # 文件清理服务实现
│ ├── dto/
│ │ ├── FileStorageRequest.java # 文件存储请求
│ │ ├── FileStorageResponse.java # 文件存储响应
│ │ ├── FileExtractRequest.java # 文件解压请求
│ │ ├── FileExtractResponse.java # 文件解压响应
│ │ ├── ExtractProgressCallback.java # 解压进度回调
│ │ ├── StorageSpaceInfo.java # 存储空间信息
│ │ ├── FileCleanupRequest.java # 文件清理请求
│ │ └── FileCleanupResponse.java # 文件清理响应
│ ├── enums/
│ │ ├── StorageType.java # 存储类型枚举
│ │ └── ExtractStatus.java # 解压状态枚举
│ ├── config/
│ │ ├── FileStorageConfig.java # 文件存储配置
│ │ └── OssConfig.java # OSS 配置
│ ├── exception/
│ │ ├── FileStorageException.java # 文件存储异常
│ │ ├── FileExtractException.java # 文件解压异常
│ │ └── PathSecurityException.java # 路径安全异常
│ └── util/
│ ├── FileUtil.java # 文件工具类
│ └── ZipUtil.java # Zip 工具类
├── src/main/resources/
│ ├── application.yml # 应用配置
│ └── logback-spring.xml # 日志配置
└── src/test/java/com/datai/salesforce/metadata/file/
├── service/
│ ├── LocalStorageServiceTest.java # 本地存储服务测试
│ ├── FileExtractServiceTest.java # 文件解压服务测试
│ ├── PathValidatorTest.java # 路径验证测试
│ ├── StorageMonitorServiceTest.java # 存储监控服务测试
│ └── FileCleanupServiceTest.java # 文件清理服务测试
└── util/
├── FileUtilTest.java # 文件工具类测试
└── ZipUtilTest.java # Zip 工具类测试
```
### 代码示例
#### 1. IFileStorageService.java
```java
package com.datai.salesforce.metadata.file.service;
import java.io.InputStream;
import java.nio.file.Path;
public interface IFileStorageService {
String store(String fileName, InputStream inputStream, String subPath);
InputStream retrieve(String filePath);
boolean delete(String filePath);
boolean exists(String filePath);
Path getStoragePath(String subPath);
}
```
#### 2. LocalStorageService.java
```java
package com.datai.salesforce.metadata.file.service;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Service;
import java.io.*;
import java.nio.file.*;
@Slf4j
@Service
@ConditionalOnProperty(name = "file.storage.type", havingValue = "local")
public class LocalStorageService implements IFileStorageService {
@Value("${file.storage.local.base-path}")
private String basePath;
@Override
public String store(String fileName, InputStream inputStream, String subPath) {
try {
Path targetPath = Paths.get(basePath, subPath);
Files.createDirectories(targetPath);
Path filePath = targetPath.resolve(fileName);
Files.copy(inputStream, filePath, StandardCopyOption.REPLACE_EXISTING);
log.info("File stored successfully: {}", filePath);
return filePath.toString();
} catch (IOException e) {
log.error("Failed to store file: {}", fileName, e);
throw new RuntimeException("Failed to store file", e);
}
}
@Override
public InputStream retrieve(String filePath) {
try {
return Files.newInputStream(Paths.get(filePath));
} catch (IOException e) {
log.error("Failed to retrieve file: {}", filePath, e);
throw new RuntimeException("Failed to retrieve file", e);
}
}
@Override
public boolean delete(String filePath) {
try {
boolean deleted = Files.deleteIfExists(Paths.get(filePath));
log.info("File deleted: {}, result: {}", filePath, deleted);
return deleted;
} catch (IOException e) {
log.error("Failed to delete file: {}", filePath, e);
return false;
}
}
@Override
public boolean exists(String filePath) {
return Files.exists(Paths.get(filePath));
}
@Override
public Path getStoragePath(String subPath) {
return Paths.get(basePath, subPath);
}
}
```
#### 3. IFileExtractService.java
```java
package com.datai.salesforce.metadata.file.service;
import java.io.InputStream;
public interface IFileExtractService {
void extract(InputStream zipInputStream, String targetPath);
void extract(InputStream zipInputStream, String targetPath, ExtractProgressCallback callback);
void extract(InputStream zipInputStream, String targetPath, ExtractProgressCallback callback, boolean cancelFlag);
}
```
#### 4. FileExtractServiceImpl.java
```java
package com.datai.salesforce.metadata.file.service;
import com.datai.salesforce.metadata.file.dto.ExtractProgressCallback;
import com.datai.salesforce.metadata.file.exception.FileExtractException;
import com.datai.salesforce.metadata.file.util.PathValidator;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.io.*;
import java.nio.file.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
@Slf4j
@Service
public class FileExtractServiceImpl implements IFileExtractService {
@Autowired
private PathValidator pathValidator;
private static final int BUFFER_SIZE = 8192;
@Override
public void extract(InputStream zipInputStream, String targetPath) {
extract(zipInputStream, targetPath, null);
}
@Override
public void extract(InputStream zipInputStream, String targetPath, ExtractProgressCallback callback) {
extract(zipInputStream, targetPath, callback, false);
}
@Override
public void extract(InputStream zipInputStream, String targetPath, ExtractProgressCallback callback, boolean cancelFlag) {
Path basePath = Paths.get(targetPath);
try {
Files.createDirectories(basePath);
try (ZipInputStream zis = new ZipInputStream(new BufferedInputStream(zipInputStream))) {
ZipEntry entry;
int totalEntries = 0;
int processedEntries = 0;
while ((entry = zis.getNextEntry()) != null) {
if (cancelFlag) {
log.info("Extract cancelled");
break;
}
totalEntries++;
Path entryPath = basePath.resolve(entry.getName());
pathValidator.validatePath(basePath, entryPath);
if (entry.isDirectory()) {
Files.createDirectories(entryPath);
} else {
Files.createDirectories(entryPath.getParent());
try (BufferedOutputStream bos = new BufferedOutputStream(
new FileOutputStream(entryPath.toFile()), BUFFER_SIZE)) {
byte[] buffer = new byte[BUFFER_SIZE];
int len;
while ((len = zis.read(buffer)) > 0) {
bos.write(buffer, 0, len);
}
}
}
processedEntries++;
if (callback != null) {
callback.onProgress(processedEntries, totalEntries, entry.getName());
}
zis.closeEntry();
}
}
log.info("Extract completed: {} entries processed", processedEntries);
if (callback != null) {
callback.onComplete(processedEntries);
}
} catch (IOException e) {
log.error("Failed to extract file", e);
throw new FileExtractException("Failed to extract file", e);
}
}
}
```
#### 5. PathValidator.java
```java
package com.datai.salesforce.metadata.file.util;
import com.datai.salesforce.metadata.file.exception.PathSecurityException;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.regex.Pattern;
@Slf4j
@Component
public class PathValidator {
private static final Pattern FILENAME_PATTERN = Pattern.compile(
"^[a-zA-Z0-9._\\-]+$"
);
private static final Pattern PATH_PATTERN = Pattern.compile(
"^[a-zA-Z0-9._\\-/\\\\]+$"
);
public void validatePath(Path basePath, Path targetPath) {
Path normalizedPath = targetPath.normalize();
if (!normalizedPath.startsWith(basePath.normalize())) {
log.error("Path traversal attempt detected: {}", targetPath);
throw new PathSecurityException("Path traversal attempt detected");
}
if (targetPath.toString().contains("..")) {
log.error("Path contains '..': {}", targetPath);
throw new PathSecurityException("Path contains '..'");
}
String fileName = targetPath.getFileName().toString();
if (!FILENAME_PATTERN.matcher(fileName).matches()) {
log.error("Invalid filename: {}", fileName);
throw new PathSecurityException("Invalid filename");
}
}
public void validatePathString(String path) {
if (!PATH_PATTERN.matcher(path).matches()) {
log.error("Invalid path: {}", path);
throw new PathSecurityException("Invalid path");
}
if (path.contains("..")) {
log.error("Path contains '..': {}", path);
throw new PathSecurityException("Path contains '..'");
}
}
public Path normalizePath(Path path) {
return path.normalize();
}
}
```
#### 6. StorageMonitorServiceImpl.java
```java
package com.datai.salesforce.metadata.file.service;
import com.datai.salesforce.metadata.file.dto.StorageSpaceInfo;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.io.File;
import java.nio.file.FileStore;
import java.nio.file.Files;
@Slf4j
@Service
public class StorageMonitorServiceImpl implements IStorageMonitorService {
@Autowired
private INotificationService notificationService;
@Value("${file.storage.warning-threshold:80}")
private int warningThreshold;
@Value("${file.storage.local.base-path}")
private String basePath;
@Override
public StorageSpaceInfo getStorageSpaceInfo() {
try {
File file = new File(basePath);
FileStore store = Files.getFileStore(file.toPath());
long totalSpace = store.getTotalSpace();
long usableSpace = store.getUsableSpace();
long usedSpace = totalSpace - usableSpace;
double usagePercentage = (double) usedSpace / totalSpace * 100;
StorageSpaceInfo info = new StorageSpaceInfo();
info.setTotalSpace(totalSpace);
info.setUsedSpace(usedSpace);
info.setUsableSpace(usableSpace);
info.setUsagePercentage(usagePercentage);
info.setWarning(usagePercentage >= warningThreshold);
return info;
} catch (IOException e) {
log.error("Failed to get storage space info", e);
throw new RuntimeException("Failed to get storage space info", e);
}
}
@Scheduled(cron = "${file.storage.monitor.cron:0 0 * * * ?}")
public void monitorStorageSpace() {
StorageSpaceInfo info = getStorageSpaceInfo();
if (info.isWarning()) {
log.warn("Storage space usage is high: {}%", info.getUsagePercentage());
notificationService.sendNotification(
"Storage space warning",
String.format("Storage space usage is %.2f%%", info.getUsagePercentage())
);
}
}
}
```
#### 7. FileCleanupServiceImpl.java
```java
package com.datai.salesforce.metadata.file.service;
import com.datai.salesforce.metadata.file.dto.FileCleanupRequest;
import com.datai.salesforce.metadata.file.dto.FileCleanupResponse;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
@Slf4j
@Service
public class FileCleanupServiceImpl implements IFileCleanupService {
@Autowired
private IFileStorageService fileStorageService;
@Value("${file.storage.local.base-path}")
private String basePath;
@Value("${file.cleanup.retention-days:30}")
private int retentionDays;
@Override
public FileCleanupResponse cleanupFiles(FileCleanupRequest request) {
FileCleanupResponse response = new FileCleanupResponse();
List<String> deletedFiles = new ArrayList<>();
long totalSize = 0;
try {
Path baseDir = Paths.get(basePath);
Instant cutoffTime = Instant.now().minus(retentionDays, ChronoUnit.DAYS);
Files.walk(baseDir)
.filter(Files::isRegularFile)
.filter(path -> {
try {
FileTime lastModifiedTime = Files.getLastModifiedTime(path);
return lastModifiedTime.toInstant().isBefore(cutoffTime);
} catch (IOException e) {
log.error("Failed to get last modified time for: {}", path, e);
return false;
}
})
.forEach(path -> {
try {
long size = Files.size(path);
if (fileStorageService.delete(path.toString())) {
deletedFiles.add(path.toString());
totalSize += size;
log.info("Deleted file: {}", path);
}
} catch (IOException e) {
log.error("Failed to delete file: {}", path, e);
}
});
response.setDeletedCount(deletedFiles.size());
response.setDeletedFiles(deletedFiles);
response.setTotalSize(totalSize);
response.setSuccess(true);
log.info("File cleanup completed: {} files deleted, total size: {} bytes",
deletedFiles.size(), totalSize);
} catch (IOException e) {
log.error("Failed to cleanup files", e);
response.setSuccess(false);
response.setErrorMessage(e.getMessage());
}
return response;
}
@Scheduled(cron = "${file.cleanup.cron:0 0 2 * * ?}")
public void scheduledCleanup() {
log.info("Starting scheduled file cleanup");
FileCleanupRequest request = new FileCleanupRequest();
request.setRetentionDays(retentionDays);
cleanupFiles(request);
}
}
```
#### 8. application.yml
```yaml
file:
storage:
type: local # local or oss
local:
base-path: /data/metadata
oss:
endpoint: https://oss-cn-hangzhou.aliyuncs.com
access-key-id: ${OSS_ACCESS_KEY_ID}
access-key-secret: ${OSS_ACCESS_KEY_SECRET}
bucket-name: datai-metadata
base-path: metadata/
warning-threshold: 80
monitor:
cron: "0 0 * * * ?"
cleanup:
retention-days: 30
cron: "0 0 2 * * ?"
```
## 约束
### 技术栈约束
- 必须基于 Spring Boot 3
- 必须使用 Java 17+
- 必须使用 MyBatis Plus
- 必须使用 MySQL
### 性能约束
- 文件存储速度:本地文件系统 > 50MB/sOSS > 20MB/s
- 文件解压速度 > 30MB/s
- 内存使用 < 512MB
- 支持大文件处理> 100MB
### 安全性约束
- 必须验证路径安全性,防止路径遍历攻击
- 必须验证文件名格式,防止文件名注入攻击
- 必须限制文件大小,防止 DoS 攻击
- 必须记录所有文件操作日志
### 兼容性约束
- 必须支持 Windows 和 Linux 操作系统
- 必须支持 Java NIO 的 Path API
- 必须支持 OSS SDK
## Rule Set
"请严格参考 @Authentication.canvas 中的状态机转移逻辑,不要自行发挥。"
**具体规则**
- 必须使用 Canvas 中定义的类名和方法名
- 必须遵循 Canvas 中定义的调用关系
- 必须参考 Canvas 中的流程图逻辑
- 必须使用策略模式实现文件存储
- 必须使用流式处理实现文件解压
- 必须使用 Path API 处理路径
- 必须使用定时任务实现存储空间监控和文件清理
## 验收标准
### 功能验收标准
- [ ] 本地文件系统存储成功
- [ ] OSS 存储成功
- [ ] 文件解压成功
- [ ] 支持大文件处理(> 100MB
- [ ] 支持流式解压
- [ ] 内存使用合理(< 512MB
- [ ] 文件路径管理正常工作
- [ ] 路径安全性验证通过
- [ ] 存储空间监控正常工作
- [ ] 文件清理功能正常工作
### 性能验收标准
- [ ] 本地文件系统存储速度 > 50MB/s
- [ ] OSS 存储速度 > 20MB/s
- [ ] 解压速度 > 30MB/s
- [ ] 并发存储 10 个文件无性能下降
- [ ] 并发解压 5 个文件无性能下降
### 安全性验收标准
- [ ] 防止路径遍历攻击
- [ ] 防止文件名注入攻击
- [ ] 防止符号链接攻击
- [ ] 文件操作日志记录完整
### 代码质量验收标准
- [ ] 代码符合项目编码规范
- [ ] 有清晰的注释
- [ ] 单元测试覆盖率 > 80%
- [ ] 集成测试覆盖主要场景
- [ ] 通过代码审查
## 风险
### 大文件处理风险
- **风险描述**: 大文件处理不当可能导致内存溢出
- **缓解措施**: 使用流式处理,限制内存占用
- **监控指标**: 监控 JVM 内存使用情况
### 路径安全风险
- **风险描述**: 路径验证不完善可能导致路径遍历攻击
- **缓解措施**: 使用 Path API 和正则表达式双重验证
- **监控指标**: 记录所有文件操作日志
### OSS 存储风险
- **风险描述**: OSS 存储不稳定可能导致文件存储失败
- **缓解措施**: 实现重试机制和降级策略
- **监控指标**: 监控 OSS 存储成功率和响应时间
### 存储空间风险
- **风险描述**: 存储空间不足可能导致文件存储失败
- **缓解措施**: 实现存储空间监控和告警机制
- **监控指标**: 监控存储空间使用率
### 文件清理风险
- **风险描述**: 文件清理策略不当可能导致重要文件被删除
- **缓解措施**: 清理前记录日志,支持文件恢复
- **监控指标**: 记录清理操作日志
## 使用记录
| 日期 | 使用场景 | 输入参数 | 输出结果 | 反馈 | 改进措施 |
|------|---------|---------|---------|------|----------|
| 2026-01-19 | 实现文件存储和解压处理功能 | REQ-010-7 需求文档、ADR-0016 决策文档 | 完整的代码实现和测试用例 | 待使用 | 待改进 |