- Implement PrefetchManager with configurable worker pool and deduplication
- Add AccessPatternDetector for sequential, strided, and ML-specific patterns
- Create MLReaderCache with ML-aware prefetching capabilities
- Add comprehensive unit tests for prefetch manager
- Include foundation for detecting training datasets, model loading, and epoch patterns
- Support configurable prefetch parameters optimized for ML workloads
Features:
- Concurrent prefetch workers (8 by default)
- Pattern detection for sequential, model, epoch, and strided access
- ML-specific heuristics for large file and dataset access
- Comprehensive metrics and monitoring
- Graceful shutdown and cleanup
Tests:
- PrefetchManager: All tests passing (9/9)
- AccessPatternDetector: Core functionality implemented
- MLReaderCache: Basic functionality and integration tests
chrislu
e76f632907