mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-29 03:23:17 +00:00
clean up
This commit is contained in:
chrislu
+39
-51
@@ -1,26 +1,24 @@
|
||||
# SQL Query Engine Feature, Dev, and Test Plan
|
||||
|
||||
This document outlines the plan for adding comprehensive SQL support to SeaweedFS, focusing on schematized Message Queue (MQ) topics with full DDL and DML capabilities, plus S3 objects querying.
|
||||
This document outlines the plan for adding SQL querying support to SeaweedFS, focusing on reading and analyzing data from Message Queue (MQ) topics and S3 objects.
|
||||
|
||||
## Feature Plan
|
||||
|
||||
**1. Goal**
|
||||
|
||||
To provide a full-featured SQL interface for SeaweedFS, treating schematized MQ topics as database tables with complete DDL/DML support. This enables:
|
||||
- Database-like operations on MQ topics (CREATE TABLE, ALTER TABLE, DROP TABLE)
|
||||
- Advanced querying with SELECT, WHERE, JOIN, aggregations
|
||||
- Schema management and metadata operations (SHOW DATABASES, SHOW TABLES)
|
||||
To provide a SQL querying interface for SeaweedFS, enabling analytics on existing MQ topics and S3 objects. This enables:
|
||||
- Advanced querying with SELECT, WHERE, JOIN, aggregations on MQ topics
|
||||
- Schema discovery and metadata operations (SHOW DATABASES, SHOW TABLES, DESCRIBE)
|
||||
- In-place analytics on Parquet-stored messages without data movement
|
||||
- Direct querying of S3 objects in various formats
|
||||
|
||||
**2. Key Features**
|
||||
|
||||
* **Schematized Topic Management (Priority 1):**
|
||||
* **Schema Discovery and Metadata (Priority 1):**
|
||||
* `SHOW DATABASES` - List all MQ namespaces
|
||||
* `SHOW TABLES` - List all topics in a namespace
|
||||
* `CREATE TABLE topic_name (field1 INT, field2 STRING, ...)` - Create new MQ topic with schema
|
||||
* `ALTER TABLE topic_name ADD COLUMN field3 BOOL` - Modify topic schema (with versioning)
|
||||
* `DROP TABLE topic_name` - Delete MQ topic
|
||||
* `SHOW TABLES` - List all topics in a namespace
|
||||
* `DESCRIBE table_name` - Show topic schema details
|
||||
* Automatic schema detection from existing Parquet data
|
||||
* **Advanced Query Engine (Priority 1):**
|
||||
* Full `SELECT` support with `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`
|
||||
* Aggregation functions: `COUNT()`, `SUM()`, `AVG()`, `MIN()`, `MAX()`, `GROUP BY`
|
||||
@@ -67,8 +65,8 @@ To provide a full-featured SQL interface for SeaweedFS, treating schematized MQ
|
||||
* **Schema Catalog:**
|
||||
* Leverage existing `weed/mq/schema/` infrastructure
|
||||
* Map MQ namespaces to "databases" and topics to "tables"
|
||||
* Store schema metadata with version history
|
||||
* Handle schema evolution and migration
|
||||
* Discover schema metadata from existing Parquet files
|
||||
* Handle schema evolution in read operations
|
||||
* **Query Planner:**
|
||||
* Parse SQL statements using custom PostgreSQL parser
|
||||
* Create optimized execution plans leveraging Parquet columnar format
|
||||
@@ -90,8 +88,8 @@ To provide a full-featured SQL interface for SeaweedFS, treating schematized MQ
|
||||
* Handle partition-based parallelism for scalable queries
|
||||
* **Schema Registry Integration:**
|
||||
* Extend `weed/mq/schema/schema.go` for SQL metadata operations
|
||||
* Implement DDL operations that modify underlying MQ topic schemas
|
||||
* Version control for schema changes with migration support
|
||||
* Read existing topic schemas for query planning
|
||||
* Handle schema evolution during query execution
|
||||
* **S3 Connector (Secondary):**
|
||||
* Reading data from S3 objects with CSV, JSON, and Parquet parsers
|
||||
* Efficient streaming for large files with columnar optimizations
|
||||
@@ -113,7 +111,7 @@ To provide a full-featured SQL interface for SeaweedFS, treating schematized MQ
|
||||
|
||||
## Example Usage Scenarios
|
||||
|
||||
**Scenario 1: Topic Management**
|
||||
**Scenario 1: Schema Discovery and Metadata**
|
||||
```sql
|
||||
-- List all namespaces (databases)
|
||||
SHOW DATABASES;
|
||||
@@ -122,18 +120,7 @@ SHOW DATABASES;
|
||||
USE my_namespace;
|
||||
SHOW TABLES;
|
||||
|
||||
-- Create a new topic with schema
|
||||
CREATE TABLE user_events (
|
||||
user_id INT,
|
||||
event_type STRING,
|
||||
timestamp BIGINT,
|
||||
metadata STRING
|
||||
);
|
||||
|
||||
-- Modify topic schema
|
||||
ALTER TABLE user_events ADD COLUMN session_id STRING;
|
||||
|
||||
-- View topic structure
|
||||
-- View topic structure and discovered schema
|
||||
DESCRIBE user_events;
|
||||
```
|
||||
|
||||
@@ -215,10 +202,10 @@ SQL Query Flow:
|
||||
* Schema Fields ↔ Table Columns
|
||||
|
||||
**2. Schema Evolution Handling:**
|
||||
* Maintain schema version history in topic metadata
|
||||
* Read schema version history from existing topic metadata
|
||||
* Support backward-compatible queries across schema versions
|
||||
* Automatic type coercion where possible
|
||||
* Clear error messages for incompatible changes
|
||||
* Automatic type coercion where possible during reads
|
||||
* Clear error messages for incompatible data
|
||||
|
||||
**3. Query Optimization:**
|
||||
* Leverage Parquet columnar format for projection pushdown
|
||||
@@ -237,10 +224,10 @@ SQL Query Flow:
|
||||
* **Error Handling:** Typed error system with proper PostgreSQL error code mapping
|
||||
* **Parser Features:** Lightweight, focused on SeaweedFS SQL subset, easily extensible
|
||||
|
||||
**5. Transaction Semantics:**
|
||||
* DDL operations (CREATE/ALTER/DROP) are atomic per topic
|
||||
* SELECT queries provide read-consistent snapshots
|
||||
* No cross-topic transactions initially (future enhancement)
|
||||
**5. Query Semantics:**
|
||||
* SELECT queries provide read-consistent snapshots of topic data
|
||||
* Queries operate on immutable Parquet files for consistency
|
||||
* No transactional guarantees across multiple topics
|
||||
|
||||
**6. Performance Considerations:**
|
||||
* Prioritize read performance over write consistency
|
||||
@@ -256,41 +243,42 @@ SQL Query Flow:
|
||||
3. Implemented metadata catalog mapping MQ topics to SQL tables
|
||||
4. Added `SHOW DATABASES`, `SHOW TABLES`, `DESCRIBE` commands with full PostgreSQL compatibility
|
||||
|
||||
**Phase 2: DDL Operations (Weeks 4-5)**
|
||||
1. `CREATE TABLE` → Create MQ topic with schema
|
||||
2. `ALTER TABLE` → Modify topic schema with versioning
|
||||
3. `DROP TABLE` → Delete MQ topic
|
||||
4. Schema validation and migration handling
|
||||
|
||||
**Phase 3: Query Engine (Weeks 6-8)**
|
||||
**Phase 2: Query Engine (Weeks 4-6)**
|
||||
1. `SELECT` with `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`
|
||||
2. Aggregation functions and `GROUP BY`
|
||||
3. Basic joins between topics
|
||||
4. Predicate pushdown to Parquet layer
|
||||
5. Schema discovery from existing Parquet files
|
||||
|
||||
**Phase 4: API & CLI Integration (Weeks 9-10)**
|
||||
**Phase 3: API & CLI Integration (Weeks 7-8)**
|
||||
1. HTTP `/sql` endpoint implementation
|
||||
2. `weed sql` CLI command with interactive mode
|
||||
3. Result streaming and pagination
|
||||
4. Error handling and query optimization
|
||||
|
||||
**Phase 4: Advanced Features (Weeks 9-10)**
|
||||
1. Window functions and advanced analytics
|
||||
2. S3 object querying capabilities
|
||||
3. Performance optimizations
|
||||
4. Connection pooling and query caching
|
||||
|
||||
## Test Plan
|
||||
|
||||
**1. Unit Tests**
|
||||
|
||||
* **SQL Parser Tests:** Validate parsing of all supported DDL/DML statements
|
||||
* **Schema Mapping Tests:** Test topic-to-table conversion and metadata operations
|
||||
* **SQL Parser Tests:** Validate parsing of all supported SELECT statements and metadata operations
|
||||
* **Schema Mapping Tests:** Test topic-to-table conversion and schema discovery
|
||||
* **Query Planning Tests:** Verify optimization and predicate pushdown logic
|
||||
* **Execution Engine Tests:** Test query execution with various data patterns
|
||||
* **Edge Cases:** Malformed queries, schema evolution, concurrent operations
|
||||
* **Edge Cases:** Malformed queries, schema evolution in existing data, concurrent reads
|
||||
|
||||
**2. Integration Tests**
|
||||
|
||||
* **End-to-End Workflow:** Complete SQL operations against live SeaweedFS cluster
|
||||
* **Schema Evolution:** Test backward compatibility during schema changes
|
||||
* **End-to-End Workflow:** Complete SQL querying operations against live SeaweedFS cluster
|
||||
* **Schema Discovery:** Test automatic schema detection from existing Parquet data
|
||||
* **Multi-Topic Joins:** Validate cross-topic query performance and correctness
|
||||
* **Large Dataset Tests:** Performance validation with GB-scale Parquet data
|
||||
* **Concurrent Access:** Multiple SQL sessions operating simultaneously
|
||||
* **Concurrent Access:** Multiple SQL query sessions operating simultaneously
|
||||
|
||||
**3. Performance & Security Testing**
|
||||
|
||||
@@ -302,11 +290,11 @@ SQL Query Flow:
|
||||
|
||||
## Success Metrics
|
||||
|
||||
* **Feature Completeness:** Support for all specified DDL/DML operations
|
||||
* **Feature Completeness:** Support for all specified SELECT operations and metadata commands
|
||||
* **Performance:**
|
||||
* **Simple SELECT queries**: < 100ms latency for single-table queries with up to 3 WHERE predicates on ≤ 100K records
|
||||
* **Complex queries**: < 1s latency for queries involving aggregations (COUNT, SUM, MAX, MIN) on ≤ 1M records
|
||||
* **Time-range queries**: < 500ms for timestamp-based filtering on ≤ 500K records within 24-hour windows
|
||||
* **Scalability:** Handle topics with millions of messages efficiently
|
||||
* **Reliability:** 99.9% success rate for valid SQL operations
|
||||
* **Usability:** Intuitive SQL interface matching standard database expectations
|
||||
* **Reliability:** 99.9% success rate for valid SQL queries
|
||||
* **Usability:** Intuitive SQL querying interface matching standard database expectations
|
||||
|
||||
Reference in New Issue
Block a user