Commit Graph

11928 Commits

Author SHA1 Message Date
chrislu
3570027656 feat: Enable publishers to use Parquet logical types
Enhanced MQ publishers to utilize the new logical types:
- Updated convertToRecordValue() to use TimestampValue instead of string RFC3339
- Added DateValue support for birth_date field (days since epoch)
- Added DecimalValue support for precise_amount field with configurable precision/scale
- Enhanced UserEvent struct with PreciseAmount and BirthDate fields
- Added convertToDecimal() helper using big.Rat for precise decimal conversion
- Updated test data generator to produce varied birth dates (1970-2005) and precise amounts

Publishers now generate structured data with proper logical types:
-  TIMESTAMP: Microsecond precision UTC timestamps
-  DATE: Birth dates as days since Unix epoch
-  DECIMAL: Precise amounts with 18-digit precision, 4-decimal scale

Successfully tested with PostgreSQL integration - all topics created with logical type data.
2025-09-03 07:26:36 -07:00
chrislu
ec1e74a6e8 feat: Add Parquet logical types to mq_schema.proto
Added support for Parquet logical types in SeaweedFS message queue schema:
- TIMESTAMP: UTC timestamp in microseconds since epoch with timezone flag
- DATE: Date as days since Unix epoch (1970-01-01)
- DECIMAL: Arbitrary precision decimal with configurable precision/scale
- TIME: Time of day in microseconds since midnight

These types enable advanced analytics features:
- Time-based filtering and window functions
- Date arithmetic and year/month/day extraction
- High-precision numeric calculations
- Proper time zone handling for global deployments

Regenerated protobuf Go code with new scalar types and value messages.
2025-09-03 07:18:58 -07:00
chrislu
d60c542ecc feat: Replace pg_query_go with lightweight SQL parser (no CGO required)
- Remove github.com/pganalyze/pg_query_go/v6 dependency to avoid CGO requirement
- Implement lightweight SQL parser for basic SELECT, SHOW, and DDL statements
- Fix operator precedence in WHERE clause parsing (handle AND/OR before comparisons)
- Support INTEGER, FLOAT, and STRING literals in WHERE conditions
- All SQL engine tests passing with new parser
- PostgreSQL integration tests can now build without CGO

The lightweight parser handles the essential SQL features needed for the
SeaweedFS query engine while maintaining compatibility and avoiding CGO
dependencies that caused Docker build issues.
2025-09-03 07:11:18 -07:00
chrislu
88d86374ea fix: Enable CGO in Docker build for pg_query_go dependency
The pg_query_go library requires CGO to be enabled as it wraps the libpg_query C library.
Added gcc and musl-dev dependencies to the Docker build for proper compilation.
2025-09-03 00:59:11 -07:00
chrislu
4d9de40c5c fmt 2025-09-03 00:48:09 -07:00
chrislu
42661ac110 fix tests 2025-09-03 00:47:08 -07:00
chrislu
991247facf fix tests 2025-09-03 00:40:03 -07:00
chrislu
e3e369c264 change to pg_query_go 2025-09-03 00:10:47 -07:00
chrislu
ba4a8b91d5 fmt 2025-09-02 22:31:53 -07:00
chrislu
59d6806146 fix empty spaces and coercion 2025-09-02 22:30:52 -07:00
Chris Lu
f29dd385cc Merge branch 'master' into add-sql-querying 2025-09-02 22:14:21 -07:00
chrislu
3fa7670557 fix todo 2025-09-02 22:12:47 -07:00
chrislu
687c5d6bfd fix tests 2025-09-02 21:21:59 -07:00
chrislu
e14a316aeb use schema instead of inferred result types 2025-09-02 20:59:13 -07:00
chrislu
316d1cdda7 address some comments 2025-09-02 19:58:41 -07:00
chrislu
a7eb178cec Update engine.go 2025-09-02 18:37:31 -07:00
chrislu
60066a6a4c read broker, logs, and parquet files 2025-09-02 18:15:26 -07:00
chrislu
59ec4eb68a address comments 2025-09-02 17:37:52 -07:00
chrislu
d29f54e0be de-support alter table and drop table 2025-09-02 17:23:46 -07:00
chrislu
8e15fdf2c7 remove sample data 2025-09-02 17:18:19 -07:00
chrislu
f776a49322 avoid sample data 2025-09-02 17:15:13 -07:00
chrislu
ed7102df6e column name can be on left or right in where conditions 2025-09-02 17:05:31 -07:00
chrislu
900bd94456 address some comments 2025-09-02 16:54:34 -07:00
chrislu
e901abffd3 address comments 2025-09-02 15:40:38 -07:00
chrislu
ac83adc6ce fix versioning 2025-09-02 15:14:46 -07:00
chrislu
586a795b02 return fast on error 2025-09-02 14:09:38 -07:00
chrislu
1a2225c0f5 rename to "weed db" 2025-09-02 11:02:08 -07:00
chrislu
c9e093194d setup integration test for postgres 2025-09-02 10:51:37 -07:00
chrislu
8b4914be55 hook up seaweed sql engine 2025-09-02 08:42:36 -07:00
chrislu
5a8c31fcda Revert "add jdbc"
This reverts commit a6e48b7690.
2025-09-02 08:25:39 -07:00
chrislu
85306bb2b0 add postgres protocol 2025-09-02 08:25:29 -07:00
chrislu
a6e48b7690 add jdbc 2025-09-02 06:56:49 -07:00
chrislu
6fb88a8edb buffer start stored as 8 bytes 2025-09-02 00:57:48 -07:00
chrislu
61bacd23b0 parquet file can query messages in broker memory, if log files do not exist 2025-09-02 00:49:12 -07:00
chrislu
618cb89885 the parquet file should also remember the first buffer_start attribute from the sources 2025-09-02 00:42:37 -07:00
chrislu
db75742e37 explain with broker buffer 2025-09-02 00:35:52 -07:00
chrislu
de866bfd09 dedup with buffer start index 2025-09-02 00:26:35 -07:00
chrislu
e3a56d7c30 filter out already flushed messages 2025-09-02 00:10:07 -07:00
chrislu
467034c8c7 process buffer from brokers 2025-09-01 23:45:52 -07:00
chrislu
7ca3b59c44 save source buffer index start for log files 2025-09-01 23:33:35 -07:00
chrislu
f5ed25f755 fix decoding data 2025-09-01 22:17:17 -07:00
chrislu
99bfe95e51 detailed logs during explain. Fix bugs on reading live logs. 2025-09-01 22:16:46 -07:00
chrislu
c7a0b89067 fix after refactoring 2025-09-01 21:40:24 -07:00
chrislu
e385f0ce7d refactor 2025-09-01 20:19:59 -07:00
chrislu
61ad3c39ac add tests 2025-09-01 20:05:24 -07:00
chrislu
4214d765cf use mock for testing 2025-09-01 19:36:47 -07:00
chrislu
a3f6a5da27 skip 2025-09-01 18:55:14 -07:00
chrislu
dfd0897e49 improve tests 2025-09-01 18:52:22 -07:00
chrislu
7d88a81482 add tests 2025-09-01 18:00:55 -07:00
chrislu
eaa7136c92 explain the execution plan 2025-09-01 17:19:36 -07:00