- Modify HEAD object logic to return 404 only for PyArrow-style directory markers
(0-byte files with generic MIME types that have children)
- Maintain AWS S3 compatibility by allowing actual directories to return 200
- Update tests to reflect the new targeted logic
- This provides better compatibility with both PyArrow and AWS S3 clients