11 Commits

Author SHA1 Message Date
Yoshiyuki Mineo
f167a9d531 Add Compose method and corresponding tests (#79)
* Implemented the Compose function to create a Sonyflake ID from its components, including validation for start time, sequence, and machine ID.
* Added a new test, TestCompose, to verify the functionality of the Compose method, ensuring correct decomposition of generated IDs.
* Introduced a new error for invalid sequence numbers to enhance error handling in the Sonyflake package.
2025-07-05 17:50:03 +09:00
Yoshiyuki Mineo
5c401f9c06 Make unit tests stabler (#78)
* Refactor Sonyflake to use a customizable time function for improved testability. Update currentElapsedTime and sleep methods to utilize the new time function instead of directly calling time.Now().

* Update TimeUnit in ToTime test to use time.Millisecond for improved accuracy in timestamp validation. Adjust time duration comparison to ensure correct validation of generated timestamps.

* Refactor TestNextID to use a customizable time function for improved accuracy in timestamp validation. Adjust time comparison to ensure expected results in generated IDs.

* Refactor TestNextID_InSequence to use a consistent start time for improved clarity in timestamp validation. Update max sequence comparison to ensure accurate validation of generated IDs.

* Refactor tests in sonyflake_test.go to replace fmt.Println with t.Log for better test output management. Update error handling to use errors.New for consistency.
2025-06-28 19:24:22 +09:00
Yoshiyuki Mineo
0cdef9e4fe Update TimeUnit in ToTime test to use 100 milliseconds for improved accuracy in timestamp validation. (#77) 2025-06-23 23:07:41 +09:00
Yoshiyuki Mineo
114716564a Fix time duration comparison in ToTime test to ensure correct validation of generated timestamps. (#76) 2025-06-23 21:45:29 +09:00
Yoshiyuki Mineo
2343cac676 Check compose args (#74)
* v2: take only 'bitsMachine' least significant bits from generated machine IDs so it doesn't corrupt the other ID parts (#72)

Co-authored-by: ok32 <e.starikoff@gmail.com>
Co-authored-by: Yoshiyuki Mineo <Yoshiyuki.Mineo@jp.sony.com>

* Refactor Sonyflake Compose method to return errors for invalid parameters and update related tests. Consolidate error handling for start time, sequence, and machine ID validations. Remove unused variable in tests for clarity.

---------

Co-authored-by: ok32 <artuh.gubanova@gmail.com>
Co-authored-by: ok32 <e.starikoff@gmail.com>
2025-05-18 15:56:36 +09:00
Yoshiyuki Mineo
5347433c8c Enhance Sonyflake error handling by adding tests for invalid machine IDs, including cases for too large and negative values. (#73) 2025-05-18 15:16:04 +09:00
Yoshiyuki Mineo
774342570a Add Compose method (#71) 2025-05-07 13:17:30 +09:00
Yoshiyuki Mineo
59c47aeab1 Fix lint errors (#70) 2025-05-05 19:51:28 +09:00
Yoshiyuki Mineo
d764be18d5 Update README.md (#69)
* Update README.md

* Update README.md

* Update test-v2.yml

* Update test-v1.yml
2025-05-05 14:46:40 +09:00
Yoshiyuki Mineo
7ee8f154df feat(v2): make bit assignment for time/sequence/machine customizable … (#68)
* feat(v2): make bit assignment for time/sequence/machine customizable via Settings; update all logic and tests

* gofmt
2025-05-05 14:16:21 +09:00
Yoshiyuki Mineo
357b2ee5f0 Upgrade Sonyflake to v2 (#66)
* Add v2

* Introduce staticcheck

* Introduce golangci-lint

* Add error checks

* Add error checks

* Lint v2 code

* Improve CI trigger

* Use io.ReadAll

* Use int64

* Remove NewSonyflake

* Fix errors

* v2: Change MachineID, sequence, and AmazonEC2MachineID to int; update all usage and tests for type consistency

* docs: update Settings struct in README to use int for MachineID and CheckMachineID (v2)

* docs(v2): clarify Settings, StartTime, MachineID, and CheckMachineID comments and update README links and explanations

* docs(v2/mock): improve comments and docstrings for mock implementations

* docs(types): unify and clarify package and type docstrings for types.go in v1 and v2

* test(v2): refactor and modernize tests, improve error assertions, and update mocks for v2

* test(v2): normalize whitespace in pseudoSleep calls for consistency

* feat(v2): add configurable TimeUnit and refactor time handling for So… (#67)

* feat(v2): add configurable TimeUnit and refactor time handling for Sonyflake v2

* test(v2): add ToTime tests, clarify TimeUnit behavior, and update docs for v2

* gofmt
2025-05-05 10:53:08 +09:00
23 changed files with 1244 additions and 81 deletions

49
.github/workflows/test-v1.yml vendored Normal file
View File

@@ -0,0 +1,49 @@
name: CI for v1 Module
on:
push:
paths-ignore:
- README.md
- .github/workflows/test-v2.yml
- v2/**
pull_request:
paths-ignore:
- README.md
- .github/workflows/test-v2.yml
- v2/**
permissions: read-all
jobs:
test-v1:
strategy:
matrix:
go-version: [1.23.x, 1.24.x]
os: [ubuntu-latest]
runs-on: ${{matrix.os}}
defaults:
run:
working-directory: ./
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{matrix.go-version}}
- name: Lint Code
uses: golangci/golangci-lint-action@v6
- name: Check format
run: test -z "`gofmt -l .`"
- name: Run tests
run: go test -v ./...
- name: Build example
run: cd example && ./linux64_build.sh

54
.github/workflows/test-v2.yml vendored Normal file
View File

@@ -0,0 +1,54 @@
name: CI for v2 Module
on:
push:
paths:
- README.md
- .github/workflows/test-v2.yml
- v2/**
pull_request:
paths:
- README.md
- .github/workflows/test-v2.yml
- v2/**
permissions: read-all
jobs:
test-v2:
strategy:
matrix:
go-version: [1.23.x, 1.24.x]
os: [ubuntu-latest]
runs-on: ${{matrix.os}}
defaults:
run:
working-directory: ./v2
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: ./v2/go.mod
- name: Tidy modules
run: go mod tidy
- name: Lint Code
uses: golangci/golangci-lint-action@v6
with:
working-directory: ./v2
- name: Check format
run: test -z "`gofmt -l .`"
- name: Run tests
run: go test ./... -v
- name: Build example
run: cd example && ./linux64_build.sh

View File

@@ -1,24 +0,0 @@
on: [push, pull_request]
name: Test
jobs:
test:
strategy:
matrix:
go-version: [1.23.x, 1.24.x]
os: [ubuntu-latest]
runs-on: ${{matrix.os}}
steps:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: ${{matrix.go-version}}
- name: Checkout
uses: actions/checkout@v2
- name: gofmt
run: test -z "`gofmt -l .`"
- name: golint
run: test -z "`golint ./...`"
- name: go test
run: go test -v ./...
- name: Build example
run: cd example && ./linux64_build.sh

3
.gitignore vendored
View File

@@ -1 +1,4 @@
example/example
example/sonyflake_server
v2/example/example
v2/example/sonyflake_server

View File

@@ -1,14 +1,13 @@
Sonyflake
=========
# Sonyflake
[![GoDoc](https://godoc.org/github.com/sony/sonyflake?status.svg)](http://godoc.org/github.com/sony/sonyflake)
[![Go Report Card](https://goreportcard.com/badge/github.com/sony/sonyflake)](https://goreportcard.com/report/github.com/sony/sonyflake)
[![GoDoc](https://pkg.go.dev/badge/github.com/sony/sonyflake/v2?utm_source=godoc)](https://pkg.go.dev/github.com/sony/sonyflake/v2)
[![Go Report Card](https://goreportcard.com/badge/github.com/sony/sonyflake/v2)](https://goreportcard.com/report/github.com/sony/sonyflake/v2)
Sonyflake is a distributed unique ID generator inspired by [Twitter's Snowflake](https://blog.twitter.com/2010/announcing-snowflake).
Sonyflake focuses on lifetime and performance on many host/core environment.
So it has a different bit assignment from Snowflake.
A Sonyflake ID is composed of
By default, a Sonyflake ID is composed of
39 bits for time in units of 10 msec
8 bits for a sequence number
@@ -18,20 +17,21 @@ As a result, Sonyflake has the following advantages and disadvantages:
- The lifetime (174 years) is longer than that of Snowflake (69 years)
- It can work in more distributed machines (2^16) than Snowflake (2^10)
- It can generate 2^8 IDs per 10 msec at most in a single machine/thread (slower than Snowflake)
- It can generate 2^8 IDs per 10 msec at most in a single instance (fewer than Snowflake)
However, if you want more generation rate in a single host,
you can easily run multiple Sonyflake ID generators concurrently using goroutines.
you can easily run multiple Sonyflake instances parallelly using goroutines.
Installation
------------
In addition, you can adjust the lifetime and generation rate of Sonyflake
by customizing the bit assignment and the time unit.
## Installation
```
go get github.com/sony/sonyflake
go get github.com/sony/sonyflake/v2
```
Usage
-----
## Usage
The function New creates a new Sonyflake instance.
@@ -43,42 +43,54 @@ You can configure Sonyflake by the struct Settings:
```go
type Settings struct {
BitsSequence int
BitsMachineID int
TimeUnit time.Duration
StartTime time.Time
MachineID func() (uint16, error)
CheckMachineID func(uint16) bool
MachineID func() (int, error)
CheckMachineID func(int) bool
}
```
- BitsSequence is the bit length of a sequence number.
If BitsSequence is 0, the default bit length is used, which is 8.
If BitsSequence is 31 or more, an error is returned.
- BitsMachineID is the bit length of a machine ID.
If BitsMachineID is 0, the default bit length is used, which is 16.
If BitsMachineID is 31 or more, an error is returned.
- TimeUnit is the time unit of Sonyflake.
If TimeUnit is 0, the default time unit is used, which is 10 msec.
TimeUnit must be 1 msec or longer.
- StartTime is the time since which the Sonyflake time is defined as the elapsed time.
If StartTime is 0, the start time of the Sonyflake is set to "2014-09-01 00:00:00 +0000 UTC".
If StartTime is ahead of the current time, Sonyflake is not created.
If StartTime is 0, the start time of the Sonyflake instance is set to "2025-01-01 00:00:00 +0000 UTC".
StartTime must be before the current time.
- MachineID returns the unique ID of the Sonyflake instance.
If MachineID returns an error, Sonyflake is not created.
If MachineID is nil, default MachineID is used.
Default MachineID returns the lower 16 bits of the private IP address.
- MachineID returns the unique ID of a Sonyflake instance.
If MachineID returns an error, the instance will not be created.
If MachineID is nil, the default MachineID is used, which returns the lower 16 bits of the private IP address.
- CheckMachineID validates the uniqueness of the machine ID.
If CheckMachineID returns false, Sonyflake is not created.
- CheckMachineID validates the uniqueness of a machine ID.
If CheckMachineID returns false, the instance will not be created.
If CheckMachineID is nil, no validation is done.
The bit length of time is calculated by 63 - BitsSequence - BitsMachineID.
If it is less than 32, an error is returned.
In order to get a new unique ID, you just have to call the method NextID.
```go
func (sf *Sonyflake) NextID() (uint64, error)
func (sf *Sonyflake) NextID() (int64, error)
```
NextID can continue to generate IDs for about 174 years from StartTime.
NextID can continue to generate IDs for about 174 years from StartTime by default.
But after the Sonyflake time is over the limit, NextID returns an error.
> **Note:**
> Sonyflake currently does not use the most significant bit of IDs,
> so you can convert Sonyflake IDs from `uint64` to `int64` safely.
## AWS VPC and Docker
AWS VPC and Docker
------------------
The [awsutil](https://github.com/sony/sonyflake/blob/master/awsutil) package provides
The [awsutil](https://github.com/sony/sonyflake/blob/master/v2/awsutil) package provides
the function AmazonEC2MachineID that returns the lower 16-bit private IP address of the Amazon EC2 instance.
It also works correctly on Docker
by retrieving [instance metadata](http://docs.aws.amazon.com/en_us/AWSEC2/latest/UserGuide/ec2-instance-metadata.html).
@@ -89,10 +101,9 @@ So if each EC2 instance has a unique private IP address in AWS VPC,
the lower 16 bits of the address is also unique.
In this common case, you can use AmazonEC2MachineID as Settings.MachineID.
See [example](https://github.com/sony/sonyflake/blob/master/example) that runs Sonyflake on AWS Elastic Beanstalk.
See [example](https://github.com/sony/sonyflake/blob/master/v2/example) that runs Sonyflake on AWS Elastic Beanstalk.
License
-------
## License
The MIT License (MIT)

View File

@@ -1,6 +1,9 @@
FROM ubuntu:14.04
ADD ./sonyflake_server /
ENTRYPOINT ["/sonyflake_server"]
COPY ./sonyflake_server /sonyflake_server
RUN useradd -m sonyflake
USER sonyflake
ENTRYPOINT ["/sonyflake_server"]
EXPOSE 8080

View File

@@ -1,21 +1,19 @@
Example
=======
# Example
This example runs Sonyflake on AWS Elastic Beanstalk.
Setup
-----
## Setup
1. Build the cross compiler for linux/amd64 if using other platforms.
```
cd $GOROOT/src && GOOS=linux GOARCH=amd64 ./make.bash
```
```bash
cd $GOROOT/src && GOOS=linux GOARCH=amd64 ./make.bash
```
2. Build sonyflake_server in the example directory.
```
./linux64_build.sh
```
```bash
./linux64_build.sh
```
3. Upload the example directory to AWS Elastic Beanstalk.

View File

@@ -33,10 +33,17 @@ func handler(w http.ResponseWriter, r *http.Request) {
}
w.Header()["Content-Type"] = []string{"application/json; charset=utf-8"}
w.Write(body)
_, err = w.Write(body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
err := http.ListenAndServe(":8080", nil)
if err != nil {
panic(err)
}
}

View File

@@ -1,6 +1,5 @@
// Package mock offers implementations of interfaces defined in types.go
// This allows complete control over input / output for any given method that consumes
// a given type
// Package mock offers mock implementations of interfaces defined in types.go.
// This allows complete control over input / output for any given method that consumes a given type.
package mock
import (
@@ -10,7 +9,7 @@ import (
"github.com/sony/sonyflake/types"
)
// NewSuccessfulInterfaceAddrs returns a single private IP address
// NewSuccessfulInterfaceAddrs returns a single private IP address.
func NewSuccessfulInterfaceAddrs() types.InterfaceAddrs {
ifat := make([]net.Addr, 0, 1)
ifat = append(ifat, &net.IPNet{IP: []byte{192, 168, 0, 1}, Mask: []byte{255, 0, 0, 0}})
@@ -20,14 +19,14 @@ func NewSuccessfulInterfaceAddrs() types.InterfaceAddrs {
}
}
// NewFailingInterfaceAddrs returns an error
// NewFailingInterfaceAddrs returns an error.
func NewFailingInterfaceAddrs() types.InterfaceAddrs {
return func() ([]net.Addr, error) {
return nil, fmt.Errorf("test error")
}
}
// NewFailingInterfaceAddrs returns an empty slice of addresses
// NewNilInterfaceAddrs returns an empty slice of addresses.
func NewNilInterfaceAddrs() types.InterfaceAddrs {
return func() ([]net.Addr, error) {
return []net.Addr{}, nil

View File

@@ -57,6 +57,7 @@ var (
ErrNoPrivateAddress = errors.New("no private ip address")
ErrOverTimeLimit = errors.New("over the time limit")
ErrInvalidMachineID = errors.New("invalid machine id")
ErrInvalidSequence = errors.New("invalid sequence number")
)
var defaultInterfaceAddrs = net.InterfaceAddrs
@@ -213,6 +214,25 @@ func MachineID(id uint64) uint64 {
return id & maskMachineID
}
// Compose creates a Sonyflake ID from its parts.
func Compose(sf *Sonyflake, t time.Time, sequence uint16, machineID uint16) (uint64, error) {
elapsedTime := toSonyflakeTime(t.UTC()) - sf.startTime
if elapsedTime < 0 {
return 0, ErrStartTimeAhead
}
if elapsedTime >= 1<<BitLenTime {
return 0, ErrOverTimeLimit
}
if sequence >= 1<<BitLenSequence {
return 0, ErrInvalidSequence
}
return uint64(elapsedTime)<<(BitLenSequence+BitLenMachineID) |
uint64(sequence)<<BitLenMachineID |
uint64(machineID), nil
}
// Decompose returns a set of Sonyflake ID parts.
func Decompose(id uint64) map[string]uint64 {
msb := id >> 63

View File

@@ -312,3 +312,36 @@ func TestSonyflakeTimeUnit(t *testing.T) {
t.Errorf("unexpected time unit")
}
}
func TestCompose(t *testing.T) {
var st Settings
st.StartTime = time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)
sf, err := New(st)
if err != nil {
t.Fatal(err)
}
now := time.Now()
sequence := uint16(123)
machineID := uint16(456)
id, err := Compose(sf, now, sequence, machineID)
if err != nil {
t.Fatal(err)
}
parts := Decompose(id)
actualTime := toSonyflakeTime(now) - toSonyflakeTime(st.StartTime)
if parts["time"] != uint64(actualTime) {
t.Errorf("unexpected time: %d", parts["time"])
}
if parts["sequence"] != uint64(sequence) {
t.Errorf("unexpected sequence: %d", parts["sequence"])
}
if parts["machine-id"] != uint64(machineID) {
t.Errorf("unexpected machine id: %d", parts["machine-id"])
}
}

View File

@@ -1,8 +1,8 @@
// Package Types defines type signatures used throughout SonyFlake. This allows for
// fine-tuned control over imports, and the ability to mock out imports as well
// Package types defines type signatures used throughout sonyflake.
// This provides the ability to mock out imports.
package types
import "net"
// InterfaceAddrs defines the interface used for retrieving network addresses
// InterfaceAddrs defines the interface used for retrieving network addresses.
type InterfaceAddrs func() ([]net.Addr, error)

64
v2/awsutil/awsutil.go Normal file
View File

@@ -0,0 +1,64 @@
// Package awsutil provides utility functions for using Sonyflake on AWS.
package awsutil
import (
"errors"
"io"
"net"
"net/http"
"os/exec"
"regexp"
"strconv"
"time"
)
func amazonEC2PrivateIPv4() (net.IP, error) {
res, err := http.Get("http://169.254.169.254/latest/meta-data/local-ipv4")
if err != nil {
return nil, err
}
defer res.Body.Close()
body, err := io.ReadAll(res.Body)
if err != nil {
return nil, err
}
ip := net.ParseIP(string(body))
if ip == nil {
return nil, errors.New("invalid ip address")
}
return ip.To4(), nil
}
// AmazonEC2MachineID retrieves the private IP address of the Amazon EC2 instance
// and returns its lower 16 bits.
// It works correctly on Docker as well.
func AmazonEC2MachineID() (int, error) {
ip, err := amazonEC2PrivateIPv4()
if err != nil {
return 0, err
}
return int(ip[2])<<8 + int(ip[3]), nil
}
// TimeDifference returns the time difference between the localhost and the given NTP server.
func TimeDifference(server string) (time.Duration, error) {
output, err := exec.Command("/usr/sbin/ntpdate", "-q", server).CombinedOutput()
if err != nil {
return time.Duration(0), err
}
re, _ := regexp.Compile("offset (.*) sec")
submatched := re.FindSubmatch(output)
if len(submatched) != 2 {
return time.Duration(0), errors.New("invalid ntpdate output")
}
f, err := strconv.ParseFloat(string(submatched[1]), 64)
if err != nil {
return time.Duration(0), err
}
return time.Duration(f*1000) * time.Millisecond, nil
}

9
v2/example/Dockerfile Normal file
View File

@@ -0,0 +1,9 @@
FROM ubuntu:24.04
COPY ./sonyflake_server /sonyflake_server
RUN useradd -m sonyflake
USER sonyflake
ENTRYPOINT ["/sonyflake_server"]
EXPOSE 8080

View File

@@ -0,0 +1,8 @@
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": "8080"
}
]
}

19
v2/example/README.md Normal file
View File

@@ -0,0 +1,19 @@
# Example
This example runs Sonyflake on AWS Elastic Beanstalk.
## Setup
1. Build the cross compiler for linux/amd64 if using other platforms.
```bash
cd $GOROOT/src && GOOS=linux GOARCH=amd64 ./make.bash
```
2. Build sonyflake_server in the example directory.
```bash
./linux64_build.sh
```
3. Upload the example directory to AWS Elastic Beanstalk.

2
v2/example/linux64_build.sh Executable file
View File

@@ -0,0 +1,2 @@
#!/bin/sh
GOOS=linux GOARCH=amd64 go build sonyflake_server.go

View File

@@ -0,0 +1,51 @@
package main
import (
"encoding/json"
"net/http"
"github.com/sony/sonyflake/v2"
"github.com/sony/sonyflake/v2/awsutil"
)
var sf *sonyflake.Sonyflake
func init() {
var st sonyflake.Settings
st.MachineID = awsutil.AmazonEC2MachineID
var err error
sf, err = sonyflake.New(st)
if err != nil {
panic(err)
}
}
func handler(w http.ResponseWriter, r *http.Request) {
id, err := sf.NextID()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
body, err := json.Marshal(sf.Decompose(id))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header()["Content-Type"] = []string{"application/json; charset=utf-8"}
_, err = w.Write(body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
func main() {
http.HandleFunc("/", handler)
err := http.ListenAndServe(":8080", nil)
if err != nil {
panic(err)
}
}

3
v2/go.mod Normal file
View File

@@ -0,0 +1,3 @@
module github.com/sony/sonyflake/v2
go 1.22

36
v2/mock/mock.go Normal file
View File

@@ -0,0 +1,36 @@
// Package mock offers mock implementations of interfaces defined in types.go.
// This allows complete control over input / output for any given method that consumes a given type.
package mock
import (
"errors"
"net"
"github.com/sony/sonyflake/v2/types"
)
// NewSuccessfulInterfaceAddrs returns a single private IP address.
func NewSuccessfulInterfaceAddrs() types.InterfaceAddrs {
ifat := make([]net.Addr, 0, 1)
ifat = append(ifat, &net.IPNet{IP: []byte{192, 168, 0, 1}, Mask: []byte{255, 0, 0, 0}})
return func() ([]net.Addr, error) {
return ifat, nil
}
}
var ErrFailedToGetAddresses = errors.New("failed to get addresses")
// NewFailingInterfaceAddrs returns an error.
func NewFailingInterfaceAddrs() types.InterfaceAddrs {
return func() ([]net.Addr, error) {
return nil, ErrFailedToGetAddresses
}
}
// NewNilInterfaceAddrs returns an empty slice of addresses.
func NewNilInterfaceAddrs() types.InterfaceAddrs {
return func() ([]net.Addr, error) {
return []net.Addr{}, nil
}
}

314
v2/sonyflake.go Normal file
View File

@@ -0,0 +1,314 @@
// Package sonyflake implements Sonyflake, a distributed unique ID generator inspired by Twitter's Snowflake.
//
// By default, a Sonyflake ID is composed of
//
// 39 bits for time in units of 10 msec
// 8 bits for a sequence number
// 16 bits for a machine id
package sonyflake
import (
"errors"
"net"
"sync"
"time"
"github.com/sony/sonyflake/v2/types"
)
// Settings configures Sonyflake:
//
// BitsSequence is the bit length of a sequence number.
// If BitsSequence is 0, the default bit length is used, which is 8.
// If BitsSequence is 31 or more, an error is returned.
//
// BitsMachineID is the bit length of a machine ID.
// If BitsMachineID is 0, the default bit length is used, which is 16.
// If BitsMachineID is 31 or more, an error is returned.
//
// TimeUnit is the time unit of Sonyflake.
// If TimeUnit is 0, the default time unit is used, which is 10 msec.
// TimeUnit must be 1 msec or longer.
//
// StartTime is the time since which the Sonyflake time is defined as the elapsed time.
// If StartTime is 0, the start time of the Sonyflake instance is set to "2025-01-01 00:00:00 +0000 UTC".
// StartTime must be before the current time.
//
// MachineID returns the unique ID of a Sonyflake instance.
// If MachineID returns an error, the instance will not be created.
// If MachineID is nil, the default MachineID is used, which returns the lower 16 bits of the private IP address.
//
// CheckMachineID validates the uniqueness of a machine ID.
// If CheckMachineID returns false, the instance will not be created.
// If CheckMachineID is nil, no validation is done.
//
// The bit length of time is calculated by 63 - BitsSequence - BitsMachineID.
// If it is less than 32, an error is returned.
type Settings struct {
BitsSequence int
BitsMachineID int
TimeUnit time.Duration
StartTime time.Time
MachineID func() (int, error)
CheckMachineID func(int) bool
}
// Sonyflake is a distributed unique ID generator.
type Sonyflake struct {
mutex *sync.Mutex
bitsTime int
bitsSequence int
bitsMachine int
timeUnit int64
startTime int64
elapsedTime int64
sequence int
machine int
now func() time.Time
}
var (
ErrInvalidBitsTime = errors.New("bit length for time must be 32 or more")
ErrInvalidBitsSequence = errors.New("invalid bit length for sequence number")
ErrInvalidBitsMachineID = errors.New("invalid bit length for machine id")
ErrInvalidTimeUnit = errors.New("invalid time unit")
ErrInvalidSequence = errors.New("invalid sequence number")
ErrInvalidMachineID = errors.New("invalid machine id")
ErrStartTimeAhead = errors.New("start time is ahead")
ErrOverTimeLimit = errors.New("over the time limit")
ErrNoPrivateAddress = errors.New("no private ip address")
)
const (
defaultTimeUnit = 1e7 // nsec, i.e. 10 msec
defaultBitsTime = 39
defaultBitsSequence = 8
defaultBitsMachine = 16
)
var defaultInterfaceAddrs = net.InterfaceAddrs
// New returns a new Sonyflake configured with the given Settings.
// New returns an error in the following cases:
// - Settings.BitsSequence is less than 0 or greater than 30.
// - Settings.BitsMachineID is less than 0 or greater than 30.
// - Settings.BitsSequence + Settings.BitsMachineID is 32 or more.
// - Settings.TimeUnit is less than 1 msec.
// - Settings.StartTime is ahead of the current time.
// - Settings.MachineID returns an error.
// - Settings.CheckMachineID returns false.
func New(st Settings) (*Sonyflake, error) {
if st.BitsSequence < 0 || st.BitsSequence > 30 {
return nil, ErrInvalidBitsSequence
}
if st.BitsMachineID < 0 || st.BitsMachineID > 30 {
return nil, ErrInvalidBitsMachineID
}
if st.TimeUnit < 0 || (st.TimeUnit > 0 && st.TimeUnit < time.Millisecond) {
return nil, ErrInvalidTimeUnit
}
if st.StartTime.After(time.Now()) {
return nil, ErrStartTimeAhead
}
sf := new(Sonyflake)
sf.mutex = new(sync.Mutex)
sf.now = time.Now
if st.BitsSequence == 0 {
sf.bitsSequence = defaultBitsSequence
} else {
sf.bitsSequence = st.BitsSequence
}
if st.BitsMachineID == 0 {
sf.bitsMachine = defaultBitsMachine
} else {
sf.bitsMachine = st.BitsMachineID
}
sf.bitsTime = 63 - sf.bitsSequence - sf.bitsMachine
if sf.bitsTime < 32 {
return nil, ErrInvalidBitsTime
}
if st.TimeUnit == 0 {
sf.timeUnit = defaultTimeUnit
} else {
sf.timeUnit = int64(st.TimeUnit)
}
if st.StartTime.IsZero() {
sf.startTime = sf.toInternalTime(time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC))
} else {
sf.startTime = sf.toInternalTime(st.StartTime)
}
sf.sequence = 1<<sf.bitsSequence - 1
var err error
if st.MachineID == nil {
sf.machine, err = lower16BitPrivateIP(defaultInterfaceAddrs)
} else {
sf.machine, err = st.MachineID()
}
if err != nil {
return nil, err
}
if sf.machine < 0 || sf.machine >= 1<<sf.bitsMachine {
return nil, ErrInvalidMachineID
}
if st.CheckMachineID != nil && !st.CheckMachineID(sf.machine) {
return nil, ErrInvalidMachineID
}
return sf, nil
}
// NextID generates a next unique ID as int64.
// After the Sonyflake time overflows, NextID returns an error.
func (sf *Sonyflake) NextID() (int64, error) {
maskSequence := 1<<sf.bitsSequence - 1
sf.mutex.Lock()
defer sf.mutex.Unlock()
current := sf.currentElapsedTime()
if sf.elapsedTime < current {
sf.elapsedTime = current
sf.sequence = 0
} else {
sf.sequence = (sf.sequence + 1) & maskSequence
if sf.sequence == 0 {
sf.elapsedTime++
overtime := sf.elapsedTime - current
sf.sleep(overtime)
}
}
return sf.toID()
}
func (sf *Sonyflake) toInternalTime(t time.Time) int64 {
return t.UTC().UnixNano() / sf.timeUnit
}
func (sf *Sonyflake) currentElapsedTime() int64 {
return sf.toInternalTime(sf.now()) - sf.startTime
}
func (sf *Sonyflake) sleep(overtime int64) {
sleepTime := time.Duration(overtime*sf.timeUnit) -
time.Duration(sf.now().UTC().UnixNano()%sf.timeUnit)
time.Sleep(sleepTime)
}
func (sf *Sonyflake) toID() (int64, error) {
if sf.elapsedTime >= 1<<sf.bitsTime {
return 0, ErrOverTimeLimit
}
return sf.elapsedTime<<(sf.bitsSequence+sf.bitsMachine) |
int64(sf.sequence)<<sf.bitsMachine |
int64(sf.machine), nil
}
func privateIPv4(interfaceAddrs types.InterfaceAddrs) (net.IP, error) {
as, err := interfaceAddrs()
if err != nil {
return nil, err
}
for _, a := range as {
ipnet, ok := a.(*net.IPNet)
if !ok || ipnet.IP.IsLoopback() {
continue
}
ip := ipnet.IP.To4()
if isPrivateIPv4(ip) {
return ip, nil
}
}
return nil, ErrNoPrivateAddress
}
func isPrivateIPv4(ip net.IP) bool {
// Allow private IP addresses (RFC1918) and link-local addresses (RFC3927)
return ip != nil &&
(ip[0] == 10 || ip[0] == 172 && (ip[1] >= 16 && ip[1] < 32) || ip[0] == 192 && ip[1] == 168 || ip[0] == 169 && ip[1] == 254)
}
func lower16BitPrivateIP(interfaceAddrs types.InterfaceAddrs) (int, error) {
ip, err := privateIPv4(interfaceAddrs)
if err != nil {
return 0, err
}
return int(ip[2])<<8 + int(ip[3]), nil
}
// ToTime returns the time when the given ID was generated.
func (sf *Sonyflake) ToTime(id int64) time.Time {
return time.Unix(0, (sf.startTime+sf.timePart(id))*sf.timeUnit)
}
// Compose creates a Sonyflake ID from its components.
// The time parameter should be the time when the ID was generated.
// The sequence parameter should be between 0 and 2^BitsSequence-1 (inclusive).
// The machineID parameter should be between 0 and 2^BitsMachineID-1 (inclusive).
func (sf *Sonyflake) Compose(t time.Time, sequence, machineID int) (int64, error) {
elapsedTime := sf.toInternalTime(t.UTC()) - sf.startTime
if elapsedTime < 0 {
return 0, ErrStartTimeAhead
}
if elapsedTime >= 1<<sf.bitsTime {
return 0, ErrOverTimeLimit
}
if sequence < 0 || sequence >= 1<<sf.bitsSequence {
return 0, ErrInvalidSequence
}
if machineID < 0 || machineID >= 1<<sf.bitsMachine {
return 0, ErrInvalidMachineID
}
return elapsedTime<<(sf.bitsSequence+sf.bitsMachine) |
int64(sequence)<<sf.bitsMachine |
int64(machineID), nil
}
// Decompose returns a set of Sonyflake ID parts.
func (sf *Sonyflake) Decompose(id int64) map[string]int64 {
time := sf.timePart(id)
sequence := sf.sequencePart(id)
machine := sf.machinePart(id)
return map[string]int64{
"id": id,
"time": time,
"sequence": sequence,
"machine": machine,
}
}
func (sf *Sonyflake) timePart(id int64) int64 {
return id >> (sf.bitsSequence + sf.bitsMachine)
}
func (sf *Sonyflake) sequencePart(id int64) int64 {
maskSequence := int64((1<<sf.bitsSequence - 1) << sf.bitsMachine)
return (id & maskSequence) >> sf.bitsMachine
}
func (sf *Sonyflake) machinePart(id int64) int64 {
maskMachine := int64(1<<sf.bitsMachine - 1)
return id & maskMachine
}

496
v2/sonyflake_test.go Normal file
View File

@@ -0,0 +1,496 @@
package sonyflake
import (
"errors"
"net"
"runtime"
"testing"
"time"
"github.com/sony/sonyflake/v2/mock"
"github.com/sony/sonyflake/v2/types"
)
func TestNew(t *testing.T) {
errGetMachineID := errors.New("failed to get machine id")
testCases := []struct {
name string
settings Settings
err error
}{
{
name: "invalid bit length for time",
settings: Settings{
BitsSequence: 16,
BitsMachineID: 16,
},
err: ErrInvalidBitsTime,
},
{
name: "invalid bit length for sequence number",
settings: Settings{
BitsSequence: -1,
},
err: ErrInvalidBitsSequence,
},
{
name: "invalid bit length for machine id",
settings: Settings{
BitsMachineID: 31,
},
err: ErrInvalidBitsMachineID,
},
{
name: "invalid time unit",
settings: Settings{
TimeUnit: time.Microsecond,
},
err: ErrInvalidTimeUnit,
},
{
name: "start time ahead",
settings: Settings{
StartTime: time.Now().Add(time.Minute),
},
err: ErrStartTimeAhead,
},
{
name: "cannot get machine id",
settings: Settings{
MachineID: func() (int, error) {
return 0, errGetMachineID
},
},
err: errGetMachineID,
},
{
name: "too large machine id",
settings: Settings{
MachineID: func() (int, error) {
return 1 << defaultBitsMachine, nil
},
},
err: ErrInvalidMachineID,
},
{
name: "negative machine id",
settings: Settings{
MachineID: func() (int, error) {
return -1, nil
},
},
err: ErrInvalidMachineID,
},
{
name: "invalid machine id",
settings: Settings{
CheckMachineID: func(int) bool {
return false
},
},
err: ErrInvalidMachineID,
},
{
name: "success",
settings: Settings{},
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
sf, err := New(tc.settings)
if !errors.Is(err, tc.err) {
t.Fatalf("unexpected error: %v", err)
}
if err == nil && sf == nil {
t.Fatal("sonyflake instance must be created")
}
})
}
}
func newSonyflake(t *testing.T, st Settings) *Sonyflake {
sf, err := New(st)
if err != nil {
t.Fatalf("failed to create sonyflake: %v", err)
}
return sf
}
func nextID(t *testing.T, sf *Sonyflake) int64 {
id, err := sf.NextID()
if err != nil {
t.Fatalf("failed to generate id: %v", err)
}
return id
}
func defaultMachineID(t *testing.T) int {
ip, err := lower16BitPrivateIP(defaultInterfaceAddrs)
if err != nil {
t.Fatalf("failed to get private ip address: %v", err)
}
return ip
}
func TestNextID(t *testing.T) {
start := time.Now()
sf := newSonyflake(t, Settings{StartTime: start})
sleepTime := int64(50)
sf.now = func() time.Time { return start.Add(time.Duration(sleepTime * sf.timeUnit)) }
id := nextID(t, sf)
actualTime := sf.timePart(id)
if actualTime != sleepTime {
t.Errorf("unexpected time: %d", actualTime)
}
actualSequence := sf.sequencePart(id)
if actualSequence != 0 {
t.Errorf("unexpected sequence: %d", actualSequence)
}
actualMachine := sf.machinePart(id)
if actualMachine != int64(defaultMachineID(t)) {
t.Errorf("unexpected machine: %d", actualMachine)
}
t.Log("sonyflake id:", id)
t.Log("decompose:", sf.Decompose(id))
}
func TestNextID_InSequence(t *testing.T) {
start := time.Now()
sf := newSonyflake(t, Settings{
TimeUnit: time.Millisecond,
StartTime: start,
})
startTime := sf.toInternalTime(start)
machineID := int64(defaultMachineID(t))
var numID int
var lastID int64
var maxSeq int64
currentTime := startTime
for currentTime-startTime < 100 {
id := nextID(t, sf)
currentTime = sf.toInternalTime(time.Now())
numID++
if id == lastID {
t.Fatal("duplicated id")
}
if id < lastID {
t.Fatal("must increase with time")
}
lastID = id
parts := sf.Decompose(id)
actualTime := parts["time"]
overtime := startTime + actualTime - currentTime
if overtime > 0 {
t.Errorf("unexpected overtime: %d", overtime)
}
actualSequence := parts["sequence"]
if actualSequence > maxSeq {
maxSeq = actualSequence
}
actualMachine := parts["machine"]
if actualMachine != machineID {
t.Errorf("unexpected machine: %d", actualMachine)
}
}
if maxSeq > 1<<sf.bitsSequence-1 {
t.Errorf("unexpected max sequence: %d", maxSeq)
}
t.Log("max sequence:", maxSeq)
t.Log("number of id:", numID)
}
func TestNextID_InParallel(t *testing.T) {
sf1 := newSonyflake(t, Settings{MachineID: func() (int, error) { return 1, nil }})
sf2 := newSonyflake(t, Settings{MachineID: func() (int, error) { return 2, nil }})
numCPU := runtime.NumCPU()
runtime.GOMAXPROCS(numCPU)
t.Log("number of cpu:", numCPU)
consumer := make(chan int64)
const numID = 1000
generate := func(sf *Sonyflake) {
for i := 0; i < numID; i++ {
id := nextID(t, sf)
consumer <- id
}
}
var numGenerator int
for i := 0; i < numCPU/2; i++ {
go generate(sf1)
go generate(sf2)
numGenerator += 2
}
set := make(map[int64]struct{})
for i := 0; i < numID*numGenerator; i++ {
id := <-consumer
if _, ok := set[id]; ok {
t.Fatal("duplicated id")
}
set[id] = struct{}{}
}
t.Log("number of id:", len(set))
}
func pseudoSleep(sf *Sonyflake, period time.Duration) {
sf.startTime -= int64(period) / sf.timeUnit
}
const year = time.Duration(365*24) * time.Hour
func TestNextID_ReturnsError(t *testing.T) {
sf := newSonyflake(t, Settings{StartTime: time.Now()})
pseudoSleep(sf, time.Duration(174)*year)
nextID(t, sf)
pseudoSleep(sf, time.Duration(1)*year)
_, err := sf.NextID()
if err == nil {
t.Errorf("time is not over")
}
}
func TestPrivateIPv4(t *testing.T) {
testCases := []struct {
description string
expected net.IP
interfaceAddrs types.InterfaceAddrs
error error
}{
{
description: "returns an error",
expected: nil,
interfaceAddrs: mock.NewFailingInterfaceAddrs(),
error: mock.ErrFailedToGetAddresses,
},
{
description: "empty address list",
expected: nil,
interfaceAddrs: mock.NewNilInterfaceAddrs(),
error: ErrNoPrivateAddress,
},
{
description: "success",
expected: net.IP{192, 168, 0, 1},
interfaceAddrs: mock.NewSuccessfulInterfaceAddrs(),
error: nil,
},
}
for _, tc := range testCases {
t.Run(tc.description, func(t *testing.T) {
actual, err := privateIPv4(tc.interfaceAddrs)
if !errors.Is(err, tc.error) {
t.Fatalf("unexpected error: %v", err)
}
if !net.IP.Equal(actual, tc.expected) {
t.Errorf("unexpected ip: %s", actual)
}
})
}
}
func TestLower16BitPrivateIP(t *testing.T) {
testCases := []struct {
description string
expected int
interfaceAddrs types.InterfaceAddrs
error error
}{
{
description: "returns an error",
expected: 0,
interfaceAddrs: mock.NewFailingInterfaceAddrs(),
error: mock.ErrFailedToGetAddresses,
},
{
description: "empty address list",
expected: 0,
interfaceAddrs: mock.NewNilInterfaceAddrs(),
error: ErrNoPrivateAddress,
},
{
description: "success",
expected: 1,
interfaceAddrs: mock.NewSuccessfulInterfaceAddrs(),
error: nil,
},
}
for _, tc := range testCases {
t.Run(tc.description, func(t *testing.T) {
actual, err := lower16BitPrivateIP(tc.interfaceAddrs)
if !errors.Is(err, tc.error) {
t.Fatalf("unexpected error: %v", err)
}
if actual != tc.expected {
t.Errorf("unexpected ip: %d", actual)
}
})
}
}
func TestToTime(t *testing.T) {
start := time.Now()
sf := newSonyflake(t, Settings{
TimeUnit: time.Millisecond,
StartTime: start,
})
sf.now = func() time.Time { return start }
id := nextID(t, sf)
tm := sf.ToTime(id)
diff := tm.Sub(start)
if diff < 0 || diff >= time.Duration(sf.timeUnit) {
t.Errorf("unexpected time: %v", diff)
}
}
func TestComposeAndDecompose(t *testing.T) {
now := time.Now()
sf := newSonyflake(t, Settings{
TimeUnit: time.Millisecond,
StartTime: now,
})
testCases := []struct {
name string
time time.Time
sequence int
machineID int
}{
{
name: "zero values",
time: now,
sequence: 0,
machineID: 0,
},
{
name: "max sequence",
time: now,
sequence: 1<<sf.bitsSequence - 1,
machineID: 0,
},
{
name: "max machine id",
time: now,
sequence: 0,
machineID: 1<<sf.bitsMachine - 1,
},
{
name: "future time",
time: now.Add(time.Hour),
sequence: 0,
machineID: 0,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
id, err := sf.Compose(tc.time, tc.sequence, tc.machineID)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
parts := sf.Decompose(id)
// Verify time part
expectedTime := sf.toInternalTime(tc.time.UTC()) - sf.startTime
if parts["time"] != expectedTime {
t.Errorf("time mismatch: got %d, want %d", parts["time"], expectedTime)
}
// Verify sequence part
if parts["sequence"] != int64(tc.sequence) {
t.Errorf("sequence mismatch: got %d, want %d", parts["sequence"], tc.sequence)
}
// Verify machine id part
if parts["machine"] != int64(tc.machineID) {
t.Errorf("machine id mismatch: got %d, want %d", parts["machine"], tc.machineID)
}
// Verify id part
if parts["id"] != id {
t.Errorf("id mismatch: got %d, want %d", parts["id"], id)
}
})
}
}
func TestCompose_ReturnsError(t *testing.T) {
start := time.Now()
sf := newSonyflake(t, Settings{StartTime: start})
testCases := []struct {
name string
time time.Time
sequence int
machineID int
err error
}{
{
name: "start time ahead",
time: start.Add(-time.Second),
sequence: 0,
machineID: 0,
err: ErrStartTimeAhead,
},
{
name: "over time limit",
time: start.Add(time.Duration(175) * year),
sequence: 0,
machineID: 0,
err: ErrOverTimeLimit,
},
{
name: "invalid sequence",
time: start,
sequence: 1 << sf.bitsSequence,
machineID: 0,
err: ErrInvalidSequence,
},
{
name: "invalid machine id",
time: start,
sequence: 0,
machineID: 1 << sf.bitsMachine,
err: ErrInvalidMachineID,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
_, err := sf.Compose(tc.time, tc.sequence, tc.machineID)
if !errors.Is(err, tc.err) {
t.Errorf("unexpected error: %v", err)
}
})
}
}

8
v2/types/types.go Normal file
View File

@@ -0,0 +1,8 @@
// Package types defines type signatures used throughout sonyflake.
// This provides the ability to mock out imports.
package types
import "net"
// InterfaceAddrs defines the interface used for retrieving network addresses.
type InterfaceAddrs func() ([]net.Addr, error)