mirrors/minio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure codebase move crypto, checksum to top-level, move `split` into donut, move crypto/keys into api/auth

Browse Source
This commit is contained in:
Harshavardhana
2015-07-03 15:17:44 -07:00
parent 8a4e7bcdcf
commit 30fc14e703
52 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

202
pkg/hash/crc32c/LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

51
pkg/hash/crc32c/crc32c_darwin.go Normal file
View File

@@ -0,0 +1,51 @@
/*
* Minimalist Object Storage, (C) 2014 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package crc32c
import (
"hash/crc32"
"io"
)
var castanagoliTable = crc32.MakeTable(crc32.Castagnoli)
/// Convenience functions
// Sum32 - single caller crc helper
func Sum32(buffer []byte) uint32 {
crc := crc32.New(castanagoliTable)
crc.Reset()
crc.Write(buffer)
return crc.Sum32()
}
// Sum - io.Reader based crc helper
func Sum(reader io.Reader) (uint32, error) {
h := crc32.New(castanagoliTable)
var err error
for err == nil {
length := 0
byteBuffer := make([]byte, 1024*1024)
length, err = reader.Read(byteBuffer)
byteBuffer = byteBuffer[0:length]
h.Write(byteBuffer)
}
if err != io.EOF {
return 0, err
}
return h.Sum32(), nil
}

28
pkg/hash/crc32c/crc32c_intel_linux.go Normal file
View File

@@ -0,0 +1,28 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Golang project:
// https://github.com/golang/go/blob/master/LICENSE
// Using this part of Minio codebase under the license
// Apache License Version 2.0 with modifications
// Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32,
// checksum. See http://en.wikipedia.org/wiki/Cyclic_redundancy_check for
// information.
package crc32c
// #include <stdint.h>
// uint32_t crc32c_pcl(uint8_t *buf, int32_t len, uint32_t prev_crc);
import "C"
import (
"unsafe"
)
func updateCastanagoliPCL(crc uint32, p []byte) uint32 {
if len(p) == 0 {
return 0
}
return uint32(C.crc32c_pcl((*C.uint8_t)(unsafe.Pointer(&p[0])), C.int32_t(len(p)), C.uint32_t(crc)))
}

722
pkg/hash/crc32c/crc32c_linux.S Normal file
View File

@@ -0,0 +1,722 @@
/*
* Implement fast CRC32C with PCLMULQDQ instructions. (x86_64)
*
* The white papers on CRC32C calculations with PCLMULQDQ instruction can be
* downloaded from:
* http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf
* http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-paper.pdf
*
* Copyright (C) 2012 Intel Corporation.
*
* Authors:
* Wajdi Feghali <wajdi.k.feghali@intel.com>
* James Guilford <james.guilford@intel.com>
* David Cote <david.m.cote@intel.com>
* Tim Chen <tim.c.chen@linux.intel.com>
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
/* ISCSI CRC 32 Implementation with crc32 and pclmulqdq Instruction */
#ifndef ENTRY
#define ENTRY(name) \
.globl name ; \
.align 4,0x90 ; \
name:
#endif
#ifndef END
#define END(name) \
.size name, .-name
#endif
#ifndef ENDPROC
#define ENDPROC(name) \
.type name, @function ; \
END(name)
#endif
#define NUM_INVALID 100
#define TYPE_R32 0
#define TYPE_R64 1
#define TYPE_XMM 2
#define TYPE_INVALID 100
.macro R32_NUM opd r32
\opd = NUM_INVALID
.ifc \r32,%eax
\opd = 0
.endif
.ifc \r32,%ecx
\opd = 1
.endif
.ifc \r32,%edx
\opd = 2
.endif
.ifc \r32,%ebx
\opd = 3
.endif
.ifc \r32,%esp
\opd = 4
.endif
.ifc \r32,%ebp
\opd = 5
.endif
.ifc \r32,%esi
\opd = 6
.endif
.ifc \r32,%edi
\opd = 7
.endif
#ifdef X86_64
.ifc \r32,%r8d
\opd = 8
.endif
.ifc \r32,%r9d
\opd = 9
.endif
.ifc \r32,%r10d
\opd = 10
.endif
.ifc \r32,%r11d
\opd = 11
.endif
.ifc \r32,%r12d
\opd = 12
.endif
.ifc \r32,%r13d
\opd = 13
.endif
.ifc \r32,%r14d
\opd = 14
.endif
.ifc \r32,%r15d
\opd = 15
.endif
#endif
.endm
.macro R64_NUM opd r64
\opd = NUM_INVALID
#ifdef X86_64
.ifc \r64,%rax
\opd = 0
.endif
.ifc \r64,%rcx
\opd = 1
.endif
.ifc \r64,%rdx
\opd = 2
.endif
.ifc \r64,%rbx
\opd = 3
.endif
.ifc \r64,%rsp
\opd = 4
.endif
.ifc \r64,%rbp
\opd = 5
.endif
.ifc \r64,%rsi
\opd = 6
.endif
.ifc \r64,%rdi
\opd = 7
.endif
.ifc \r64,%r8
\opd = 8
.endif
.ifc \r64,%r9
\opd = 9
.endif
.ifc \r64,%r10
\opd = 10
.endif
.ifc \r64,%r11
\opd = 11
.endif
.ifc \r64,%r12
\opd = 12
.endif
.ifc \r64,%r13
\opd = 13
.endif
.ifc \r64,%r14
\opd = 14
.endif
.ifc \r64,%r15
\opd = 15
.endif
#endif
.endm
.macro XMM_NUM opd xmm
\opd = NUM_INVALID
.ifc \xmm,%xmm0
\opd = 0
.endif
.ifc \xmm,%xmm1
\opd = 1
.endif
.ifc \xmm,%xmm2
\opd = 2
.endif
.ifc \xmm,%xmm3
\opd = 3
.endif
.ifc \xmm,%xmm4
\opd = 4
.endif
.ifc \xmm,%xmm5
\opd = 5
.endif
.ifc \xmm,%xmm6
\opd = 6
.endif
.ifc \xmm,%xmm7
\opd = 7
.endif
.ifc \xmm,%xmm8
\opd = 8
.endif
.ifc \xmm,%xmm9
\opd = 9
.endif
.ifc \xmm,%xmm10
\opd = 10
.endif
.ifc \xmm,%xmm11
\opd = 11
.endif
.ifc \xmm,%xmm12
\opd = 12
.endif
.ifc \xmm,%xmm13
\opd = 13
.endif
.ifc \xmm,%xmm14
\opd = 14
.endif
.ifc \xmm,%xmm15
\opd = 15
.endif
.endm
.macro TYPE type reg
R32_NUM reg_type_r32 \reg
R64_NUM reg_type_r64 \reg
XMM_NUM reg_type_xmm \reg
.if reg_type_r64 <> NUM_INVALID
\type = TYPE_R64
.elseif reg_type_r32 <> NUM_INVALID
\type = TYPE_R32
.elseif reg_type_xmm <> NUM_INVALID
\type = TYPE_XMM
.else
\type = TYPE_INVALID
.endif
.endm
.macro PFX_OPD_SIZE
.byte 0x66
.endm
.macro PFX_REX opd1 opd2 W=0
.if ((\opd1 | \opd2) & 8) || \W
.byte 0x40 | ((\opd1 & 8) >> 3) | ((\opd2 & 8) >> 1) | (\W << 3)
.endif
.endm
.macro MODRM mod opd1 opd2
.byte \mod | (\opd1 & 7) | ((\opd2 & 7) << 3)
.endm
.macro PSHUFB_XMM xmm1 xmm2
XMM_NUM pshufb_opd1 \xmm1
XMM_NUM pshufb_opd2 \xmm2
PFX_OPD_SIZE
PFX_REX pshufb_opd1 pshufb_opd2
.byte 0x0f, 0x38, 0x00
MODRM 0xc0 pshufb_opd1 pshufb_opd2
.endm
.macro PCLMULQDQ imm8 xmm1 xmm2
XMM_NUM clmul_opd1 \xmm1
XMM_NUM clmul_opd2 \xmm2
PFX_OPD_SIZE
PFX_REX clmul_opd1 clmul_opd2
.byte 0x0f, 0x3a, 0x44
MODRM 0xc0 clmul_opd1 clmul_opd2
.byte \imm8
.endm
.macro PEXTRD imm8 xmm gpr
R32_NUM extrd_opd1 \gpr
XMM_NUM extrd_opd2 \xmm
PFX_OPD_SIZE
PFX_REX extrd_opd1 extrd_opd2
.byte 0x0f, 0x3a, 0x16
MODRM 0xc0 extrd_opd1 extrd_opd2
.byte \imm8
.endm
.macro MOVQ_R64_XMM opd1 opd2
TYPE movq_r64_xmm_opd1_type \opd1
.if movq_r64_xmm_opd1_type == TYPE_XMM
XMM_NUM movq_r64_xmm_opd1 \opd1
R64_NUM movq_r64_xmm_opd2 \opd2
.else
R64_NUM movq_r64_xmm_opd1 \opd1
XMM_NUM movq_r64_xmm_opd2 \opd2
.endif
PFX_OPD_SIZE
PFX_REX movq_r64_xmm_opd1 movq_r64_xmm_opd2 1
.if movq_r64_xmm_opd1_type == TYPE_XMM
.byte 0x0f, 0x7e
.else
.byte 0x0f, 0x6e
.endif
MODRM 0xc0 movq_r64_xmm_opd1 movq_r64_xmm_opd2
.endm
.macro LABEL prefix n
\prefix\n\():
.endm
.macro JMPTBL_ENTRY i
.word crc_\i - crc_array
.endm
.macro JNC_LESS_THAN j
jnc less_than_\j
.endm
# Define threshold where buffers are considered "small" and routed to more
# efficient "by-1" code. This "by-1" code only handles up to 255 bytes, so
# SMALL_SIZE can be no larger than 255.
#define SMALL_SIZE 200
.if (SMALL_SIZE > 255)
.error "SMALL_ SIZE must be < 256"
.endif
# unsigned int crc32c_pcl(u8 *buffer, int len, unsigned int crc_init);
.text
ENTRY(crc32c_pcl)
#define bufp %rdi
#define bufp_dw %edi
#define bufp_w %di
#define bufp_b %dil
#define bufptmp %rcx
#define block_0 %rcx
#define block_1 %rdx
#define block_2 %r11
#define len %rsi
#define len_dw %esi
#define len_w %si
#define len_b %sil
#define crc_init_arg %rdx
#define tmp %rbx
#define crc_init %r8
#define crc_init_dw %r8d
#define crc1 %r9
#define crc2 %r10
pushq %rbx
pushq %rdi
pushq %rsi
## Move crc_init for Linux to a different
mov crc_init_arg, crc_init
################################################################
## 1) ALIGN:
################################################################
mov bufp, bufptmp # rdi = *buf
neg bufp
and $7, bufp # calculate the unalignment amount of
# the address
je proc_block # Skip if aligned
## If len is less than 8 and we're unaligned, we need to jump
## to special code to avoid reading beyond the end of the buffer
cmp $8, len
jae do_align
# less_than_8 expects length in upper 3 bits of len_dw
# less_than_8_post_shl1 expects length = carryflag * 8 + len_dw[31:30]
shl $32-3+1, len_dw
jmp less_than_8_post_shl1
do_align:
#### Calculate CRC of unaligned bytes of the buffer (if any)
movq (bufptmp), tmp # load a quadward from the buffer
add bufp, bufptmp # align buffer pointer for quadword
# processing
sub bufp, len # update buffer length
align_loop:
crc32b %bl, crc_init_dw # compute crc32 of 1-byte
shr $8, tmp # get next byte
dec bufp
jne align_loop
proc_block:
################################################################
## 2) PROCESS BLOCKS:
################################################################
## compute num of bytes to be processed
movq len, tmp # save num bytes in tmp
cmpq $128*24, len
jae full_block
continue_block:
cmpq $SMALL_SIZE, len
jb small
## len < 128*24
movq $2731, %rax # 2731 = ceil(2^16 / 24)
mul len_dw
shrq $16, %rax
## eax contains floor(bytes / 24) = num 24-byte chunks to do
## process rax 24-byte chunks (128 >= rax >= 0)
## compute end address of each block
## block 0 (base addr + RAX * 8)
## block 1 (base addr + RAX * 16)
## block 2 (base addr + RAX * 24)
lea (bufptmp, %rax, 8), block_0
lea (block_0, %rax, 8), block_1
lea (block_1, %rax, 8), block_2
xor crc1, crc1
xor crc2, crc2
## branch into array
lea jump_table(%rip), bufp
movzwq (bufp, %rax, 2), len
offset=crc_array-jump_table
lea offset(bufp, len, 1), bufp
jmp *bufp
################################################################
## 2a) PROCESS FULL BLOCKS:
################################################################
full_block:
movq $128,%rax
lea 128*8*2(block_0), block_1
lea 128*8*3(block_0), block_2
add $128*8*1, block_0
xor crc1,crc1
xor crc2,crc2
# Fall thruogh into top of crc array (crc_128)
################################################################
## 3) CRC Array:
################################################################
crc_array:
i=128
.rept 128-1
.altmacro
LABEL crc_ %i
.noaltmacro
crc32q -i*8(block_0), crc_init
crc32q -i*8(block_1), crc1
crc32q -i*8(block_2), crc2
i=(i-1)
.endr
.altmacro
LABEL crc_ %i
.noaltmacro
crc32q -i*8(block_0), crc_init
crc32q -i*8(block_1), crc1
# SKIP crc32 -i*8(block_2), crc2 ; Don't do this one yet
mov block_2, block_0
################################################################
## 4) Combine three results:
################################################################
lea (K_table-8)(%rip), bufp # first entry is for idx 1
shlq $3, %rax # rax *= 8
pmovzxdq (bufp,%rax), %xmm0 # 2 consts: K1:K2
leal (%eax,%eax,2), %eax # rax *= 3 (total *24)
subq %rax, tmp # tmp -= rax*24
movq crc_init, %xmm1 # CRC for block 1
PCLMULQDQ 0x00, %xmm0, %xmm1 # Multiply by K2
movq crc1, %xmm2 # CRC for block 2
PCLMULQDQ 0x10, %xmm0, %xmm2 # Multiply by K1
pxor %xmm2,%xmm1
movq %xmm1, %rax
xor -i*8(block_2), %rax
mov crc2, crc_init
crc32 %rax, crc_init
################################################################
## 5) Check for end:
################################################################
LABEL crc_ 0
mov tmp, len
cmp $128*24, tmp
jae full_block
cmp $24, tmp
jae continue_block
less_than_24:
shl $32-4, len_dw # less_than_16 expects length
# in upper 4 bits of len_dw
jnc less_than_16
crc32q (bufptmp), crc_init
crc32q 8(bufptmp), crc_init
jz do_return
add $16, bufptmp
# len is less than 8 if we got here
# less_than_8 expects length in upper 3 bits of len_dw
# less_than_8_post_shl1 expects length = carryflag * 8 + len_dw[31:30]
shl $2, len_dw
jmp less_than_8_post_shl1
#######################################################################
## 6) LESS THAN 256-bytes REMAIN AT THIS POINT (8-bits of len are full)
#######################################################################
small:
shl $32-8, len_dw # Prepare len_dw for less_than_256
j=256
.rept 5 # j = {256, 128, 64, 32, 16}
.altmacro
LABEL less_than_ %j # less_than_j: Length should be in
# upper lg(j) bits of len_dw
j=(j/2)
shl $1, len_dw # Get next MSB
JNC_LESS_THAN %j
.noaltmacro
i=0
.rept (j/8)
crc32q i(bufptmp), crc_init # Compute crc32 of 8-byte data
i=i+8
.endr
jz do_return # Return if remaining length is zero
add $j, bufptmp # Advance buf
.endr
less_than_8: # Length should be stored in
# upper 3 bits of len_dw
shl $1, len_dw
less_than_8_post_shl1:
jnc less_than_4
crc32l (bufptmp), crc_init_dw # CRC of 4 bytes
jz do_return # return if remaining data is zero
add $4, bufptmp
less_than_4: # Length should be stored in
# upper 2 bits of len_dw
shl $1, len_dw
jnc less_than_2
crc32w (bufptmp), crc_init_dw # CRC of 2 bytes
jz do_return # return if remaining data is zero
add $2, bufptmp
less_than_2: # Length should be stored in the MSB
# of len_dw
shl $1, len_dw
jnc less_than_1
crc32b (bufptmp), crc_init_dw # CRC of 1 byte
less_than_1: # Length should be zero
do_return:
movq crc_init, %rax
popq %rsi
popq %rdi
popq %rbx
ret
################################################################
## jump table Table is 129 entries x 2 bytes each
################################################################
.align 4
jump_table:
i=0
.rept 129
.altmacro
JMPTBL_ENTRY %i
.noaltmacro
i=i+1
.endr
ENDPROC(crc32c_pcl)
################################################################
## PCLMULQDQ tables
## Table is 128 entries x 2 words (8 bytes) each
################################################################
.section .rotata, "a", %progbits
.align 8
K_table:
.long 0x493c7d27, 0x00000001
.long 0xba4fc28e, 0x493c7d27
.long 0xddc0152b, 0xf20c0dfe
.long 0x9e4addf8, 0xba4fc28e
.long 0x39d3b296, 0x3da6d0cb
.long 0x0715ce53, 0xddc0152b
.long 0x47db8317, 0x1c291d04
.long 0x0d3b6092, 0x9e4addf8
.long 0xc96cfdc0, 0x740eef02
.long 0x878a92a7, 0x39d3b296
.long 0xdaece73e, 0x083a6eec
.long 0xab7aff2a, 0x0715ce53
.long 0x2162d385, 0xc49f4f67
.long 0x83348832, 0x47db8317
.long 0x299847d5, 0x2ad91c30
.long 0xb9e02b86, 0x0d3b6092
.long 0x18b33a4e, 0x6992cea2
.long 0xb6dd949b, 0xc96cfdc0
.long 0x78d9ccb7, 0x7e908048
.long 0xbac2fd7b, 0x878a92a7
.long 0xa60ce07b, 0x1b3d8f29
.long 0xce7f39f4, 0xdaece73e
.long 0x61d82e56, 0xf1d0f55e
.long 0xd270f1a2, 0xab7aff2a
.long 0xc619809d, 0xa87ab8a8
.long 0x2b3cac5d, 0x2162d385
.long 0x65863b64, 0x8462d800
.long 0x1b03397f, 0x83348832
.long 0xebb883bd, 0x71d111a8
.long 0xb3e32c28, 0x299847d5
.long 0x064f7f26, 0xffd852c6
.long 0xdd7e3b0c, 0xb9e02b86
.long 0xf285651c, 0xdcb17aa4
.long 0x10746f3c, 0x18b33a4e
.long 0xc7a68855, 0xf37c5aee
.long 0x271d9844, 0xb6dd949b
.long 0x8e766a0c, 0x6051d5a2
.long 0x93a5f730, 0x78d9ccb7
.long 0x6cb08e5c, 0x18b0d4ff
.long 0x6b749fb2, 0xbac2fd7b
.long 0x1393e203, 0x21f3d99c
.long 0xcec3662e, 0xa60ce07b
.long 0x96c515bb, 0x8f158014
.long 0xe6fc4e6a, 0xce7f39f4
.long 0x8227bb8a, 0xa00457f7
.long 0xb0cd4768, 0x61d82e56
.long 0x39c7ff35, 0x8d6d2c43
.long 0xd7a4825c, 0xd270f1a2
.long 0x0ab3844b, 0x00ac29cf
.long 0x0167d312, 0xc619809d
.long 0xf6076544, 0xe9adf796
.long 0x26f6a60a, 0x2b3cac5d
.long 0xa741c1bf, 0x96638b34
.long 0x98d8d9cb, 0x65863b64
.long 0x49c3cc9c, 0xe0e9f351
.long 0x68bce87a, 0x1b03397f
.long 0x57a3d037, 0x9af01f2d
.long 0x6956fc3b, 0xebb883bd
.long 0x42d98888, 0x2cff42cf
.long 0x3771e98f, 0xb3e32c28
.long 0xb42ae3d9, 0x88f25a3a
.long 0x2178513a, 0x064f7f26
.long 0xe0ac139e, 0x4e36f0b0
.long 0x170076fa, 0xdd7e3b0c
.long 0x444dd413, 0xbd6f81f8
.long 0x6f345e45, 0xf285651c
.long 0x41d17b64, 0x91c9bd4b
.long 0xff0dba97, 0x10746f3c
.long 0xa2b73df1, 0x885f087b
.long 0xf872e54c, 0xc7a68855
.long 0x1e41e9fc, 0x4c144932
.long 0x86d8e4d2, 0x271d9844
.long 0x651bd98b, 0x52148f02
.long 0x5bb8f1bc, 0x8e766a0c
.long 0xa90fd27a, 0xa3c6f37a
.long 0xb3af077a, 0x93a5f730
.long 0x4984d782, 0xd7c0557f
.long 0xca6ef3ac, 0x6cb08e5c
.long 0x234e0b26, 0x63ded06a
.long 0xdd66cbbb, 0x6b749fb2
.long 0x4597456a, 0x4d56973c
.long 0xe9e28eb4, 0x1393e203
.long 0x7b3ff57a, 0x9669c9df
.long 0xc9c8b782, 0xcec3662e
.long 0x3f70cc6f, 0xe417f38a
.long 0x93e106a4, 0x96c515bb
.long 0x62ec6c6d, 0x4b9e0f71
.long 0xd813b325, 0xe6fc4e6a
.long 0x0df04680, 0xd104b8fc
.long 0x2342001e, 0x8227bb8a
.long 0x0a2a8d7e, 0x5b397730
.long 0x6d9a4957, 0xb0cd4768
.long 0xe8b6368b, 0xe78eb416
.long 0xd2c3ed1a, 0x39c7ff35
.long 0x995a5724, 0x61ff0e01
.long 0x9ef68d35, 0xd7a4825c
.long 0x0c139b31, 0x8d96551c
.long 0xf2271e60, 0x0ab3844b
.long 0x0b0bf8ca, 0x0bf80dd2
.long 0x2664fd8b, 0x0167d312
.long 0xed64812d, 0x8821abed
.long 0x02ee03b2, 0xf6076544
.long 0x8604ae0f, 0x6a45d2b2
.long 0x363bd6b3, 0x26f6a60a
.long 0x135c83fd, 0xd8d26619
.long 0x5fabe670, 0xa741c1bf
.long 0x35ec3279, 0xde87806c
.long 0x00bcf5f6, 0x98d8d9cb
.long 0x8ae00689, 0x14338754
.long 0x17f27698, 0x49c3cc9c
.long 0x58ca5f00, 0x5bd2011f
.long 0xaa7c7ad5, 0x68bce87a
.long 0xb5cfca28, 0xdd07448e
.long 0xded288f8, 0x57a3d037
.long 0x59f229bc, 0xdde8f5b9
.long 0x6d390dec, 0x6956fc3b
.long 0x37170390, 0xa3e3e02c
.long 0x6353c1cc, 0x42d98888
.long 0xc4584f5c, 0xd73c7bea
.long 0xf48642e9, 0x3771e98f
.long 0x531377e2, 0x80ff0093
.long 0xdd35bc8d, 0xb42ae3d9
.long 0xb25b29f2, 0x8fe4c34d
.long 0x9a5ede41, 0x2178513a
.long 0xa563905d, 0xdf99fc11
.long 0x45cddf4e, 0xe0ac139e
.long 0xacfa3103, 0x6c23e841
.long 0xa51b6135, 0x170076fa

89
pkg/hash/crc32c/crc32c_linux.go Normal file
View File

@@ -0,0 +1,89 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Golang project:
// https://github.com/golang/go/blob/master/LICENSE
// Using this part of Minio codebase under the license
// Apache License Version 2.0 with modifications
// Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32,
// checksum. See http://en.wikipedia.org/wiki/Cyclic_redundancy_check for
// information.
package crc32c
import (
"hash"
"io"
)
// The size of a CRC-32 checksum in bytes.
const Size = 4
// digest represents the partial evaluation of a checksum.
type digest struct {
crc uint32
}
// New creates a new hash.Hash32 computing the CRC-32 checksum
// using the polynomial represented by the Table.
func New() hash.Hash32 {
return &digest{crc: 0}
}
// Return size of crc
func (d *digest) Size() int { return Size }
// Stub
func (d *digest) BlockSize() int { return 1 }
// Get crc in bytes
func (d *digest) Sum(in []byte) []byte {
s := d.crc
return append(in, byte(s>>24), byte(s>>16), byte(s>>8), byte(s))
}
// Sum32 - return current crc in digest object
func (d *digest) Sum32() uint32 { return d.crc }
// Reset default crc
func (d *digest) Reset() { d.crc = 0 }
// Update returns the result of adding the bytes in p to the crc.
func (d *digest) update(crc uint32, p []byte) uint32 {
return updateCastanagoliPCL(crc, p)
}
// Write data
func (d *digest) Write(p []byte) (n int, err error) {
d.crc = d.update(d.crc, p)
return len(p), nil
}
/// Convenience functions
// Sum32 - single caller crc helper
func Sum32(data []byte) uint32 {
crc32 := New()
crc32.Reset()
crc32.Write(data)
return crc32.Sum32()
}
// Sum - low memory footprint io.Reader based crc helper
func Sum(reader io.Reader) (uint32, error) {
h := New()
var err error
for err == nil {
length := 0
byteBuffer := make([]byte, 1024*1024)
length, err = reader.Read(byteBuffer)
byteBuffer = byteBuffer[0:length]
h.Write(byteBuffer)
}
if err != io.EOF {
return 0, err
}
return h.Sum32(), nil
}

80
pkg/hash/crc32c/crc32c_linux_test.go Normal file
View File

@@ -0,0 +1,80 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Golang project:
// https://github.com/golang/go/blob/master/LICENSE
// Using this part of Minio codebase under the license
// Apache License Version 2.0 with modifications
package crc32c
import (
"testing"
)
type test struct {
castagnoli uint32
in string
}
var golden = []test{
{0x0, ""},
{0x93ad1061, "a"},
{0x13c35ee4, "ab"},
{0x562f9ccd, "abc"},
{0xdaaf41f6, "abcd"},
{0x8122a0a2, "abcde"},
{0x496937b, "abcdef"},
{0x5d199e2c, "abcdefg"},
{0x86bc933d, "abcdefgh"},
{0x9639f15f, "abcdefghi"},
{0x584645c, "abcdefghij"},
{0x8c13a060, "Discard medicine more than two years old."},
{0x629077d4, "He who has a shady past knows that nice guys finish last."},
{0xd20036a4, "I wouldn't marry him with a ten foot pole."},
{0xf283b768, "Free! Free!/A trip/to Mars/for 900/empty jars/Burma Shave"},
{0x9cd61a9f, "The days of the digital watch are numbered. -Tom Stoppard"},
{0x637702f5, "Nepal premier won't resign."},
{0x6c595588, "For every action there is an equal and opposite government program."},
{0x19532076, "His money is twice tainted: 'taint yours and 'taint mine."},
{0x9b82c857, "There is no reason for any individual to have a computer in their home. -Ken Olsen, 1977"},
{0x2b485952, "It's a tiny change to the code and not completely disgusting. - Bob Manchek"},
{0xd3d0980c, "size: a.out: bad magic"},
{0x12aad0bb, "The major problem is with sendmail. -Mark Horton"},
{0x83a0339b, "Give me a rock, paper and scissors and I will move the world. CCFestoon"},
{0x1eb28fde, "If the enemy is within range, then so are you."},
{0xce34d559, "It's well we cannot hear the screams/That we create in others' dreams."},
{0x71576691, "You remind me of a TV show, but that's all right: I watch it anyway."},
{0x54bf536f, "C is as portable as Stonehedge!!"},
{0x2313a94d, "Even if I could be Shakespeare, I think I should still choose to be Faraday. - A. Huxley"},
{0x9d4e3629, "The fugacity of a constituent in a mixture of gases at a given temperature is proportional to its mole fraction. Lewis-Randall Rule"},
{0xc9991fb9, "How can you write a big system without C++? -Paul Glick"},
}
func TestGolden(t *testing.T) {
for _, g := range golden {
s := Sum32([]byte(g.in))
if s != g.castagnoli {
t.Errorf("Castagnoli(%s) = 0x%x want 0x%x", g.in, s, g.castagnoli)
}
}
}
func BenchmarkCrc32KB(b *testing.B) {
b.SetBytes(1024)
data := make([]byte, 1024)
for i := range data {
data[i] = byte(i)
}
h := New()
in := make([]byte, 0, h.Size())
b.ResetTimer()
for i := 0; i < b.N; i++ {
h.Reset()
h.Write(data)
h.Sum(in)
}
}

51
pkg/hash/crc32c/crc32c_windows.go Normal file
View File

@@ -0,0 +1,51 @@
/*
* Minimalist Object Storage, (C) 2014 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package crc32c
import (
"hash/crc32"
"io"
)
var castanagoliTable = crc32.MakeTable(crc32.Castagnoli)
/// Convenience functions
// Sum32 - single caller crc helper
func Sum32(buffer []byte) uint32 {
crc := crc32.New(castanagoliTable)
crc.Reset()
crc.Write(buffer)
return crc.Sum32()
}
// Sum - io.Reader based crc helper
func Sum(reader io.Reader) (uint32, error) {
h := crc32.New(castanagoliTable)
var err error
for err == nil {
length := 0
byteBuffer := make([]byte, 1024*1024)
length, err = reader.Read(byteBuffer)
byteBuffer = byteBuffer[0:length]
h.Write(byteBuffer)
}
if err != io.EOF {
return 0, err
}
return h.Sum32(), nil
}

10
pkg/hash/crc32c/doc.go Normal file
View File

@@ -0,0 +1,10 @@
// Package crc32c provides wrapper around Intel's fast CRC32C with PCLMULQDQ instructions.
// The white papers on CRC32C calculations with PCLMULQDQ instruction can be downloaded from:
//
// http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf
// http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-paper.pdf
//
// Example
//
// crc32c.Crc32c(value) - value can be any []byte, return value is uint32 value
package crc32c