Compare commits

...

2035 Commits

Author SHA1 Message Date
Harshavardhana
1cfd4a48d9 Add specific headers in CORS, along with wildcard (#7726)
Fixes #7492
2019-05-31 09:23:55 -07:00
Harshavardhana
993a79d9c6 Disable http2 until we have upstream bugs fixed (#7711)
We should revert this PR in future once we
have upstream bugs fixed regarding http2 behavior
2019-05-30 19:49:33 -07:00
Harshavardhana
0c16b1c9a7 Fix docs.min.io linking issues (#7710) 2019-05-30 17:19:25 -07:00
Kanagaraj M
900cc27b51 validate keys before updating for IAM user (#7720)
New secretkey should be validated before updating
it on the config.

Fixes #7715
2019-05-30 05:14:35 -07:00
Kanagaraj M
8528017ad3 add min length validation to access key and secret key (#7721)
While changing the credentials through MinIO browser,
Update button will be disabled if keys are lesser than
minimum length.

Fixes #7713
2019-05-30 13:58:22 +05:30
Praveen raj Mani
a73da7755e Remove senstive encryption entries from event data (#7719)
Fixes #7716
2019-05-29 22:29:37 -07:00
Harshavardhana
2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Praveen raj Mani
763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Kanagaraj M
da8214845a allow users to change password through browser (#7683)
Allow IAM users to change the password using
browser UI.
2019-05-29 13:18:46 -07:00
Krishna Srinivas
74e2fe0879 Return "SlowDown" to S3 clients for network related errors (#7610)
Consider errors returned by httpClient.Do() as network errors. This is because
the http clients returns different types of errors and it is hard to catch
all the error types.
2019-05-29 10:21:47 -07:00
Harshavardhana
cb7f9ba286 Add disaggregated architecture document (#7691)
This document describes how to configure Spark,
Hive to use MinIO as replacement for HDFS.
2019-05-29 08:04:51 +05:30
Harshavardhana
7906f464fd Skip non-bucket dns entry (#7699)
Fixes #7697
2019-05-28 19:13:49 -07:00
Harshavardhana
7e4c9a9e1e Properly watch for users, policies, temp users (#7701)
Users were not reloaded properly when etcd was
configured in gateway, server modes.

This PR fixes this issue.
2019-05-28 11:18:53 +05:30
Scott Edlund
f6fd407e47 Add limitation to b2.md (#7680)
The B2 gateway does not implement reporting md5sum as an etag response from PutObject.  Add to Known Limitations.
2019-05-27 21:04:43 -07:00
Nitish Tiwari
46ced81f41 Fix Gateway startup sequence to populate etcd (if set) with bucket info (#7686) 2019-05-24 08:41:52 +05:30
Minio Trusted
7a02faab72 Update yaml files to latest version RELEASE.2019-05-23T00-29-34Z 2019-05-23 00:35:13 +00:00
Dee Koder
e252114f06 Revert "cache: Rewrite to cache only on download (#7575)" (#7684)
This reverts commit a13b58f630.
2019-05-22 14:54:15 -07:00
Harshavardhana
39b3e4f9b3 Avoid using io.ReadFull() for WriteAll and CreateFile (#7676)
With these changes we are now able to peak performances
for all Write() operations across disks HDD and NVMe.

Also adds readahead for disk reads, which also increases
performance for reads by 3x.
2019-05-22 13:47:15 -07:00
Anis Elleuch
158b8c2e86 sets: Correctly set IsTruncated in listing (#7675)
IsTruncated should not be set to true if there is no further
possible entries beyond maxKeys.

This commit will also move wide testing on object API from xl
to xl sets.
2019-05-22 13:36:16 -07:00
Praveen raj Mani
c4c79f61ce Notification: Changes to persistent event store (#7658)
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
2019-05-22 13:34:48 -07:00
Harshavardhana
59e1d94770 Remove stale entry spurious logging (#7663)
The problem in current code was we were removing
an entry from a lock lockerMap without considering
the fact that different entry for same resource is
a possibility due the nature of locks that can be
acquired in parallel before we decide if the lock
is considered stale

A sequence of events is as follows

 - Lock("resource")
 - lockMaintenance(finds a long lived lock in this "resource")
 - Owner node rebooted which now retruns Expired() as true for
   this "resource"
 - Unlock("resource") which succeeded in quorum
 - Now by this time application retried and acquired a new
   Lock() on the same "resource"
 - Now that we have Expired() true from the previous call,
   we proceed to purge the entry from the local lockMap()
   local lockMap reports a different entry for the expired
   UID which results in a spurious log entry.

This PR removes this logging as this situation is an
expected scenario.
2019-05-22 12:21:36 -07:00
Andrei Mikhalenia
59e847aebe Signature v4: Allow signed headers from GET parameters 2019-05-21 21:00:02 -07:00
poornas
a13b58f630 cache: Rewrite to cache only on download (#7575)
This will allow cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

Fixes: #7458, #7573, #6265, #6630
2019-05-22 08:30:27 +05:30
Kaan Kabalak
9e7a19d6b9 Fix object share link expiry always being 7 days (#7669)
The expiry value wasn't being sent to the server on modal value change

Fixes #7668
2019-05-19 17:45:28 -07:00
Harshavardhana
16c648b109 Remove "Connection" close instead reduce MaxConns per host (#7654)
This is necessary to avoid connection build up between servers
unexpectedly for example in a situation where 16 servers are
talking to each other and one server now allows a maximum of

15*4096 = 61440 idle connections

Will be kept in pool. Such a large pool is perhaps inefficient for
many reasons and also affects overall system resources.

This PR also reduces idleConnection timeout from 120 secs to 60 secs.
2019-05-17 12:52:25 +05:30
Krishnan Parthasarathi
c871456269 File must be sync'd before closing (#7657)
- group sync and close action into a single defer statement to avoid
  evaluation order related bugs in future.
2019-05-16 18:30:51 -07:00
Harshavardhana
55aa20595f Remove an empty entry being added into XML marshal (#7656) 2019-05-16 13:09:21 -07:00
poornas
707ed2b302 gcs: use MD5Sum as ETag if present in object attrs (#7643)
Fixes: 7642
2019-05-16 12:00:12 -07:00
ebozduman
78be3f8947 Removes the incorrect coverage badge from the docs (#7651) 2019-05-16 12:11:49 +05:30
ebozduman
67d508b214 Adjusts help content dynamically according to OS (#7646) 2019-05-15 14:02:44 +05:30
Harshavardhana
0022c9d210 Add connection close proactively for Walk() http/rpc (#7645) 2019-05-14 16:10:51 -07:00
Anis Elleuch
9b4a81ee60 xl: Avoid possible race during bulk Multi Delete (#7644)
errs was passed to many goroutines but they are all allowed
to update errs if any error happens during deletion, which
can cause a data race.

This commit will avoid issuing bulk delete operations in parallel
to avoid the warning race.
2019-05-14 14:43:22 -07:00
Harshavardhana
b3f22eac56 Offload listing to posix layer (#7611)
This PR adds one API WalkCh which
sorts and sends list over the network

Each disk walks independently in a sorted manner.
2019-05-14 13:49:10 -07:00
Krishna Srinivas
a343d14f19 Simplify putObject by not breaking the stream into parts (#7199)
We broke into parts previously as we had checksum for the entire file
to tax less on memory and to have better TTFB. We dont need to now,
after the streaming-bitrot change.
2019-05-14 12:33:18 -07:00
Anis Elleuch
9c90a28546 Implement bulk delete (#7607)
Bulk delete at storage level in Multiple Delete Objects API

In order to accelerate bulk delete in Multiple Delete objects API,
a new bulk delete is introduced in storage layer, which will accept
a list of objects to delete rather than only one. Consequently,
a new API is also need to be added to Object API.
2019-05-13 12:25:49 -07:00
Praveen raj Mani
d9a7f80f68 Remove duplicate checkPutObjectArgs in PutObject and (#7396)
Fixes #7384
2019-05-13 10:12:06 -07:00
Krishna Srinivas
502456db91 Accept simpler response from OPA (#7637) 2019-05-12 18:40:49 +05:30
Harshavardhana
3eb7a8bde8 Sync before Close() to avoid random I/O (#7638) 2019-05-11 15:03:10 -07:00
Harshavardhana
ac3b59645e Add API retry functionality in mc admin (#7602) 2019-05-09 17:41:54 -07:00
Harshavardhana
72929ec05b Turn off md5sum optionally if content-md5 is not set (#7609)
This PR also brings --compat option to run MinIO in strict
S3 compatibility mode, MinIO by default will now try to run
high performance mode.
2019-05-08 18:35:40 -07:00
Harshavardhana
757c639044 Separate issue templates for bug report and feature requests (#7616) 2019-05-07 13:49:54 -07:00
Aditya Manthramurthy
589df3d5e7 Deadcode removal (#7627) 2019-05-07 13:49:15 -07:00
Harshavardhana
a71e08efb4 On macOS avoid using GNU specific options (#7621)
Fix Makefile from `cp -uf` to `cp -f`. We are only
interested in `-f` anyways.
2019-05-07 10:39:26 -07:00
kannappanr
c422f7f412 Fix: Handle regression caused by gorilla mux v1.7.0 (#7625)
PR #7595 fixed part of the regression, but did not handle the
scenario, where in docker, the internal port is different from
the port on the host.

This PR modifies the regular expression such that all the
scenarios are handled.

Fixes #7619
2019-05-07 10:36:00 -07:00
Anis Elleuch
08b9244c48 Fix listing empty directory in recursive mode (#7613)
After recent listing refactor, recursive list doesn't return empty
directories, this commit will fix the behavior and add unit tests
so it won't happen again.
2019-05-06 07:52:42 -07:00
kannappanr
efbc665ad3 Delete checked in binary file (#7615)
healthcheck binary was checked in by mistake.
2019-05-03 13:16:35 -07:00
Praveen raj Mani
998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
kannappanr
4b858b562a Compression: Handle auto encryption when size is unknown (#7600)
When size is unknown and auto encryption is enabled,
and compression is set to true, putobject API is failing.

Moving adding the SSE-S3 header as part of the request to before
checking if compression can be done, otherwise the size is set to -1
and that seems to cause problems.
2019-05-02 08:28:18 -07:00
poornas
033f3a4d51 gcs: check error on object writer close (#7606)
Fixes #7605. Object metadata should be written to storage
only when the object was written successfully
2019-05-02 08:27:10 -07:00
poornas
cf2a436bc8 Show SlowDown error message if backend is busy (#7521)
or if there are too many open file descriptors.
2019-05-02 07:09:57 -07:00
Harshavardhana
64998fc4ab Remove delayIsLeaf requirement simplify ListObjects further (#7593) 2019-05-02 10:36:57 +05:30
Krishna Srinivas
43be00ea1b Remove logs from bitrot-streaming.go as erasure layer is already logging (#7603) 2019-05-01 21:46:00 -07:00
Eco
3517f64d20 Clarify server maximums via Federation (#7608)
To eliminate confusion about 32 server limit, clarify that the limit is only per cluster, but that unlimited clusters can be Federated
2019-05-01 16:20:25 -07:00
Harshavardhana
c5f26d5cdd Fix hdfsReader fd leak upon GetObject() (#7596)
Also migrate to minio/hdfs/v3@v3.0.0
2019-05-01 14:43:21 -07:00
Praveen raj Mani
c113d4e49c Posix CreateFile should work for compressed lengths (#7584) 2019-04-30 16:27:31 -07:00
kannappanr
a436f2baa5 Change order of trace source in error log (#7599)
Change the order of trace source that gets
printed on the console.
2019-04-29 14:56:30 -07:00
parnigot
ba76cd3268 Minor TLS documentation improvements (#7592)
Added the explicit name for private.key and public.crt
2019-04-29 12:56:34 -07:00
kannappanr
781012517d Fix: Handle regression caused by gorilla mux v1.7.0 (#7595)
gorilla/mux#383 broke the compatibility with the existing code.
This PR handles that scenario.
2019-04-29 22:03:27 +05:30
Harshavardhana
091b9b661f Complain if we detect sub-optimal ordering in distributed setup (#7576)
Fixes #6156
2019-04-29 10:10:50 +05:30
Harshavardhana
af6c6a2b35 Remove timeout conn on net.Dialer (#7590)
This PR also removes conn_bug_21133 workaround
which is not valid anymore, all we need is deadline
connection with server in place

Fixes #7503
2019-04-27 15:14:16 -07:00
Krishna Srinivas
b93ef73f9b Fix divide by 0 error when directio.AlignSize is 0 (#7591) 2019-04-26 16:08:15 -07:00
Harshavardhana
83ca1a8d64 Use etcd watch to reload IAM users (#7551)
Currently we used to reload users every five minutes,
regardless of etcd is configured or not. But with etcd
configured we can do this more asynchronously to trigger
a refresh by using the watch API

Fixes #7515
2019-04-26 18:48:50 +05:30
Anis Elleuch
27ef1262bf xl: Use random UUID during complete multipart upload (#7527)
One user has seen this following error log:

API: CompleteMultipartUpload(bucket=vertica, object=perf-dss-v03/cc2/02596813aecd4e476d810148586c2a3300d00000013557ef_0.gt)
Time: 15:44:07 UTC 04/11/2019
RequestID: 159475EFF4DEDFFB
RemoteHost: 172.26.87.184
UserAgent: vertica-v9.1.1-5
Error: open /data/.minio.sys/tmp/100bb3ec-6c0d-4a37-8b36-65241050eb02/xl.json: file exists
       1: cmd/xl-v1-metadata.go:448:cmd.writeXLMetadata()
       2: cmd/xl-v1-metadata.go:501:cmd.writeUniqueXLMetadata.func1()

This can happen when CompleteMultipartUpload fails with write quorum,
the S3 client will retry (since write quorum is 500 http response),
however the second call of CompleteMultipartUpload will fail because
this latter doesn't truly use a random uuid under .minio.sys/tmp/
directory but pick the upload id.

This commit fixes the behavior to choose a random uuid for generating
xl.json
2019-04-25 07:33:26 -07:00
Harshavardhana
ae002aa724 Deprecate updating admin credentials using API calls (#7570)
Root credentials are not allowed to change in all of our
distributed setup deployments, this PR simply removes
that behavior.
2019-04-24 12:54:44 -07:00
Krishna Srinivas
a3ec71bc28 Use O_DIRECT while writing to disk (#7479)
- Use O_DIRECT while writing to disk
- Remove MINIO_DRIVE_SYNC option
2019-04-23 21:25:06 -07:00
Minio Trusted
ab711fe1a2 Update yaml files to latest version RELEASE.2019-04-23T23-50-36Z 2019-04-23 23:55:31 +00:00
Harshavardhana
35d19a4ae2 Fix STS AssumeRole route conflict with MultipartUpload (#7574)
Since AssumeRole API was introduced we have a wrong route
match which results in certain clients failing to upload objects
using multipart because, multipart POST conflicts with STS POST
AssumeRole API.

Write a proper matcher function which verifies the route more
appropriately such that both can co-exist.
2019-04-23 15:55:41 -07:00
Harshavardhana
f767a2538a Optimize listing with leaf check offloaded to posix (#7541)
Other listing optimizations include

- remove double sorting while filtering object entries
- improve error message when upload-id is not in quorum
- use jsoniter for full unmarshal json, instead of gjson
- remove unused code
2019-04-23 14:54:28 -07:00
kannappanr
0c75395abe Fix: Allow deleting multiple objects anonymously if policy supports it (#7439)
Fixes #5683
2019-04-22 20:24:43 +05:30
Roman Kalashnikov
188cf1d5ce Add more friendly error message for policy object (#7412) 2019-04-22 01:23:54 -07:00
Praveen raj Mani
d96584ef58 Allow server to start if one of local nodes in docker/kubernetes setup is resolved (#7452)
Allow server to start if one of the local nodes in docker/kubernetes setup is successfully resolved

- The rule is that we need atleast one local node to work. We dont need to resolve the
  rest at that point.

- In a non-orchestrational setup, we fail if we do not have atleast one local node up
  and running.

- In an orchestrational setup (docker-swarm and kubernetes), We retry with a sleep of 5
  seconds until any one local node shows up.

Fixes #6995
2019-04-19 10:26:44 -07:00
Minio Trusted
d42496cc74 Update yaml files to latest version RELEASE.2019-04-18T21-44-59Z 2019-04-18 21:52:22 +00:00
poornas
2c096c569f do not try to delete non-existent object in cache (#7560)
handle cache cleanup correctly when backend object was deleted.

Fixes: #7558
2019-04-18 13:53:22 -07:00
Chris Hoffman
816459d10f Azure gateway complete multipart ETag (#7500)
Compute md5 for azure multipart upload that matches s3 behavior
Reuse complete multipart md5 function in azure gateway
2019-04-17 23:50:25 -07:00
Kanagaraj M
3efcd6fa46 clear errors of previous failed login attempts (#7516)
When the first login attempt is failed(due to incorrect secretkey)
and the next attempt is successful. Error message shown for
the previous attempts should go away.

Fixes #7514
2019-04-18 11:49:51 +05:30
kannappanr
d2f42d830f Lock: Use REST API instead of RPC (#7469)
In distributed mode, use REST API to acquire and manage locks instead
of RPC.

RPC has been completely removed from MinIO source.

Since we are moving from RPC to REST, we cannot use rolling upgrades as the
nodes that have not yet been upgraded cannot talk to the ones that have
been upgraded.

We expect all minio processes on all nodes to be stopped and then the
upgrade process to be completed.

Also force http1.1 for inter-node communication
2019-04-17 23:16:27 -07:00
Minio Trusted
7686340621 Update yaml files to latest version RELEASE.2019-04-18T01-15-57Z 2019-04-18 01:22:00 +00:00
Harshavardhana
4c048963dc etcd: Handle create buckets with common prefixes properly (#7556)
common prefixes in bucket name if already created
are disallowed when etcd is configured due to the
prefix matching issue. Make sure that when we look
for bucket we are only interested in exact bucket
name not the prefix.
2019-04-17 17:29:49 -07:00
Harshavardhana
620e462413 Implement S3-HDFS gateway (#7440)
- [x] Support bucket and regular object operations
- [x] Supports Select API on HDFS
- [x] Implement multipart API support
- [x] Completion of ListObjects support
2019-04-17 09:52:08 -07:00
poornas
1d49295943 Close CacheReader before clearing cache entry if object is deleted (#7555)
Fixes: #7549
2019-04-17 11:24:50 +05:30
Nitish Tiwari
ab7252831e Fix healthcheck script to check for BadRequest Status (#7537)
As a part of #7302, MinIO server's (configured with https) response when it
encounters http request has changed from 403 to 400 and the custom message
"SSL Required" is removed.

Accordingly healthcheck script is updated to check for status 400 before
trying https request.

Fixes #7517
2019-04-12 22:29:12 -07:00
Krishnan Parthasarathi
35ef5eb236 Don't exit background append if backend specific files show up (#7519) 2019-04-12 15:51:32 -07:00
kannappanr
f409f10d18 Fix SimpleCI to use different data directory than mint (#7520)
Currently, the backend minio server uses the same data directory
as the mint test itself, causing `s3 sync` to fail often.

Now `minio` backend will use a different data directory `/data`
instead of `/mint/data`
2019-04-12 12:51:36 -07:00
Anis Elleuch
60d6887992 Make Encoding URL more compliant to S3 spec (#7360)
There is no written specification about how to encode key names
when url encoding type is passed.

However, this change will encode URLs as url.QueryEscape() does
while considering AWS S3 exceptions.
2019-04-12 12:02:37 -07:00
Andreas Auernhammer
012e4b42f9 http: opt-in to TLS 1.3 (#7483)
This commit enables TLS 1.3 on the server. For Go 1.12 TLS 1.3 is
enabled by an explicit opt-in.
2019-04-11 20:46:15 -07:00
poornas
a74cb93666 Worm: Permit key-rotation of S3 encrypted objects (#7429)
Fixes : #7399
2019-04-10 11:31:50 -07:00
Andreas Auernhammer
849e06a316 crypto: add unit test for vault config verification (#7413)
This commit adds a unit test for the vault
config verification (which covers also `IsEmpty()`).

Vault-related code is hard to test with unit tests
since a Vault service would be necessary. Therefore
this commit only adds tests for a fraction of the code.

Fixes #7409
2019-04-10 11:05:53 -07:00
Praveen raj Mani
47ca411163 Enhance the event store interface to support channeling (#7343)
- Avoids code duplication across the other targets. By having a
  centralized function call.

- Reduce the room for race.
2019-04-10 18:16:01 +05:30
Aditya Manthramurthy
ddb0d646aa Use passed lock-type in GetObjectNInfo cache implementation (#7505) 2019-04-09 14:49:45 -07:00
kannappanr
5ecac91a55 Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
kannappanr
188ac8e369 Browser: Allow users to do s3 operations, if policy allows (#7487)
Fixes #7472
2019-04-09 20:47:41 +05:30
Minio Trusted
ea052a2a00 Update yaml files to latest version RELEASE.2019-04-09T01-22-30Z 2019-04-09 01:30:27 +00:00
Harshavardhana
a2e344bf30 Preserve ETag case for S3 compatibility (#7498)
Most hadoop distributions hortonworks, cloudera all
depend on aws-sdk-java 1.7.x to 1.10.x - the releases
which have bugs related case sensitive check for
ETag header. Go changes the case of the headers set
to be canonical but only preserves them when set
through a direct map.

This fixes most compatibility issues we have had
in the past supporting older hadoop distributions.
2019-04-08 16:54:46 -07:00
poornas
10a607154d Fix ListObjectsV2 for gateway encryption mode (#7491)
Fixes #7468 by setting NextContinuationToken only if list is
truncated
2019-04-08 15:12:00 -07:00
Harshavardhana
720ed3f5e8 Add go-mod cache for .travis to speed up builds (#7477) 2019-04-05 12:00:31 +05:30
Harshavardhana
0188009c7e Expose total and available disk space (#7453) 2019-04-05 09:51:50 +05:30
Harshavardhana
979309148e Add doc updates for new go1.12 installation (#7486) 2019-04-04 13:36:31 -07:00
Minio Trusted
082bd5f31f Update yaml files to latest version RELEASE.2019-04-04T18-31-46Z 2019-04-04 18:39:37 +00:00
Harshavardhana
0146a9d592 Fix docker files to enable GO111MODULE 2019-04-04 11:06:50 -07:00
Aditya Manthramurthy
b1b1d77893 Set S3 Select record message length to 128KiB (#7475)
- Previously this limit was a little more than 1MiB, and it broke
  compatibility with AWS SDK Java causing a buffer overflow error.
2019-04-04 00:41:52 -07:00
Harshavardhana
c90999df98 Valid if bucket names are internal (#7476)
This commit fixes a privilege escalation issue against
the S3 and web handlers. An authenticated IAM user
can:

- Read from or write to the internal '.minio.sys'
bucket by simply sending a properly signed
S3 GET or PUT request. Further, the user can
- Read from or write to the internal '.minio.sys'
bucket using the 'Upload'/'Download'/'DownloadZIP'
API by sending a "browser" request authenticated
with its JWT token.
2019-04-03 23:10:37 -07:00
Andreas Auernhammer
9a740736a4 fix privilege escalation against inter-node communication (#7474)
This commit fixes another privilege escalation issue
abusing the inter-node communication of distributed
servers to obtain/modify the server configuration.

The inter-node communication is authenticated using
JWT-Tokens. Further, IAM users accessing the cluster
via the web UI also get a JWT token and the browser
will add this "user" JWT token to each the request.

Now, a user can extract that JWT token an can craft
HTTP POST requests for the inter-node communication
API endpoint. Since the server accepts ANY valid
JWT token it also accepts inter-node commands from
an authenticated user such that the user can execute
arbitrary commands bypassing the IAM policy engine
and impersonate other users, change its own IAM policy
or extract the admin access/secret key.

This is fixed by only accepting "admin" JWT tokens
(tokens containing the admin access key - and therefore
were generated with the admin secret key). Consequently,
only the admin user can execute such inter-node commands.
2019-04-03 12:16:19 -07:00
Harshavardhana
313a3a286a Migrate to go1.12 to simplify our cmd/http package (#7302)
Simplify the cmd/http package overall by removing
custom plain text v/s tls connection detection, by
migrating to go1.12 and choose minimum version
to be go1.12

Also remove all the vendored deps, since they
are not useful anymore.
2019-04-02 18:28:39 -07:00
Anis Elleuch
4c23e6fa55 rpc: Avoid using Pool since it conflicts with http2 (#7467)
A race is detected between a bytes.Buffer generated with cmd/rpc.Pool
and http2 module. An issue is raised in golang (https://github.com/golang/go/issues/31192).

Meanwhile, this commit disables Pool in RPC code and it generates a
new 1kb of bytes.Buffer for each RPC call.
2019-04-02 13:34:21 -07:00
Krishna Srinivas
ef791764e0 Do no access nsLockMap.lockMap when using dsync (#7464)
There is no need to access nsLockMap.lockMap when using dsync
2019-04-02 12:27:20 -07:00
Anis Elleuch
53011606a5 Show 401 unauthorized msg when nodes are started with different creds (#7433)
Before this commit, nodes wait indefinitely without showing any
indicate error message when a node is started with different access
and secret keys.

This PR will show '401 Unauthorized' in this case.
2019-04-02 12:25:34 -07:00
Krishnan Parthasarathi
93a9078b23 Assign deploymentID for first minio server in distributed setup (#7427)
- Pass local endpoints to functions fixing formatXL during startup
2019-04-02 10:50:13 -07:00
Ashish Kumar Sinha
a4bdcba503 Add check for extra input field (#7437)
fixes #6559
2019-04-02 12:15:32 +05:30
poornas
023866642c canonicalize ETag correctly (#7442)
Fixes #7441 
Trim extra quotes prefixing/suffixing ETag in
CompleteMultipartUpload request.
2019-04-01 12:19:52 -07:00
Harshavardhana
619611933a Remove policy nesting errors (#7449)
Policy nesting has been supported for a while
now, we should remove references of code and
docs indicating nesting is not allowed anymore.
2019-03-31 08:42:43 -07:00
poornas
dd1d69ab5c fix mysql config for native password authentication (#7436)
fixes #7430
2019-03-28 10:45:22 -07:00
Harshavardhana
16ba679103 Fix markdown table formatting (#7435) 2019-03-27 16:16:58 -07:00
Anis Elleuch
e13c99ed82 Makefile: Fix getting misspell code when already installed (#7434)
Grouping misspell installation code so it won't executed when
misspell is found in the current system.
2019-03-27 16:16:17 -07:00
Minio Trusted
8cf707779c Update yaml files to latest version RELEASE.2019-03-27T22-35-21Z 2019-03-27 22:41:57 +00:00
Harshavardhana
6df05e489d Set Read/Write timeouts only for net.Conn not http.Server (#7431)
Fixes #7425
2019-03-27 22:10:06 +05:30
Harshavardhana
4a698c731b HealObjects should remove objects without quorum (#7407)
This PR adds a way to list objects without quorum
such that they can purged by `mc admin heal --remove`
2019-03-26 14:57:44 -07:00
Harshavardhana
9629de8230 Add proper context based logging when bitrot stream calls fail (#7415) 2019-03-26 13:59:33 -07:00
Harshavardhana
0250f7de67 Cleanup stale multipart uploads older than 3 days (#7424)
Fixes #6627
2019-03-25 13:41:05 -07:00
kannappanr
7154b8a568 Error log: Correct error type in anonymous mode (#7414)
Currently message is set to error type value.
Message field is not used in error logs. it is used only in the case of info logs.

This PR sets error message field to store error type correctly.
2019-03-25 13:40:08 -07:00
Anis Elleuch
8689ec258b Don't decrypt ETag in validation when source is SSEC multipart (#7423)
Copying an encrypted SSEC object when this latter is uploaded using
multipart mechanism was failing because ETag in case of encrypted
multipart upload is not encrypted.

This PR fixes the behavior.
2019-03-25 12:17:31 -07:00
Krishnan Parthasarathi
aac9e2a7dd Return deploymentID in ServerInfo REST call (#7422)
Makes deploymentID information uniform in distributed setup
2019-03-25 11:55:28 -07:00
Harshavardhana
719d21efd8 Generate coverage across all sub-dirs (#7416) 2019-03-25 11:54:14 -07:00
Harshavardhana
e0a87e96de Populate host value from GetSourceIP directly (#7417) 2019-03-25 11:45:42 -07:00
Harshavardhana
0a44e70177 Change findEndpoint to return errors for Scanlines (#7390) 2019-03-25 13:31:02 +05:30
Rushan
22b4fe0a51 Update browser UI with new logo and colors (#7408) 2019-03-23 09:57:09 -07:00
Praveen raj Mani
89e45d0695 Restart process should use the current process' pid (#7373)
This fixes varying pids for server-respawns. And avoids duplicate process
creating multiple pids when the server restart signal is triggered with
service restart enabled.

Fixes #7350
2019-03-20 22:20:30 -07:00
poornas
8e1e701d35 Azure:ETag returned by ListObjects to be consistent with GetObjectInfo (#7301) 2019-03-20 18:11:46 -07:00
kannappanr
87cf51d5ab unused code: Remove LoadCredentials function (#7369)
It is required to set the environment variable in the case of distributed
minio. LoadCredentials is used to notify peers of the change and will not work if
environment variable is set. so, this function will never be called.
2019-03-20 18:09:57 -07:00
Minio Trusted
8af45f0135 Update yaml files to latest version RELEASE.2019-03-20T22-38-47Z 2019-03-20 22:49:06 +00:00
Harshavardhana
a9032b52b8 Change storageRESTTimeout to 1minute (#7398) 2019-03-20 13:20:09 -07:00
Harshavardhana
c184038b6a Add proper custom errors object creations (#7387)
In scenario 1

```
- bucket/object-prefix
- bucket/object-prefix/object
```

Server responds with `XMinioParentIsObject`

In scenario 2

```
- bucket/object-prefix/object
- bucket/object-prefix
```

Server responds with `XMinioObjectExistsAsDirectory`

Fixes #6566
2019-03-20 13:06:53 -07:00
poornas
12b79d9f3b Remove duplicate error in switch case. (#7381)
Fixes: #7380

crypto.ErrInvalidCustomerKey was being handled twice in toAPIErrorCode()
2019-03-19 17:21:05 -07:00
Sidhartha Mani
6bc0de2a75 add go modules file and start running go 1.11 style builds (#7354) 2019-03-19 13:50:58 -07:00
Krishnan Parthasarathi
8a77a298f2 Add deploymentID to ServerInfo (#7372) 2019-03-19 16:12:24 +05:30
Harshavardhana
328eb74cbb Fix regression in peer clients in TLS setups (#7391)
Regression was introduced in eb69c4f946
2019-03-19 09:44:49 +05:30
zy
73be3ed0ca format import style (#7383) 2019-03-18 13:07:58 -07:00
Kirill Motkov
3d29ab4059 Rewrite if-else chains to switch statements (#7382) 2019-03-18 07:46:20 -07:00
Harshavardhana
6702d23d52 Simplify ReadFileStream closer, make sure to flush all HTTP responses (#7374) 2019-03-18 10:50:26 +05:30
poornas
1011d21416 Fix credential parsing in signature v4 (#7377)
Fixes #7376
2019-03-16 22:45:42 -07:00
Kirill Motkov
85c5acc088 fix staticcheck warning (#7378) 2019-03-16 22:44:43 -07:00
kannappanr
eb69c4f946 Use REST api for inter node communication (#7205) 2019-03-14 16:27:31 -07:00
Anis Elleuch
facbd653ba Add normal/deep type of heal scanning (#7251)
Healing scan used to read all objects parts to check for bitrot
checksum. This commit will add a quicker way of healing scan
by only checking if parts are actually present in disks or not.
2019-03-14 13:08:51 -07:00
Harshavardhana
233824bf92 Configure http2 with higher maxconcurrent streams (#7363)
This value is needed for Minio's internode communication,
read the meaning of this value as per the HTTP 2.0 spec

https://http2.github.io/http2-spec/#rfc.section.5.1.2
2019-03-14 11:57:35 -07:00
poornas
a0ee7be050 update link on README (#7361) 2019-03-14 15:18:12 +05:30
Harshavardhana
91d85a0d53 Fix stale locks held by SelectParquet API (#7364)
Vendorize upstream parquet-go to fix this issue.
2019-03-13 20:33:18 -07:00
Harshavardhana
7079abc931 Implement HealObjects API to simplify healing (#7351) 2019-03-13 17:35:09 -07:00
Minio Trusted
8377d00574 Update yaml files to latest version RELEASE.2019-03-13T21-59-47Z 2019-03-13 22:06:26 +00:00
Harshavardhana
285c09fe6b Support buckets with '.' with etcd+coreDNS (#7353)
Fixes #7340
2019-03-12 17:57:08 -07:00
kannappanr
ce4563370c Distributed: Allow healing if all disks are on root partitions (#7358)
If all the disks are on root partitions in distributed mode, consider it
to be a test setup and allow healing to proceed.

Fixes #7346
2019-03-12 16:47:06 -07:00
Aditya Manthramurthy
e463386921 Add JSON Path expression evaluation support (#7315)
- Includes support for FROM clause JSON path
2019-03-09 08:13:37 -08:00
Aditya Manthramurthy
b296b3cf8b Skip non-bucket dns entry in federated bucket list (#7339) 2019-03-08 14:22:58 -08:00
Harshavardhana
0b96ad4fdc http2 throws custom error Content-Length shorter handle it (#7334)
We should internally handle when http2 input stream has smaller
content than its content-length header

Upstream issue reported https://github.com/golang/go/issues/30648

This a change which we need to handle internally until Go fixes it
correctly, till now our code doesn't expect a custom error to be returned.
2019-03-07 16:11:28 -08:00
Aditya Manthramurthy
f4879ed96d Use jstream to serialize records to JSON format in S3Select (#7318)
- Also, switch to jstream to generate internal record representation
  from CSV/JSON readers

- This fixes a bug in which JSON output objects have their keys
  reversed from the order they are specified in the Select columns.

- Also includes a fix for tests.
2019-03-07 00:20:10 -08:00
Aditya Manthramurthy
f97a33a63f Downgrade node temporarily (#7333)
To fix jest related build failure as recommended at https://github.com/facebook/jest/issues/8069#issuecomment-470307590
2019-03-06 16:05:49 -08:00
Minio Trusted
6387407e79 Update yaml files to latest version RELEASE.2019-03-06T22-47-10Z 2019-03-06 22:56:46 +00:00
Anis Elleuch
b05825ffe8 s3: Fix precondition failed in CopyObjectPart when src is encrypted (#7276)
CopyObject precondition checks into GetObjectReader
in order to perform SSE-C pre-condition checks using the
last 32 bytes of encrypted ETag rather than the decrypted
ETag

This also necessitates moving precondition checks for
gateways to gateway layer rather than object handler check
2019-03-06 12:38:41 -08:00
kannappanr
39ddb78c75 CORS: Expose all headers on response (#7331)
Fixes #7289
2019-03-06 11:58:53 -08:00
Harshavardhana
12eb71828b Fix posix tests for SimpleCI (#7328) 2019-03-05 19:53:01 -08:00
Praveen raj Mani
c0a1369b73 Construct dynamic XML error responses for postpolicyform validation (#7321)
Fixes #7314
2019-03-05 12:10:47 -08:00
kannappanr
c57159a0fe fs mode: List already existing buckets with capital letters (#7244)
if a bucket with `Captialized letters` is created, `InvalidBucketName` error
will be returned. 
In the case of pre-existing buckets, it will be listed.

Fixes #6938
2019-03-05 10:42:32 -08:00
Kale Blankenship
ef132c5714 Replace snappy.Writer/io.Pipe with snappyCompressReader. (#7316)
Prevents deferred close functions from being called while still
  attempting to copy reader to snappyWriter.
 Reduces code duplication when compressing objects.
2019-03-05 08:35:37 -08:00
Aditya Manthramurthy
c54b0c0ca1 Fix a race in tests (#7326) 2019-03-05 21:34:17 +05:30
Aditya Manthramurthy
e8e9cd3e74 Close GlobalServiceDoneCh when quitting (#7322)
This change allows indefinitely running go-routines to cleanup
gracefully.

This channel is now closed at the beginning of each test so that
long-running go-routines quit and a new one is assigned.
2019-03-04 14:33:14 -08:00
Sidhartha Mani
f7611bcd48 update simple-ci scripts (#7309) 2019-03-03 21:59:14 +05:30
Aditya Manthramurthy
19c10cb4d0 Update vendored sjson (includes a bug fix) (#7317) 2019-03-01 15:04:38 -08:00
poornas
6b4c6f69af Remove minio-go validator from vendor dir (#7312) 2019-02-28 12:16:26 -08:00
poornas
2564147ab4 Filter Expires header from user metadata (#7269)
Instead save it as a struct field in ObjectInfo as it is
a standard HTTP header - Fixes minio/mc#2690
2019-02-28 11:01:25 -08:00
Harshavardhana
c3ca954684 Implement AssumeRole API for Minio users (#7267)
For actual API reference read here

https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html

Documentation is added and updated as well at docs/sts/assume-role.md

Fixes #6381
2019-02-27 17:46:55 -08:00
Harshavardhana
ce588d1489 Improve ListObjects performance by listing in parallel (#7270)
The side affect of this change memory
increase, but this is a trade-off between
performance and actual memory usage.

For all practical scenarios this should be
an adequate change.
2019-02-27 14:39:22 -08:00
Sidhartha Mani
b983da957d run gateway mint test in full mode (#7296) 2019-02-27 10:03:23 -08:00
Harshavardhana
5e69a107d8 Support detecting mountpoints correctly (#7288)
Currently windows support was relying on Symlink as
a way to detect a drive, this doesn't work in latest
Windows 2016, fix this to use a proper mechanism by
using win32 APIs.

Additionally also add support for detecting bind mounts
on Linux.
2019-02-26 18:04:53 -08:00
Dee Koder
5085bef397 Share button icon is changed (#7300) 2019-02-26 16:34:29 -08:00
poornas
8022a6efd9 Return ETag for 0-byte object prefixes (#7291)
Fixes: #7290
2019-02-26 15:09:14 -08:00
Minio Trusted
d3125857a8 Update yaml files to latest version RELEASE.2019-02-26T19-51-46Z 2019-02-26 19:58:09 +00:00
Praveen raj Mani
78d116c487 Event persistence for MQTT (#7268)
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.

The events are replayed back when the mqtt broker is back online.
2019-02-25 18:01:13 -08:00
dcharbonnier
2fc341394d fix json syntax (#7285) 2019-02-25 13:02:59 +05:30
Anis Elleuch
6584c7ea2b s3: Encode StartAfter when encoding type is passed (#7281)
In ObjectList V2, StartAfter needs to be encoded when encoding-type
is specified.
2019-02-24 18:50:28 -08:00
Harshavardhana
2520e535a0 Allow lazyQuotes for certain types of CSV (#7278)
Set lazyQuotes to true, to allow a quote to appear
in an unquote field and a non-doubled quote may
appear in a quoted field.
2019-02-24 06:51:02 -08:00
Anis Elleuch
5efbe8a1b3 s3: Add support of encodingType parameter (#7265)
This commit honors encoding-type parameter in object listing,
parts listing and multipart uploads listing.
2019-02-24 11:44:24 +05:30
Nitish Tiwari
dab314900d Fix healthcheck script to wait for netstat command output (#7275)
Fixes #7272
2019-02-23 09:32:53 -08:00
Harshavardhana
7923b83953 Support multiple-domains in MINIO_DOMAIN (#7274)
Fixes #7173
2019-02-23 08:48:01 +05:30
Aditya Manthramurthy
80a351633f Update vendorized bcicen/jstream (#7257)
- Includes an error handling fix that is waiting to be merged upstream
- Uses order-preserving (un)marshalling for JSON objects.
2019-02-20 23:59:23 -08:00
Harshavardhana
bedcb7442a Write xml.Header first instead of spaces to handle XML parsers (#7253)
Clients like AWS SDK Java and AWS cli XML parsers are
unable to handle on `\r\n` characters to avoid these
errors send XML header first and write white space characters
instead.

Also handle cases to avoid double WriteHeader calls
2019-02-21 11:50:15 +05:30
Harshavardhana
2232b0b55f Fix mac build failure for healthcheck binary (#7263) 2019-02-21 08:11:16 +05:30
Harshavardhana
91576d416d Fix GetLocalPeer usage in perf handlers (#7249)
GetLocalPeer usage should be fixed and used only
once per call for not all local endpoints.
2019-02-20 16:04:55 -08:00
Minio Trusted
3aabe45fd9 Update yaml files to latest version RELEASE.2019-02-20T22-44-29Z 2019-02-20 22:49:40 +00:00
Krishna Srinivas
6dd26b8231 Detect change in underlying mounted disks (#7229) 2019-02-20 13:32:29 -08:00
poornas
e098852a80 Revert PR #7241 to fix vault renewal (#7259)
- Current implementation was spawning renewer goroutines
without waiting for the lease duration to end. Remove vault renewer
and call vault.RenewToken directly and manage reauthentication if
lease expired.
2019-02-20 12:23:59 -08:00
Nitish Tiwari
1e82c4a7c4 Implement Docker healthcheck script in Go (#7105)
Go script makes it easy to read/maintain. Also updated the timeout
in Dockerfiles from 5s to default 30s and test interval to 1m

Higher timeout makes sense as server may sometimes respond slowly
if under high load as reported in #6974

Fixes #6974
2019-02-20 21:42:03 +05:30
Krishna Srinivas
ce960565b1 Validate and reject unusual requests (#7258) 2019-02-19 21:02:41 -08:00
poornas
755e675d5c Fix: send decrypted size to notification event (#7248) 2019-02-19 14:14:26 +05:30
Harshavardhana
b6c00405ec Do not pro-actively return false in isObjectDir() (#7246)
We should change the logic for both isObject()
and isObjectDir() leaf detection to be done
with quorum, due to how our directory navigation
works - this allows for properly deleting all
the dangling directories or objects if any.
2019-02-15 16:21:19 -08:00
Harshavardhana
8f62935448 Add proper requestID for STS errors (#7245) 2019-02-14 17:54:33 -08:00
Harshavardhana
396d78352d Support HTTP/2.0 (#7204)
Fixes #6704
2019-02-14 17:53:46 -08:00
Aditya Manthramurthy
8a405cab2f COUNT() function in select should return an int (#7243) 2019-02-13 16:32:59 -08:00
Minio Trusted
6d778a883f Update yaml files to latest version RELEASE.2019-02-14T00-21-45Z 2019-02-14 00:27:52 +00:00
Harshavardhana
a51781e5cf Use context to fill in more details about error XML (#7232) 2019-02-13 16:07:21 -08:00
Krishna Srinivas
90213ff1b2 Detect peer reboots to invalidate current storage REST clients (#7227) 2019-02-13 15:29:46 -08:00
Andreas Auernhammer
6f764a8efd crypto: fix nil pointer dereference of vault secret (#7241)
This commit fixes a nil pointer dereference issue
that can occur when the Vault KMS returns e.g. a 404
with an empty HTTP response. The Vault client SDK
does not treat that as error and returns nil for
the error and the secret.

Further it simplifies the token renewal and
re-authentication mechanism by using a single
background go-routine.

The control-flow of Vault authentications looks
like this:
1. `authenticate()`: Initial login and start of background job
2. Background job starts a `vault.Renewer` to renew the token
3. a) If this succeeds the token gets updated
   b) If this fails the background job tries to login again
4. If the login in 3b. succeeded goto 2. If it fails
   goto 3b.
2019-02-13 15:25:32 -08:00
Harshavardhana
df35d7db9d Introduce staticcheck for stricter builds (#7035) 2019-02-13 18:29:36 +05:30
Harshavardhana
4ba77a916d Select should return early errors as XML (#7230)
Currently, we were sending errors in Select binary format,
which is incompatible with AWS S3 behavior, errors in binary
are  sent after HTTP status code is already 200 OK - i.e it
happens during the evaluation of the record reader.
2019-02-13 13:18:11 +05:30
Anis Elleuch
f9fecf0e76 storage: Increase the timeout of storage REST requests (#7218)
This commit increases storage REST requests to 5 minutes, this includes
the opening TCP connection, and sending/receiving data. This will reduce
clients receiving errors when the server is under high load.
2019-02-12 23:27:33 -08:00
Minio Trusted
9f9e0fe085 Update yaml files to latest version RELEASE.2019-02-12T21-58-47Z 2019-02-12 22:07:04 +00:00
Aditya Manthramurthy
ee5b3622a5 Evaluate where clause in aggregation queries (#7235) 2019-02-12 13:54:26 -08:00
Krishna Srinivas
14544d8d84 Validate incoming requests (#7234) 2019-02-12 13:24:14 -08:00
Harshavardhana
118270d76f Vendor the upstream changes with Avx512 (#7225)
Thanks to @fwessels we have Avx512 support with 4x improvement
2019-02-12 17:32:27 +05:30
Harshavardhana
fef5416b3c Support unknown gateway errors and convert at handler layer (#7219)
Different gateway implementations due to different backend
API errors, might return different unsupported errors at
our handler layer. Current code posed a problem for us because
this information was lost and we would convert it to InternalError
in this situation all S3 clients end up retrying the request.

To avoid this unexpected situation implement a way to support
this cleanly such that the underlying information is not lost
which is returned by gateway.
2019-02-12 14:55:52 +05:30
Harshavardhana
9f87283cd5 Revert and bring back B2 gateway implementation (#7224)
This PR is simply a revert of 3265112d04
just for B2 gateway.
2019-02-12 12:44:22 +05:30
Harshavardhana
b8955fe577 Fix DummyHandlers to authorize and send/validate correct XMLs (#7223) 2019-02-11 17:58:26 -08:00
Nitish Tiwari
13c3b8afe2 Update Redis client used for bucket notifications (#7213)
Current Redis client used https://github.com/garyburd/redigo/
has been archived and further development is done at
https://github.com/gomodule/redigo

This commit updates the client library accordingly.

Fixes #6392
2019-02-11 19:06:27 +05:30
Harshavardhana
a8cd70f3e5 Remove GPL go-lzo dependency for parquet-go (#7220)
Also remove any other unused dependencies
2019-02-11 14:57:24 +05:30
Harshavardhana
082f777281 Revamp bucket metadata healing (#7208)
Bucket metadata healing in the current code was executed multiple
times each time for a given set. Bucket metadata just like
objects are hashed in accordance with its name on any given set,
to allow hashing to play a role we should let the top level
code decide where to navigate.

Current code also had 3 bucket metadata files hardcoded, whereas
we should make it generic by listing and navigating the .minio.sys
to heal such objects.

We also had another bug where due to isObjectDangling changes
without pre-existing bucket metadata files, we were erroneously
reporting it as grey/corrupted objects.

This PR fixes all of the above items.
2019-02-11 09:23:13 +05:30
John Liu
9600e2b35e Comment Typo: Changed 'jason' to 'json` (#7216) 2019-02-10 05:49:00 -08:00
Prashant Shahi
f75f707ff4 Added documentation for MINIO_HTTP_TRACE (#7207) 2019-02-10 11:57:43 +05:30
poornas
40b8d11209 Move metadata into ObjectOptions for NewMultipart and PutObject (#7060) 2019-02-09 11:01:06 +05:30
Sidhartha Mani
c1b3f1994b remove unnecessary buffer while discarding stream (#7214) 2019-02-08 19:29:51 -08:00
Adam
18c4ecbbef Fix mc event add event flag (#7210) 2019-02-08 16:18:34 -08:00
ebozduman
dd52e5ebe9 Implements dummy tagging handlers for Terraform (#7040) 2019-02-08 16:18:13 -08:00
Praveen raj Mani
8af1f0cc7b Improved error message for user and access key conflict (#7190) 2019-02-07 17:25:58 -08:00
Harshavardhana
85e939636f Fix JSON parser handling for certain objects (#7162)
This PR also adds some comments and simplifies
the code. Primary handling is done to ensure
that we make sure to honor cached buffer.

Added unit tests as well

Fixes #7141
2019-02-07 08:04:42 +05:30
poornas
d203e7e1cc azure gateway: return MD5Sum as ETag for S3 API compatibility (#6884)
Fixes #6872.

This PR refactors multipart upload implementation to use a per
part metadata file which is cleaned up at the end of the upload
2019-02-06 16:58:43 -08:00
Aditya Manthramurthy
4aa9ee153b Fix S3 Select request XML parsing (#7202) 2019-02-06 13:25:52 -08:00
Minio Trusted
5fb813a5cc Update yaml files to latest version RELEASE.2019-02-06T21-16-36Z 2019-02-06 21:24:08 +00:00
Harshavardhana
817269475f Make sure to drain body upon an error (#7197)
Also cleanup redundant code and use it at a common place
2019-02-06 12:07:03 -08:00
Krishna Srinivas
2d168b532b Allow format.json healing on dev/test setup (single node XL, all root disks) (#7170) 2019-02-06 11:44:19 -08:00
Aditya Manthramurthy
fd4e15c116 Flush the records staging buffer periodically (#7193)
- Staging buffer is flushed every 500ms. In cases where the result
  records are slowly generated (e.g. when a where condition
  matches very few records), this change causes the server to send
  results even though the staging buffer is not full.

- Refactor messageWriter code to use simpler channel based
  co-ordination instead of atomic variables.
2019-02-06 16:03:05 +05:30
Krishna Srinivas
3dfbe0f68c Send white spaces to client till completeMultipart() process completes (#7198) 2019-02-05 20:58:09 -08:00
Harshavardhana
30135eed86 Redo how to handle stale dangling files (#7171)
foo.CORRUPTED should never be created because when
multiple sets are involved we would hash the file
to wrong a location, this PR removes the code.

But allows DeleteBucket() to work properly to delete
dangling buckets/objects. Also adds another option
to Healing where a user needs to specify `--remove`
such that all dangling objects will be deleted with
user confirmation.
2019-02-05 17:58:48 -08:00
Harshavardhana
e4081aee62 Added support for reading body in STS API (#7188)
STS API supports both URL query params and reading
from a body.
2019-02-05 15:47:11 -08:00
kannappanr
df418a2783 Create Cors handler with permissive configuration (#7186)
Create new Cors handler allowing all origins with all standard
methods with any header and credentials.

Fixes #7181
2019-02-05 14:06:52 -08:00
kannappanr
9a65f6dc97 Remove duplicate code in object-handlers.go (#7176)
removed duplicate code in CompleteMultipartUploadHandler
and CopyObjectPartHandler.
2019-02-05 13:36:38 -08:00
Aditya Manthramurthy
f04f8bbc78 Add support for Timestamp data type in SQL Select (#7185)
This change adds support for casting strings to Timestamp via CAST:
`CAST('2010T' AS TIMESTAMP)`

It also implements the following date-time functions:
  - UTCNOW()
  - DATE_ADD()
  - DATE_DIFF()
  - EXTRACT()

For values passed to these functions, date-types are automatically
inferred.
2019-02-04 20:54:45 -08:00
Harshavardhana
ea6d61ab1f Use loadCachedConfigs appropriately to load ENVs (#7187) 2019-02-04 10:31:11 +05:30
Krishna Srinivas
6f08edfb36 Use O_EXCL when creating file as we never overwrite an existing file (#7189) 2019-02-01 19:01:06 -08:00
Sidhartha Mani
e9fdea05c6 Enable CI control from repository: Add Dockerfile.simpleci (#7122) 2019-02-01 12:04:28 -08:00
Harshavardhana
e005910051 Add more information in our select docs (#7177) 2019-02-01 11:34:56 -08:00
Anis Elleuch
de2c106386 xl: ListObjectParts uses the latest valid xl meta (#7184)
ListObjectParts is using xl.readXLMetaParts which picks the first
xl meta found in any disk, which is an inconsistent information.

E.g.: In a middle of a multipart upload, one node can go offline
and get back later with an outdated multipart information.
2019-02-01 08:58:41 -08:00
Harshavardhana
32a6dd1dd6 Remove sporadic tests which fail on windows (#7178) 2019-01-31 16:48:47 -08:00
Harshavardhana
432aec73d9 Return proper errors for invalid bodies (#7179) 2019-01-31 07:19:09 -08:00
Anis Elleuch
36dae04671 CopyObjectPart: remove duplicated etag decryption (#7174) 2019-01-30 19:33:31 -08:00
Minio Trusted
9dc9f03c02 Update yaml files to latest version RELEASE.2019-01-31T00-31-19Z 2019-01-31 00:37:43 +00:00
Krishna Srinivas
b18c0478e7 Only heal on disks where we are sure that healing is needed (#7148) 2019-01-30 10:53:57 -08:00
Anis Elleuch
2d9860e875 heal: Fix healing empty directories (#7154)
This commit fixes the computation of Before/After healing state
for empty directories.

Issues before the commit:
- Before state doesn't reflect the real status (no StatVol() called)
- For any MakeVol() error, healObjectDir is exited directly, which is
  wrong.
2019-01-30 10:51:56 -08:00
kannappanr
d3553f8dfc Bucket Heal: Do not add empty endpoint entry (#7172)
Currently during a heal of a bucket, if one disk is offline an empty endpoint entry is added.
Then another entry with the missing endpoint is also added.

This results in more entries than disks being added.

Code that adds empty endpoint has been removed.
2019-01-30 10:40:43 -08:00
Harshavardhana
e1ae90c12b Make sure to pass the right username for correct ConditionValues (#7169)
Without passing proper username value would result in AccessDenied
errors when policies with `{aws:username}` substitutions are used.

Fixes #7165
2019-01-30 14:21:09 +05:30
Sidhartha Mani
34e7259f95 Add Historic CPU and memory stats (#7136)
Collect historic cpu and mem stats.  Also, use actual values 
instead of formatted strings while returning to the client. The string 
formatting prevents values from being processed by the server or 
by the client without parsing it. 

This change will allow the values to be processed (eg. 
compute rolling-average over the lifetime of the minio server)
and offloads the formatting to the client.
2019-01-30 12:47:32 +05:30
poornas
d0015b4d66 update kms docs example to set a longer period for token renewal (#7149) 2019-01-29 08:04:07 -08:00
poornas
3467460456 Fix vault client to autorenew or reauthenticate (#7161)
Switch to Vault API's Renewer for token renewal.If
token can no longer be renewed, reauthenticate to
get a fresh token.
2019-01-29 16:57:23 +05:30
Harshavardhana
64b5701971 Support AWS envs creds for non-aws endpoints in S3 gateway (#7156)
We made a change previously in #7111 which moved support
for AWS envs only for AWS S3 endpoint. Some users requested
that this be added back to Non-AWS endpoints as well as
they require separate credentials for backend authentication
from security point of view.
2019-01-29 16:05:20 +05:30
Praveen raj Mani
fad59da29d clientID removed in the MQTT config (#7157)
More than one client can't use the same clientID for MQTT connection. 
This causes problem in distributed deployments where config is shared 
across nodes, as each Minio instance tries to connect to MQTT using the
same clientID.

This commit removes the clientID field in config, and allows
MQTT client to create random clientID for each node.
2019-01-29 15:00:15 +05:30
Aditya Manthramurthy
91c839ad28 Use a buffer to collect SQL Select result rows (#7158)
Batching records into a single SQL Select message in the response
leads to significant speed up as the message header overhead is made
negligible.

This change leads to a speed up of 3-5x for queries that select many
small records.
2019-01-28 20:00:18 -08:00
Aditya Manthramurthy
2786055df4 Add new SQL parser to support S3 Select syntax (#7102)
- New parser written from scratch, allows easier and complete parsing
  of the full S3 Select SQL syntax. Parser definition is directly
  provided by the AST defined for the SQL grammar.

- Bring support to parse and interpret SQL involving JSON path
  expressions; evaluation of JSON path expressions will be
  subsequently added.

- Bring automatic type inference and conversion for untyped
  values (e.g. CSV data).
2019-01-28 17:59:48 -08:00
Harshavardhana
0a28c28a8c Avoid code which looks at local files when etcd is configured (#7144)
This situation happens only in gateway nas which supports
etcd based `config.json` to support all FS mode features.

The issue was we would try to migrate something which doesn't
exist when etcd is configured which leads to inconsistent
server configs in memory.

This PR fixes this situation by properly loading config after
initialization, avoiding backend disk config migration to be
done only if etcd is not configured.
2019-01-28 13:31:35 -08:00
Harshavardhana
526546d588 Remove '.minio.sys/tmp' files in background (#7124)
If it does happen that we have a lot files in '.minio.sys/tmp',
minio startup might block deleting this folder. Rename and
delete in background instead to allow Minio to start serving
requests.
2019-01-25 13:33:28 -08:00
Aditya Manthramurthy
2053b3414f Reduce heal parallelism (#7155)
To avoid a large number of concurrent connections between minio
servers and to reduce CPU pressure, it is better to limit the number
of objects healed in parallel to number_of_CPUs.
2019-01-25 13:11:17 -08:00
kannappanr
ce870466ff Top Locks command implementation (#7052)
API to list locks used in distributed XL mode
2019-01-24 07:22:14 -08:00
Harshavardhana
964e354d06 Fix liveness check for NAS gateway (#7142)
Current master throws '503' unavailable for liveness check
```
~ curl -v http://localhost:9000/minio/health/live
> GET /minio/health/live HTTP/1.1
...
...
< HTTP/1.1 503 Service Unavailable
```

With this fix liveness check returns error appropriately
```
~ curl -v http://localhost:9000/minio/health/live
> GET /minio/health/live HTTP/1.1
...
...
< HTTP/1.1 200 OK
```
2019-01-24 19:14:05 +05:30
kannappanr
8ee8ad777c logger: do not interpret encoded url as format string (#7110)
Error logger currently interprets encoded url in the error as a format string.
2019-01-24 00:30:00 -08:00
Krishna Srinivas
82af0be1aa Healing process should not heal root disk (#7089) 2019-01-23 15:29:29 -08:00
Minio Trusted
e8c18bc145 Update yaml files to latest version RELEASE.2019-01-23T23-18-58Z 2019-01-23 23:25:02 +00:00
Harshavardhana
bd25f31100 Use IAM creds only if endpoint is S3 (#7111)
Requirements like being able to run minio gateway in ec2
pointing to a Minio deployment wouldn't work properly
because IAM creds take precendence on ec2.

Add checks such that we only enable AWS specific features
if our backend URL points to actual AWS S3 not S3 compatible
endpoints.
2019-01-23 11:12:33 -08:00
Harshavardhana
ee7dcc2903 Handle errs returned with etcd properly for config init and migration (#7134)
Returning unexpected errors can cause problems for config handling,
which is what led gateway deployments with etcd to misbehave and
had stopped working properly
2019-01-23 11:10:59 -08:00
Harshavardhana
55ef51a99d Vendorize all recent changes to minio-go (#7135)
- Default support for S3 dualstack endpoints (IPv6 support)
- Support granular policy conditionals in List operations
- Support proxy cookies for stickiness
2019-01-23 19:22:09 +05:30
Anis Elleuch
dc2348daa5 heal: Preserve deployment ID from reference format.json (#7126)
Deployment ID is not copied into new formats after healing format. Although,
this is not critical since a new deployment ID will be generated and set in the
next cluster restart, it is still much better if we don't change the deployment
id of a cluster for a better tracking.
2019-01-22 18:32:06 -08:00
Aditya Manthramurthy
042d7f25e4 Fix regexp matcher of browser assets and paths (#7083)
Fix regexp matcher for special assets for the browser to clash with
less of the object namespace.

Assets should now be loaded with the /minio/ prefix. Previously,
favicon.ico (and others) could be loaded at any path matching
/minio/*/favicon.ico. This clashes with a large part of the object
namespace. With this change, /minio/favicon.ico will serve the favicon
but not /minio/mybucket/favicon.ico

Fixes #7077
2019-01-22 10:58:28 -08:00
Andreas Auernhammer
8c1b649b2d load system CAs before trying to load custom CAs (#7133)
This changes causes `getRootCAs` to always load system-wide CAs.
Any additional custom CAs (at `certs/CA/`) are added to the certificate pool
of system CAs.

The previous behavior was incorrect since all no system-wide CAs were
loaded if either there were CAs under `certs/CA` or the `certs/CA`
directory didn't exist at all.
2019-01-22 09:18:06 -08:00
Kumar Sukhani
f03ccec912 Fix slack Link (#7131) 2019-01-22 19:53:50 +05:30
Nitish Tiwari
0bb65f84bb Add example for IPv6 for address flag (#7127) 2019-01-22 15:55:27 +05:30
Harshavardhana
8e0910ab3e Fix build issues on BSDs in pkg/cpu (#7116)
Also add a cross compile script to test always cross
compilation for some well known platforms and architectures
, we support out of box compilation of these platforms even
if we don't make an official release build.

This script is to avoid regressions in this area when we
add platform dependent code.
2019-01-22 09:27:23 +05:30
Harshavardhana
5353edcc38 Support policy variable replacement (#7085)
This PR supports iam and bucket policies to have
policy variable replacements in resource and
condition key values.

For example
- ${aws:username}
- ${aws:userid}
2019-01-21 10:27:14 +05:30
Harshavardhana
3265112d04 Remove gateway implementations for manta, sia and b2 (#7115) 2019-01-20 08:10:58 -08:00
Harshavardhana
4fdacb8b14 Add policy conditions support for Listing operations on browser (#7106)
Fixes https://github.com/minio/minio/issues/7095
2019-01-20 12:50:01 +05:30
Krishna Srinivas
267f183fc8 Do not do StorageInfo() and ListBuckets() for FS/Erasure in health check handler (#7090)
Health checking programs very frequently use /minio/health/live 
to check health, hence we can avoid doing StorageInfo() and 
ListBuckets() for FS/Erasure backend.
2019-01-20 10:28:36 +05:30
Harshavardhana
3d22a9d84f Support rootCAs for notification targets (#7108)
Add support for RootCAs for notification targets
mqtt and webhook
2019-01-20 09:57:18 +05:30
Krishna Srinivas
51ec61ee94 Fix healing whole file bitrot (#7123)
* Use 0-byte file for bitrot verification of whole-file-bitrot files

Also pass the right checksum information for bitrot verification

* Copy xlMeta info from latest meta except []checksums and []Parts while healing
2019-01-20 07:58:40 +05:30
Harshavardhana
74c2048ea9 Add proper contexts with timeouts for etcd operations (#7097)
This fixes an issue of perceived hang when incorrect
unreachable URLs are specified in MINIO_ETCD_ENDPOINTS
variable.

Fixes #7096
2019-01-18 09:36:45 -08:00
Krishna Srinivas
730ac5381c Simplify parallelReader.Read() (#7109)
Simplify parallelReader.Read() which also fixes previous 
implementation where it was returning before all the parallel 
reading go-routines had terminated which caused race conditions.
2019-01-18 21:18:24 +05:30
Alex Simenduev
6dd8a83c5a change credential chain order in s3 gateway to mimic official docs (#7091) 2019-01-17 10:31:51 -08:00
Harshavardhana
1a7e6d4768 Handle multiple conditions in policies (#7079)
Fixes #7078
2019-01-17 10:28:24 -08:00
Krishna Srinivas
98c950aacd Streaming bitrot verification support (#7004) 2019-01-17 18:28:18 +05:30
Minio Trusted
94c52e3816 Update yaml files to latest version RELEASE.2019-01-16T21-44-08Z 2019-01-16 21:51:40 +00:00
Harshavardhana
8766c5eb22 Add version as part of Server: header (#7100)
This was agreed after discussing with @abperiasamy, we
borrowed the idea from Apache's own documentation.
2019-01-16 13:38:41 -08:00
kannappanr
e0d22359e7 Fix lint warnings (#7099) 2019-01-16 12:49:20 -08:00
Harshavardhana
6dd13e68c2 Support V2 signatures when autoencryption is enabled (#7084)
When auto-encryption is turned on, we pro-actively add SSEHeader
for all PUT, POST operations. This is unusual for V2 signature
calculation because V2 signature doesn't have a pre-defined set
of signed headers in the request like V4 signature. According to
V2 we should canonicalize all incoming supported HTTP headers.

Make sure to validate signatures before we mutate http headers
2019-01-16 12:12:06 -08:00
Harshavardhana
633001c8ba Inherit certsDir from configDir if latter is set (#7098)
This is to ensure backward compatibility for all existing
deployments which use custom config dir to point to their
certs directory.
2019-01-16 12:04:32 -08:00
Bala FA
e23a42305c Rebase minio/parquet-go and fix null handling. (#7067) 2019-01-16 21:52:04 +05:30
Krishna Srinivas
63d2583e91 Avoid holding write lock on config in situations where it is not needed (#7082)
This is to allow the cluster to come up when N/2 number of disks is available.
2019-01-16 13:59:21 +05:30
Harshavardhana
a2f66abbe8 Update STS API docs with Version query param (#7071) 2019-01-16 09:38:32 +05:30
Andreas Auernhammer
b28661b673 doc: add security documentation to provide some background info (#7028)
This commit adds some documentation about the design of the
SSE-C and SSE-S3 implementation. It describes how the Minio server
encrypt objects and manages keys.
2019-01-15 14:27:57 -08:00
Harshavardhana
e8791ae274 Remove Minio server arch, version from Server: header (#7074) 2019-01-15 13:16:11 +05:30
Scott Dunlop
309975d477 Add missing time import to counter_darwin.go (#7081) 2019-01-14 17:21:27 -08:00
Praveen raj Mani
6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
Harshavardhana
8757c963ba Migrate all Peer communication to common Notification subsystem (#7031)
Deprecate the use of Admin Peers concept and migrate all peer
communication to Notification subsystem. This finally allows
for a common subsystem for all peer notification in case of
distributed server deployments.
2019-01-14 12:14:20 +05:30
Praveen raj Mani
9a71f2fdfa link to ppc64le binary added (#7065)
Fixes #7063
2019-01-11 20:16:19 +05:30
Nick Craig-Wood
9c26fe47b0 Fix server side copy of files with ? in - fixes #7058 (#7059)
Before this change the CopyObjectHandler and the CopyObjectPartHandler
both looked for a `versionId` parameter on the `X-Amz-Copy-Source` URL
for the version of the object to be copied on the URL unescaped version
of the header.  This meant that files that had question marks in were
truncated after the question mark so that files with `?` in their
names could not be server side copied.

After this change the URL unescaping is done during the parsing of the
`versionId` parameter which fixes the problem.

This change also introduces the same logic for the
`X-Amz-Copy-Source-Version-Id` header field which was previously
ignored, namely returning an error if it is present and not `null`
since minio does not currently support versions.

S3 Docs:
- https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
2019-01-10 13:10:10 -08:00
Sidhartha Mani
f3f47d8cd3 Add ServerCPULoadInfo() and ServerMemUsageInfo() admin API (#7038) 2019-01-09 19:04:19 -08:00
Minio Trusted
de1d39e436 Update yaml files to latest version RELEASE.2019-01-10T00-21-20Z 2019-01-10 00:28:50 +00:00
poornas
ed1275a063 Fix copy from encrypted multipart to single encrypted part (#7056)
When source is encrypted multipart object and the parts are not
evenly divisible by DARE package block size, target encrypted size
will not necessarily be the same as encrypted source object.
2019-01-09 15:17:21 -08:00
kannappanr
a7d407fa42 Display message on failure to get lock on format.json in fs mode on startup (#6538)
Retry to see if the lock is free. Retry time will increase binomially.
2019-01-09 10:13:04 -08:00
Anis Elleuch
4e6e05f8e0 virtual host: Fix making new buckets (#7054)
This commit removes old code preventing PUT requests with '/' as a path,
because this is not needed anymore after the introduction of the virtual
host style in Minio server code.

'PUT /' when global domain is not configured already returns 405 Method
Not Allowed http error.
2019-01-09 11:59:41 +05:30
Bala FA
b0deea27df Refactor s3select to support parquet. (#7023)
Also handle pretty formatted JSON documents.
2019-01-08 16:53:04 -08:00
Kaan Kabalak
e98d89274f Upgrade to Webpack 4 (#7045) 2019-01-08 11:04:59 -08:00
kannappanr
c59206bcd3 GCS ListMultipartUploads: Don't return on first uploadid (#7014)
ListMultipartUploads code returns only the first uploadid.

Fixes #7011
2019-01-08 11:03:28 -08:00
Harshavardhana
7f2d439baa Avoid printing in S3 tests (#7043) 2019-01-07 22:32:30 +05:30
poornas
5a80cbec2a Add double encryption at S3 gateway. (#6423)
This PR adds pass-through, single encryption at gateway and double
encryption support (gateway encryption with pass through of SSE
headers to backend).

If KMS is set up (either with Vault as KMS or using
MINIO_SSE_MASTER_KEY),gateway will automatically perform
single encryption. If MINIO_GATEWAY_SSE is set up in addition to
Vault KMS, double encryption is performed.When neither KMS nor
MINIO_GATEWAY_SSE is set, do a pass through to backend.

When double encryption is specified, MINIO_GATEWAY_SSE can be set to
"C" for SSE-C encryption at gateway and backend, "S3" for SSE-S3
encryption at gateway/backend or both to support more than one option.

Fixes #6323, #6696
2019-01-05 14:16:42 -08:00
Harshavardhana
2d19011a1d Add support for AssumeRoleWithWebIdentity (#6985) 2019-01-04 13:48:12 -08:00
Harshavardhana
e82dcd195c Deprecate config-dir bring in certs-dir for TLS configuration (#7033)
This PR is to provide indication that config-dir will be removed
in future and all users should migrate to new --certs-dir option

Fixes #7016
Fixes #7032
2019-01-02 10:05:16 -08:00
Nitish Tiwari
fcb56d864c Add ServerDrivesPerfInfo() admin API (#6969)
This is part of implementation for mc admin health command. The
ServerDrivesPerfInfo() admin API returns read and write speed
information for all the drives (local and remote) in a given Minio
server deployment.

Part of minio/mc#2606
2018-12-31 09:46:44 -08:00
Krishnan Parthasarathi
75cd4201b0 Update go-sql-driver/mysql package (#7019) 2018-12-29 21:59:03 +05:30
Harshavardhana
f24c017e9a Move docker edge to latest Go as well (#7030) 2018-12-28 17:24:24 -08:00
Harshavardhana
b5280ba243 Migrate to Go version 1.11.4 (#7026) 2018-12-28 14:04:39 -08:00
Harshavardhana
2a0e4b6f58 Add boolean function condition support (#7027) 2018-12-28 12:18:58 -08:00
Minio Trusted
1898961ce3 Update yaml files to latest version RELEASE.2018-12-27T18-33-08Z 2018-12-27 18:41:05 +00:00
Krishnan Parthasarathi
236796ebd6 Add etcd as prerequisite for multi-user in gateway (#7022) 2018-12-27 07:22:18 +05:30
Harshavardhana
4e4f855b30 Add support for new policy conditions (#7024)
This PR implements following condition types

- StringEqualsIgnoreCase and StringNotEqualsIgnoreCase
- BinaryEquals
2018-12-26 17:39:30 -08:00
Harshavardhana
2db22deb93 Fix policy bugs Null conditions and canonical names (#7021)
This PR fixes two different issues

- Null condition implementation
- HTTP Canonical request value names

This PR fixes handling of null conditions and
handle HTTP canonical names in request values.

This PR was tested with policies mentioned in the following blog
https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/

Fixes #6955
2018-12-26 02:03:28 -08:00
Harshavardhana
fb8d0d7cf7 Add support for hostname lookups instead of IPs in MINIO_PUBLIC_IPS (#7018)
DNS names will be resolved to their respective IPs if specified
in MINIO_PUBLIC_IPS.

Fixes #6862
2018-12-23 03:08:21 -08:00
Harshavardhana
a536cf5dc0 Buffconn should buffer upto maxHeaderBytes to avoid ErrBufferFull (#7017)
It can happen with erroneous clients which do not send `Host:`
header until 4k worth of header bytes have been read. This can lead
to Peek() method of bufio to fail with ErrBufferFull.

To avoid this we should make sure that Peek buffer is as large as
our maxHeaderBytes count.
2018-12-23 12:03:04 +05:30
Harshavardhana
b9b68e9331 Add multi-stage build of docker edge image (#7005)
This is to reduce the overall size of the image,
we only retain the binary that was built in previous stage.
2018-12-22 06:36:48 +05:30
Anis Elleuch
632022971b s3: Don't set NextMarker when listing is not truncated (#7012)
Setting NextMarker when IsTruncated is not set seems to be confusing
AWS C++ SDK, this commit will avoid setting any string in NextMarker.
2018-12-20 13:30:25 -08:00
Harshavardhana
def04f01cf Update reedsolomon/highwayhash to start using ppc64le support (#7003)
Thanks to @fwessels for the upstream work on reedsolomon and
highwayhash which has resulted in 10x performance improvement
on ppc64 architecture.
2018-12-20 23:17:05 +05:30
Minio Trusted
bc67410548 Update yaml files to latest version RELEASE.2018-12-19T23-46-24Z 2018-12-19 23:54:02 +00:00
kannappanr
7881791a91 CopyObject:Set Content-Type to application/octet-stream if it is not set (#6958) 2018-12-19 14:31:45 -08:00
Harshavardhana
d2f8f8c7ee Fix ETag handling with auto-encryption with CopyObject conditions (#7000)
minio-java tests were failing under multiple places when
auto encryption was turned on, handle all the cases properly

This PR fixes

 - CopyObject should decrypt ETag before it does if-match
 - CopyObject should not try to preserve metadata of source
   when rotating keys, unless explicitly asked by the user.
 - We should not try to decrypt Compressed object etag, the
   potential case was if user sets encryption headers along
   with compression enabled.
2018-12-19 14:12:53 -08:00
kannappanr
8c32311b80 Change lock name to include names instead of hash. (#6886)
Previously lockname included the hash of the bucket, object
and uploadid.

This is a part of fix required to list oldest locks on the server.
2018-12-19 13:57:51 -08:00
Ashish Kumar Sinha
9bb88e610e Deletion of subfolders of multipart (#6961)
Delete subfolders under multipart folder upon completion of CompleteMultipartUpload, AbortMultipartUpload and cleanupStaleMultipartUploads functions
2018-12-19 11:27:10 -08:00
Harshavardhana
d1e41695fe Add support for federation on browser (#6891) 2018-12-19 18:43:47 +05:30
Aditya Manthramurthy
2aeb3fbe86 Fix csv output delimiter bug (#6994) 2018-12-19 11:49:06 +05:30
Anis Elleuch
99b843a64e Add anonymous flag to prevent logging sensitive information (#6899) 2018-12-18 16:08:11 -08:00
Harshavardhana
4f31a9a33b Reload users upon AddUser on peers (#6975)
Also migrate ReloadFormat to notification subsystem,
remove GetConfig() we do not use this API anymore
2018-12-18 14:39:21 -08:00
Nitish Tiwari
65ddff8899 Fix NAS Gateway Docker command example (#6967)
Fixes #6965
2018-12-18 14:37:17 -08:00
Harshavardhana
e7c902bbbc Return proper errors when admin API is not initialized (#6988)
Especially in gateway IAM admin APIs are not enabled
if etcd is not enabled, we should enable admin API though
but only enable IAM and Config APIs with etcd configured.
2018-12-18 13:03:26 -08:00
Andreas Auernhammer
5a5895203b add howto generate a master key and add master key disclaimer (#6992)
This commit adds a section to the master key documentation
describing how to generate a random 256 bit master key.

Further this commit adds a warning that master keys are not
recommended for production systems because it's (currently)
not possible to replace a master key (e.g. in case of compromise).
2018-12-18 13:00:32 -08:00
poornas
7da0336ac8 Make sure env are loaded before gateway layer initialization (#6989) 2018-12-18 10:42:09 -08:00
Harshavardhana
3be616de3f Send deployment ID in notification event response elements (#6991) 2018-12-18 10:05:26 -08:00
Harshavardhana
c5bf22fd90 Turn off printing IPv6 endpoints when listening on all interfaces (#6986)
By default when we listen on all interfaces, we print all the
endpoints that at local to all interfaces including IPv6
addresses. Remove IPv6 addresses in endpoint list to be
printed in endpoints unless explicitly specified with '--address'
2018-12-18 21:56:30 +05:30
poornas
7c9f934875 Disallow SSE requests when object layer has encryption disabled (#6981) 2018-12-14 21:39:59 -08:00
Eco
b6f9b24b30 Small corrections and example for auto-encryption (#6982) 2018-12-14 16:21:41 -08:00
poornas
13cb814a0e update KMS README.md to set approle env (#6978) 2018-12-14 14:03:16 -08:00
Andreas Auernhammer
d264d2c899 add auto-encryption feature (#6523)
This commit adds an auto-encryption feature which allows
the Minio operator to ensure that uploaded objects are
always encrypted.

This change adds the `autoEncryption` configuration option
as part of the KMS conifguration and the ENV. variable
`MINIO_SSE_AUTO_ENCRYPTION:{on,off}`.

It also updates the KMS documentation according to the
changes.

Fixes #6502
2018-12-14 13:35:48 -08:00
Harshavardhana
bebaff269c Support IPv6 in minio command line (#6947)
Fixes #6946
2018-12-14 13:07:46 +05:30
Harshavardhana
52b159b1db Allow versionId to be null for Delete,CopyObjectPart (#6972) 2018-12-14 11:34:37 +05:30
Nitish Tiwari
324834e4da Remove duplicate switch case (#6966)
Fixes #6948
2018-12-13 21:58:48 -08:00
Harshavardhana
c2ed1347d9 Do not list objects unless specified in policy (#6970)
Currently we use GetObject to check if we are allowed to list,
this might be a security problem since there are many users now
who actively disable a publicly readable listing, anyone who
can guess the browser URL can list the objects.

This PR turns off this behavior and provides a more expected way
based on the policies.

This PR also additionally improves the Download() object
implementation to use a more streamlined code.

These are precursor changes to facilitate federation and web
identity support in browser.
2018-12-14 09:45:09 +05:30
Anis Elleuch
50f6f9fe58 S3 api: Ignore encoding in xml body (#6953)
One user reported having discovered the following error:

API: SYSTEM()
Time: 20:06:17 UTC 12/06/2018
Error: xml: encoding "US-ASCII" declared but Decoder.CharsetReader is nil
1: cmd/handler-utils.go:43:cmd.parseLocationConstraint()
2: cmd/auth-handler.go:250:cmd.checkRequestAuthType()
3: cmd/bucket-handlers.go:411:cmd.objectAPIHandlers.PutBucketHandler()
4: cmd/api-router.go100cmd.(objectAPIHandlers).PutBucketHandler-fm()
5: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()

Hence, adding support of different xml encoding. Although there
is no clear specification about it, even setting "GARBAGE" as an xml
encoding won't change the behavior of AWS, hence the encoding seems
to be ignored.

This commit will follow that behavior and will ignore encoding field
and consider all xml as utf8 encoded.
2018-12-13 12:09:50 -08:00
Minio Trusted
48cb0ea34b Update yaml files to latest version RELEASE.2018-12-13T02-04-19Z 2018-12-13 02:09:53 +00:00
Harshavardhana
6f7c99a333 Allow versionId to be null for Copy,Get,Head API calls (#6942)
Fixes #6935
2018-12-12 11:43:44 -08:00
Harshavardhana
3498f5b0ec List exact DNS entries for a requested bucketName (#6936)
Currently we would end up considering common prefix
buckets to be part of the same DNS service record,
which leads to Minio server wrongly forwarding the
records to incorrect IPs.
2018-12-12 10:47:03 -08:00
Andreas Auernhammer
21d8c0fd13 refactor vault configuration and add master-key KMS (#6488)
This refactors the vault configuration by moving the
vault-related environment variables to `environment.go`
(Other ENV should follow in the future to have a central
place for adding / handling ENV instead of magic constants
and handling across different files)

Further this commit adds master-key SSE-S3 support.
The operator can specify a SSE-S3 master key using
`MINIO_SSE_MASTER_KEY` which will be used as master key
to derive and encrypt per-object keys for SSE-S3
requests.

This commit is also a pre-condition for SSE-S3
auto-encyption support.

Fixes #6329
2018-12-12 12:20:29 +05:30
Kale Blankenship
79b9a9ce46 Provide actual size in events instead of compressed size. (#6950)
Previous behavior did not check if the object was compressed and
incorrectly reported the stored size rather than the actual object
size.
2018-12-11 17:30:15 -08:00
Harshavardhana
b9b353db4b Add env to support synchronous ops for all calls (#6877) 2018-12-11 16:22:56 -08:00
poornas
11a9b317a3 Disable ListenBucket notifications for NAS gateway (#6954) 2018-12-11 16:16:09 -08:00
Praveen raj Mani
9af7d627ac Preserve the compression headers while copying (#6952)
Fixes #6951
2018-12-11 12:05:41 -08:00
Harshavardhana
76d9d54603 Filter listing buckets based on user level access (#6940)
Fixes #6701
2018-12-10 22:57:22 +05:30
Harshavardhana
4c7c571875 Support JSON to CSV and CSV to JSON output format conversion (#6910)
This PR implements one of the pending items in issue #6286
in S3 API a user can request CSV output for a JSON document
and a JSON output for a CSV document. This PR refactors
the code a little bit to bring this feature.
2018-12-07 14:55:32 -08:00
James Neiman, President
313ba74b09 Update to Minio GCS Gateway (#6887) 2018-12-06 10:09:37 -08:00
Harshavardhana
3e124315c8 Increase the keep alive timeout to 30 secs (#6924)
Go by default uses a 3 * minute, we should
atleast use 30 secs as 10 secs is too aggressive.
2018-12-06 22:56:16 +05:30
Minio Trusted
78a0fd951e Update yaml files to latest version RELEASE.2018-12-06T01-27-43Z 2018-12-06 01:35:43 +00:00
Anis Elleuch
40852801ea fix: Better check of RPC type requests (#6927)
guessIsRPCReq() considers all POST requests as RPC but doesn't
check if this is an object operation API or not, which is actually
confusing bucket forwarder handler when it receives a new multipart
upload API which is a POST http request.

Due to this bug, users having a federated setup are not able to
upload a multipart object using an endpoint which doesn't actually
contain the specified bucket that will store the object.

Hence this commit will fix the described issue.
2018-12-05 14:28:48 -08:00
poornas
f6980c4630 fix ConfigSys and NotificationSys initialization for NAS (#6920) 2018-12-05 14:03:42 -08:00
Harshavardhana
8fcc787cba Register notFound handler only once per root router (#6926)
registering notFound handler more than once causes
gorilla mux to return error for all registered paths
greater than > 8. This might be a bug in the gorilla/mux
but we shouldn't be using it this way. NotFound handler
should be only registered once per root router.

Fixes #6915
2018-12-05 11:54:12 -08:00
Praveen raj Mani
4e6d3c093f Errors in notification config should not crash the server (#6881)
Fixes #6870
2018-12-04 18:27:12 -08:00
Anis Elleuch
61145361fd Fallback to non-loopback IF addresses for Domain IPs (#6918)
When MINIO_PUBLIC_IPS is not specified and no endpoints are passed
as arguments, fallback to the address of non loop-back interfaces.

This is useful so users can avoid setting MINIO_PUBLIC_IPS in docker
or orchestration scripts, ince users naturally setup an internal
network that connects all instances.
2018-12-04 17:35:22 -08:00
James Neiman, President
950b4ad9af Update to How to secure access to Minio server with TLS (#6845) 2018-12-04 17:30:39 -08:00
Harshavardhana
6add646130 Fix logging of initialization errors in distributed mode (#6914) 2018-12-04 10:25:56 -08:00
Harshavardhana
20e61fb362 Redirect browser requests only if browser is enabled (#6909)
This PR fixes an issue introduced in PR #6848, when
browser is disabled we shouldn't re-direct the requests
returning AccessDenied.

Fixes #6907
2018-12-04 13:08:24 +05:30
Bala FA
18ced1102c handle post policy only if it is set. (#6852)
Previously policy in post form is assumed to be set always.  This is
fixed by doing the check when policy is set.
2018-12-03 12:01:28 -08:00
Harshavardhana
d6af3c1237 Add bucket notification support for NAS gateway (#6908)
Fixes #6885
2018-12-03 14:02:14 +05:30
Andreas Auernhammer
5549a44566 rename vault namespace env variable to be more idiomatic (#6905)
This commit renames the env variable for vault namespaces
such that it begins with `MINIO_SSE_`. This is the prefix
for all Minio SSE related env. variables (like KMS).
2018-12-01 05:28:49 -08:00
Praveen raj Mani
e7af31c2ff Removed clientID from NATS-Streaming Config (#6391)
clientID must be a unique `UUID` for each connections. Now, the
server generates it, rather considering the config.

Removing it as it is non-beneficial right now.

Fixes #6364
2018-11-30 10:46:17 +05:30
Minio Trusted
e7971b1d55 Update yaml files to latest version RELEASE.2018-11-30T03-56-59Z 2018-11-30 04:02:10 +00:00
Harshavardhana
26120d7838 Ignore permission errors on config-dir (#6894) 2018-11-29 18:14:05 -08:00
Harshavardhana
bef7c01c58 Choose right users in federation mode for CopyObject (#6895) 2018-11-29 17:35:11 -08:00
poornas
6a8ccc5925 update README.md (#6893) 2018-11-29 15:50:57 -08:00
kannappanr
d85199e9de Vendorize minio-go (#6883)
Fixes #6873
2018-11-29 11:13:03 -08:00
Harshavardhana
8608a84c23 Ignore hidden directory .snapshot for NetApp volumes (#6889) 2018-11-29 11:39:21 +05:30
Harshavardhana
b7226f4c82 Add transaction lock when migrating configs (#6878)
When migrating configs it happens often that some
servers fail to start due to version mismatch etc.

Hold a transaction lock such that all servers get
serialized.
2018-11-28 17:38:23 -08:00
Pontus Leitzler
f930ffe9e2 Fix gcs context (#6869) 2018-11-28 13:49:51 +05:30
James Neiman, President
b50a245208 Update to Minio Multi-Tenant Deployment Guide (#6871)
Initial edits.
2018-11-27 18:03:07 -08:00
poornas
45bb11e020 Set namespace on vault client if VAULT_NAMESPACE env is set (#6867) 2018-11-27 14:42:32 -08:00
jingsam
b65cf281fd Update azure.md (#6834) 2018-11-27 14:05:27 -08:00
Xie Yanbo
f781548b0c fix typo (#6812) 2018-11-27 14:04:50 -08:00
jingsam
25ee8e74f7 Update README.md (#6832) 2018-11-27 14:04:11 -08:00
jingsam
c975d2cc7e Update README.md (#6833) 2018-11-27 14:03:08 -08:00
jingsam
ea66528739 Update gcs.md (#6835) 2018-11-27 14:02:23 -08:00
Harshavardhana
e1164103d4 Reject if tokens are missing for temp credentials (#6860) 2018-11-27 13:24:04 -08:00
Harshavardhana
83fe70f710 Fix CopyObject regression calculating md5sum (#6868)
CopyObject() failed to calculate proper md5sum
when without encryption headers. This is a regression
fix perhaps introduced in commit 5f6d717b7a

Fixes https://github.com/minio/minio-go/issues/1044
2018-11-27 13:23:32 -08:00
Harshavardhana
12a6523fb2 Do not delete parts in multipart if 0 bytes (#6855)
This can create inconsistencies i.e Parts might have
lesser number of parts than ChecksumInfos. This will
result in object to be not readable.

This PR also allows for deleting previously created
corrupted objects.
2018-11-26 13:20:21 -08:00
Harshavardhana
dba61867e8 Redirect browser requests returning AccessDenied (#6848)
Anonymous requests from S3 resources returning
AccessDenied should be auto redirected to browser
for login.
2018-11-26 12:15:12 -08:00
Anis Elleuch
dd092f6c2b gateway: Properly set globalMinioPort (#6859)
globalMinioPort is used in federation which stores the address
and the port number of the server hosting the specified bucket,
this latter uses globalMinioPort but this latter is not set in
startup of the gateway mode.

This commit fixes the behavior.
2018-11-26 23:19:38 +05:30
Nitish Tiwari
2a810c7da2 Improve du thread performance (#6849) 2018-11-26 10:35:14 +05:30
Nitish Tiwari
dd8c2aa5c6 Cleanup Kubernetes documentation (#6861)
Also add details on why Readiness checks are not recommended for Minio
StatefulSets.
2018-11-25 13:34:20 -08:00
Harshavardhana
9e3fce441e Audit log claims from token (#6847) 2018-11-22 09:33:24 +05:30
Harshavardhana
d4265f9a13 Simplify OPA to use rootCAs custom transport (#6843)
Also close the connections properly to use the
connection pooling properly for HTTP clients.
2018-11-22 08:31:05 +05:30
Minio Trusted
2fc024e880 Update yaml files to latest version RELEASE.2018-11-22T02-51-56Z 2018-11-22 02:57:28 +00:00
Harshavardhana
eddf468aef Lock the targetList properly in go-routines (#6838)
Fixes #6483
2018-11-21 21:25:54 +05:30
Ashish Kumar Sinha
b0d04b9a81 Retry Connection for RabbitMQ (#6837)
Add retries to connect to RabbitMQ 5 times 
with 2s interval

Fixes #6807
2018-11-21 08:37:29 +05:30
Harshavardhana
a9de303d8b Update command line docs (#6839) 2018-11-20 17:35:33 -08:00
Anis Elleuch
69bd6df464 storage: Implement Close() in REST client (#6826)
Calling /minio/prometheuses/metrics calls xlSets.StorageInfo() which creates a new
storage REST client and closes it. However, currently, closing does nothing
to the underlying opened http client.

This commit introduces a closing behavior by calling CloseIdleConnections
provided by http.Transport upon the initialization of this latter.
2018-11-20 11:07:19 -08:00
Harshavardhana
bfb505aa8e Refactor logging in more Go idiomatic style (#6816)
This refactor brings a change which allows
targets to be added in a cleaner way and also
audit is now moved out.

This PR also simplifies logger dependency for auditing
2018-11-19 14:47:03 -08:00
poornas
d732b1ff9d Fix to cache objects on downloads (#6828)
fixes #6817
2018-11-19 11:00:46 -08:00
Igor K
ef517bd0da update for build on DilOS (#6770) 2018-11-19 19:40:02 +05:30
Minio Trusted
32d837cf88 Update yaml files to latest version RELEASE.2018-11-17T01-23-48Z 2018-11-17 01:30:29 +00:00
Anis Elleuch
7b579caf68 heal: Fix heal sequences cleanup process (#6780)
The current code triggers a timeout to cleanup a heal seq from
healSeqMap, but we don't know if the user did or not launch a new
healing sequence with the same path.

Add endTime to healSequence struct and add a periodic heal-sequence
cleaner to remove heal sequences only if this latter is older than
10 minutes.
2018-11-16 14:59:51 -08:00
Harshavardhana
272b8003d6 Honor header only when requested for use (#6815) 2018-11-16 10:27:48 -08:00
Anis Elleuch
1c24c93f73 storage: Upgrade REST version after adding WriteAll API (#6819)
Rolling update doesn't work properly because Storage REST API has
a new API WriteAll() but without API version number increase.

Also be sure to return 404 for unknown http paths.
2018-11-15 12:18:58 -08:00
Harshavardhana
712abc7958 Fix anonymous downloads URL generation (#6800)
Fixes #6778
2018-11-14 18:05:10 -08:00
poornas
5f6d717b7a Fix: Preserve MD5Sum for SSE encrypted objects (#6680)
To conform with AWS S3 Spec on ETag for SSE-S3 encrypted objects,
encrypt client sent MD5Sum and store it on backend as ETag.Extend
this behavior to SSE-C encrypted objects.
2018-11-14 17:36:41 -08:00
Harshavardhana
7e1661f4fa Performance improvements to SELECT API on certain query operations (#6752)
This improves the performance of certain queries dramatically,
such as 'count(*)' etc.

Without this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m42.464s
user	0m0.071s
sys	0m0.010s
```

With this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m17.603s
user	0m0.093s
sys	0m0.008s
```

Almost a 250% improvement in performance. This PR avoids a lot of type
conversions and instead relies on raw sequences of data and interprets
them lazily.

```
benchcmp old new
benchmark                        old ns/op       new ns/op       delta
BenchmarkSQLAggregate_100K-4     551213          259782          -52.87%
BenchmarkSQLAggregate_1M-4       6981901985      2432413729      -65.16%
BenchmarkSQLAggregate_2M-4       13511978488     4536903552      -66.42%
BenchmarkSQLAggregate_10M-4      68427084908     23266283336     -66.00%

benchmark                        old allocs     new allocs     delta
BenchmarkSQLAggregate_100K-4     2366           485            -79.50%
BenchmarkSQLAggregate_1M-4       47455492       21462860       -54.77%
BenchmarkSQLAggregate_2M-4       95163637       43110771       -54.70%
BenchmarkSQLAggregate_10M-4      476959550      216906510      -54.52%

benchmark                        old bytes       new bytes      delta
BenchmarkSQLAggregate_100K-4     1233079         1086024        -11.93%
BenchmarkSQLAggregate_1M-4       2607984120      557038536      -78.64%
BenchmarkSQLAggregate_2M-4       5254103616      1128149168     -78.53%
BenchmarkSQLAggregate_10M-4      26443524872     5722715992     -78.36%
```
2018-11-14 15:55:10 -08:00
Pontus Leitzler
f9779b24ad Enable default vet flags (#6810)
Enable default vet flags except experimental
2018-11-14 10:23:44 -08:00
Harshavardhana
f1f23f6f11 Add sync mode for 'xl.json' (#6798)
xl.json is the source of truth for all erasure
coded objects, without which we won't be able to
read the objects properly. This PR enables sync
mode for writing `xl.json` such all writes go hit
the disk and are persistent under situations such
as abrupt power failures on servers running Minio.
2018-11-14 19:48:35 +05:30
Harshavardhana
cf26c937e4 Remove UA worm and cache (#6809) 2018-11-14 13:18:55 +05:30
Krishna Srinivas
f19f957668 Remove unused repos from vendor.json (#6808) 2018-11-14 10:00:25 +05:30
Harshavardhana
d6572879a8 Check for STS Action first to allow browser requests (#6796) 2018-11-13 15:53:06 -08:00
Anis Elleuch
b6ab8f50fa azure: Support non standard Azure cloud environments (#6712)
This change will allow users to enter the endpoint of the
storage account if this latter belongs to a different Azure
cloud environment, such as US gov cloud.

e.g:

  `MINIO_ACCESS_KEY=testaccount \
     MINIO_SECRET_KEY=accountsecretkey \
     minio gateway azure https://testaccount.blob.usgovcloudapi.net`
2018-11-13 15:51:49 -08:00
Harshavardhana
c82acc599a Treat empty xl.json as file not found (#6804)
If the buffer is empty we can avoid parsing
it and treat it essentially as `xl.json`
is effectively missing.
2018-11-13 11:57:03 -08:00
Andreas Auernhammer
2447bb58dd change received system signal output to upper case (#6761)
This commit slightly improves the output of the minio server
in case of a received system signal.
2018-11-12 15:07:16 -08:00
Harshavardhana
a55a298e00 Make sure to log unhandled errors always (#6784)
In many situations, while testing we encounter
ErrInternalError, to reduce logging we have
removed logging from quite a few places which
is acceptable but when ErrInternalError occurs
we should have a facility to log the corresponding
error, this helps to debug Minio server.
2018-11-12 11:07:43 -08:00
Harshavardhana
2929c1832d Add sample STS request/response output (#6794) 2018-11-12 07:53:55 -08:00
Chester Li
aa2d8583ad Check key length before adding a new user. (#6790)
User's key should satisfy the requirement of `mc config host add`.
Check access key and secret key length before adding a new user,
avoid creating a useless user which cannot be added into config
host or log into the browser.
2018-11-09 15:48:24 -08:00
kannappanr
df2d75a2a3 Cleanup unnecessary logs (#6788) 2018-11-09 14:03:37 -08:00
Harshavardhana
b24b320807 Set notification namespace for NotificationConfiguration (#6789) 2018-11-09 10:40:03 -08:00
kannappanr
c872c1f1dc Return default ETag if fs.json is empty (#6787) 2018-11-09 10:34:59 -08:00
Harshavardhana
a40610d331 Re-populate public key if JWT fails to parse (#6786)
This is done such that if WSO2 was re-configured
with new TLS certs, and newer tokens are signed
with a newer public key. Once populated parse the JWT
again
2018-11-08 17:01:20 -08:00
Harshavardhana
38978eb2aa Avoid decrypting encrypted multipart final size (#6776)
Multipart object final size is not a contiguous
encrypted object representation, so trying to
decrypt this size will lead to an error in some
cases. The multipart object should be detected first
and then decoded with its respective parts instead.

This PR handles this situation properly, added a
test as well to detect these in the future.
2018-11-08 10:27:21 -08:00
Harshavardhana
ca7c3a3278 Add 'mc config host add' command in multi-user doc (#6777) 2018-11-08 09:42:47 -08:00
Harshavardhana
d58fc68137 Fix shadowing issue in elasticsearch target (#6774) 2018-11-07 12:09:03 -08:00
Matthias Schneider
71c66464c1 feature: added nsq as broker for events (#6740) 2018-11-07 10:23:13 -08:00
Harshavardhana
bf414068a3 Parse and return proper errors with x-amz-security-token (#6766)
This PR also simplifies the token and access key validation
across our signature handling.
2018-11-07 20:10:03 +05:30
Eco
88959ce600 Format correction in server limits doc (#6773) 2018-11-06 14:50:11 -08:00
Ashish Kumar Sinha
572719872d Event Notification for ElasticSearch (#6764)
Using access format for Event Notification for Elastic Search
2018-11-06 11:38:54 -08:00
Minio Trusted
bdea19b583 Update yaml files to latest version RELEASE.2018-11-06T01-01-02Z 2018-11-06 01:05:53 +00:00
poornas
eb1f9c9916 Update KMS readme with vault quick start guide (#6747) 2018-11-05 13:01:18 -08:00
Andreas Auernhammer
d07fb41fe8 add key-rotation for SSE-S3 objects (#6755)
This commit adds key-rotation for SSE-S3 objects.
To execute a key-rotation a SSE-S3 client must
 - specify the `X-Amz-Server-Side-Encryption: AES256` header
   for the destination
 - The source == destination for the COPY operation.

Fixes #6754
2018-11-05 10:26:10 -08:00
Harshavardhana
a9cda850ca Add forceStop flag to provide facility to stop healing (#6718)
This PR also makes sure that we deal with HTTP request
count by ignoring the on-going heal operation, i.e
do not wait on itself.
2018-11-04 19:24:16 -08:00
Harshavardhana
bef0318c36 Support audit logs with additional fields (#6738)
This PR adds support

- Request query params
- Request headers
- Response headers

AuditLogEntry is exported and versioned as well
starting with this PR.
2018-11-02 18:40:08 -07:00
Harshavardhana
3f19ea98bb Honor certs from config-dir (#6757)
This was a regression introduced in 9fe51e392b
2018-11-02 11:53:45 -07:00
Praveen raj Mani
c96073f985 Test for multiple values for x-amz-meta header added (#6753) 2018-11-02 11:32:18 -07:00
Harshavardhana
d2f240c791 Ignore windows hidden folders (#6735)
On Windows erasure coding setup if

```
~ minio server V:\ W:\ X:\ Z:\
```

is not possible due to NTFS creating couple of
hidden folders, this PR allows minio to use
the entire drive.
2018-11-02 11:31:55 -07:00
Harshavardhana
6491dfbbd6 Fix etcd TLS handling (#6748)
etcd fails to connect if TLS config is set, make TLS
conditional to input arguments instead
2018-11-01 21:41:11 -07:00
kannappanr
d9cfa5fcd3 Shouldn't require space in HTTP host header (#6743)
Fixes #6741
2018-10-31 15:31:42 -07:00
Aarushi Arya
89b14639a9 avoid using URL encoding to generate keys (#6731) 2018-10-31 15:07:20 -07:00
Harshavardhana
3f744c0361 Fix mimedb update files (#6744) 2018-10-31 14:15:27 -07:00
kannappanr
9ed7fb4916 Do not call multiple response.WriteHeader calls (#6733)
Execute method in s3Select package makes a response.WriteHeader call.

Not calling it again in SelectObjectContentHandler function in case of
error in s3Select.Execute call.
2018-10-31 14:09:26 -07:00
Eco
4280e68de3 Document not to use autogenerated keys with containers (#6737) 2018-10-31 12:09:16 +05:30
Harshavardhana
f162d7bd97 Performance improvements by re-using record buffer (#6622)
Avoid unnecessary pointer reference allocations
when not needed, for example

- *SelectFuncs{}
- *Row{}
2018-10-31 08:48:01 +05:30
Harshavardhana
36990aeafd Avoid double bucket validation in DeleteObjectHandler (#6720)
On a heavily loaded server, getBucketInfo() becomes slow,
one can easily observe deleting an object causes many
additional network calls.

This PR is to let the underlying call return the actual
error and write it back to the client.
2018-10-30 16:07:57 -07:00
Harshavardhana
9fe51e392b Support etcd TLS certficates (#6719)
This PR supports two models for etcd certs

- Client-to-server transport security with HTTPS
- Client-to-server authentication with HTTPS client certificates
2018-10-29 11:14:12 -07:00
Harshavardhana
7e879a45d5 Add policy claim support for JWT (#6660)
This way temporary credentials can use canned
policies on the server without configuring OPA.
2018-10-29 11:08:59 -07:00
poornas
1c911c5f40 Fix: Validate copy-part encryption header and metadata (#6725)
Otherwise CopyObjectPart would continue to upload part with incorrect
encryption option and fail when upload is finalized
2018-10-29 06:40:34 -07:00
poornas
bd8dc17b7a gateway s3:Make sure to convert s3 errors to ObjectLayer errors (#6717) 2018-10-28 22:11:20 -07:00
Harshavardhana
8491a29ec3 Fix healing bucket properly (#6716)
Bucket should be healed properly if it partially exists
on only one set, since bucket is common for all sets.

Fixes #6710
2018-10-28 14:13:17 -07:00
Pontus Leitzler
81d21850ec Root CAs can be used for backend without TLS (#6711) 2018-10-28 06:21:00 +05:30
kannappanr
c6ec3fdfba Return error response header only for HEAD method (#6709) 2018-10-26 18:03:17 -07:00
Anis Elleuch
88c3dd49c6 copy: Ensure that the user has GET access to the src object (#6715) 2018-10-26 16:12:44 -07:00
kannappanr
6869f6d9dd Remove unwanted logs (#6708) 2018-10-26 14:41:25 -07:00
Harshavardhana
3f643acb99 HealBucket was double counting endpoints (#6707)
Endpoint comparisons blindly without looking
if its local is wrong because the actual drive
for a local disk is always going to provide just
the path without the HTTP endpoint.

Add code such that this is taken care properly in
all situations. Without this PR HealBucket() would
wrongly conclude that the healing doesn't have quorum
when there are larger number of local disks involved.

Fixes #6703
2018-10-26 10:25:52 -07:00
Pontus Leitzler
ea73accefd Remove h2 from NextProtos since it doesn't work (#6705) 2018-10-26 12:48:39 +05:30
Harshavardhana
555d54371c Fix CopyObjectPart broken source encryption support (#6699)
Current master didn't support CopyObjectPart when source
was encrypted, this PR fixes this by allowing range
CopySource decryption at different sequence numbers.

Fixes #6698
2018-10-25 08:50:06 -07:00
Harshavardhana
bab4c90c45 Fix broken links in docs (#6700) 2018-10-25 11:39:31 +05:30
Minio Trusted
a2fc0b14d6 Update yaml files to latest version RELEASE.2018-10-25T01-27-03Z 2018-10-25 01:31:57 +00:00
Harshavardhana
bf66e9a529 Reload etcd users and policies properly (#6694)
Currently there was a bug in how we reload users and policies
which leads to users/policies going missing due to wrong path
construction.

Fixes #6693
2018-10-24 17:40:06 -07:00
Harshavardhana
fde8c38638 Add default canned policies (#6690) 2018-10-24 17:14:27 -07:00
Kaan Kabalak
e6252dee5a Fix links not working on Docs site (#6692)
The relative link paths that weren't working have been changed to
direct links to the corresponding Github pages.
2018-10-24 17:00:26 -07:00
Praveen raj Mani
ecb042aa1c Copy and CopyPart changes for compression (#6669)
This PR fixes

- The target object should be compressed even if the
  source object is not compressed.

- The actual size for an encrypted object should be the
  `decryptedSize`
2018-10-23 11:46:20 -07:00
Anis Elleuch
e29009d347 Register postgre driver in pkg/event/target (#6689)
Commit 5c13765168 removed postgre registration triggerd
by the automatic gofmt command but it was the only where pg is registered. This commit
fixes behavior and adds unit tests to check whether postgre & sql are registered or not.
2018-10-23 11:44:46 -07:00
Pontus Leitzler
9631d65552 Fix goroutine test fatalf (#6682)
Use t.Error/t.ErrorF instead if t.Fatal/t.Fatalf

Add returns to achieve same behaviour as earlier
2018-10-23 09:44:20 -07:00
Nitish Tiwari
7b7be66fa1 Cleanup Kubernetes documentation (#6678) 2018-10-23 18:22:43 +05:30
Harshavardhana
b99aaab42e Sid value can be any unicode character support it (#6676)
Fixes #6476
2018-10-23 16:11:06 +05:30
Nitish Tiwari
32bd1b31e9 Fix images for 8 node distributed deployment (#6685)
fixes #6633
2018-10-23 10:50:38 +05:30
Eco
f287b15e71 docs/minio-limits.md formatting (#6683)
Formatted docs to show missing "\" character, added "/" to list of unsupported chars and made note of the fact that list is not exhaustive.
2018-10-22 21:00:46 -07:00
Andreas Auernhammer
586466584f fix wrong actual part size assignment in CopyObjectPart (#6652)
This commit fixes a wrong assignment to `actualPartSize`.
The `actualPartSize` for an encrypted src object is not `srcInfo.Size`
because that's the encrypted object size which is larger than the
actual object size. So the actual part size for an encrypted
object is the decrypted size of `srcInfo.Size`.
2018-10-22 14:23:23 -07:00
Ashish Kumar Sinha
c0b4bf0a3e SQL select query for CSV/JSON (#6648)
select * , select column names have been implemented for CSV.
select * is implemented for JSON.
2018-10-22 12:12:22 -07:00
Praveen raj Mani
acf46cc3b5 SetConfigHandler should avoid setting an invalid notification config (#6679)
Fixes #6642 
Fixes #6641
2018-10-22 11:51:26 -07:00
Guido García
06ef8248c3 docs: add link to s3 gateway (#6666)
Minor change: Add a link to S3 gateway to make it easier to find that info.
2018-10-22 11:47:13 -07:00
Aarushi Arya
7c2ae4eaf7 Remove tmp file and multipart folder in FS mode. (#6677)
Fixes #6588
2018-10-22 07:36:30 -07:00
Harshavardhana
989d7af9ac Handle corrupted files correctly by fixing quorum (#6670)
This PR completes the missing functionality from #6592
2018-10-19 11:00:09 -07:00
Andreas Auernhammer
8a6c3aa3cd crypto: add RemoveInternalEntries function (#6616)
This commit adds a function for removing crypto-specific
internal entries from the object metadata.

See #6604
2018-10-19 10:50:52 -07:00
Harshavardhana
62b560510b Fix SSE-C source decryption handling (#6671)
Without this fix we have room for two different type of
errors.
- Source is encrypted and we didn't provide any source encryption keys

This results in Incomplete body error to be returned back to the client
since source is encrypted and we gave the reader as is to the object
layer which was of a decrypted value leading to "IncompleteBody"

- Source is not encrypted and we provided source encryption keys.

This results in a corrupted object on the destination which is
considered encrypted but cannot be read by the server and returns
the following error.

```
<Error><Code>XMinioObjectTampered</Code><Message>The requested object
was modified and may be compromised</Message><Resource>/id-platform-gamma/
</Resource><RequestId>155EDC3E86BFD4DA</RequestId><HostId>3L137</HostId>
</Error>
```
2018-10-19 10:41:13 -07:00
Harshavardhana
0edfb32621 Fix multi-user doc (#6662) 2018-10-19 12:35:44 +05:30
poornas
7e0f1eb8b5 Fix: verify client sent md5sum in encrypted PutObjectPart request (#6668)
This PR also removes check for SSE-S3 headers as this
is not required by S3 specification.
2018-10-18 16:05:05 -07:00
Pontus Leitzler
b43e8337b1 Add error handling in api-resource.go (#6651) 2018-10-18 07:31:46 -07:00
Harshavardhana
30040fba45 Add windows CI on travis (#6661) 2018-10-18 08:48:53 +05:30
Minio Trusted
44cf9ac62f Update yaml files to latest version RELEASE.2018-10-18T00-28-58Z 2018-10-18 00:34:26 +00:00
poornas
3b55357045 fix:Init globalIAMSys for ExecObjectLayerAPITest (#6636) 2018-10-17 17:25:50 -07:00
Harshavardhana
18d9a20ff6 Enable admin users API on gateway (#6659)
This is only enabled when etcd is enabled, healing is only
enabled for erasure coded backend.
2018-10-17 17:25:16 -07:00
Anis Elleuch
6590aba6d2 xl: PUT an empty dir on an existing prefix succeed (#6658)
This commit fixes a regression introduced in f187a16962
the regression returned AccessDenied when a client is trying to create an empty
directory on a existing prefix, though it should return 200 OK to be close as
much as possible to S3 specification.
2018-10-17 16:37:02 -07:00
Harshavardhana
2e81f27d27 Allow all browser calls to honor multi-users (#6645)
- GetAuth
- SetAuth
- GenerateAuth

Disallow changing bucket level metadata or creating/deleting buckets.
2018-10-17 16:23:09 -07:00
Anis Elleuch
ae3c05aa37 Avoid printing i/o closed pipe error message (#6654)
Since refactoring to GetObjectNInfo style, there are many cases
when i/o closed pipe is printed like, downloading an object
with wrong encryption key. This PR removes the log.
2018-10-17 15:52:18 -07:00
Praveen raj Mani
cef044178c Treat columns with spaces inbetween [s3Select] (#6597)
replace the double/single quotes with backticks for the xwb1989/sqlparser
to recognise such queries.

Fixes #6589
2018-10-17 11:01:26 -07:00
Pontus Leitzler
c998d1ac8c Add missing error check (#6632) 2018-10-17 10:57:12 -07:00
Eco
3457e504cf Spelling changes and fixed link (#6596) 2018-10-17 10:55:55 -07:00
Pontus Leitzler
7f0cca075a Update lint installation (#6650) 2018-10-17 21:59:04 +05:30
Wenjie
088c595e01 handle exception InvalidPart (#6649) 2018-10-17 21:50:58 +05:30
Harshavardhana
26b4b466df Fix a typo in multi-user doc (#6643) 2018-10-16 20:39:44 -07:00
Andreas Auernhammer
fdf691fdcc move SSE-C TLS enforcement into generic handler (#6639)
This commit moves the check that SSE-C requests
must be made over TLS into a generic HTTP handler.

Since the HTTP server uses custom TCP connection handling
it is not possible to use `http.Request.TLS` to check
for TLS connections. So using `globalIsSSL` is the only
option to detect whether the request is made over TLS.
By extracting this check into a separate handler it's possible
to refactor other parts of the SSE handling code further.
2018-10-16 19:22:09 -07:00
Harshavardhana
88c8c2d6cd Fix browser login with multi users (#6644) 2018-10-16 18:44:58 -07:00
Nitish Tiwari
ef585037a0 Update config documentation (#6634) 2018-10-16 16:45:04 -07:00
Anis Elleuch
362ebdcbab xl: Add '.CORRUPTED' prefix to corrupted objects (#6592)
getObjectInfo() checks if a given object has enough xl.json in disks,
returns 404 if not and added .CORRUPTED suffix to object.
2018-10-16 15:49:35 -07:00
Harshavardhana
b251454dd6 Fix toggling users status (#6640) 2018-10-16 14:55:23 -07:00
Harshavardhana
21c8693d9c Disable printing access/secrets in systemd (#6621)
Minio when run as a service in `systemd` should
avoid printing access/secret keys.
2018-10-16 13:19:12 -07:00
Harshavardhana
1e7e5e297c Add canned policy support (#6637)
This PR adds an additional API where we can create
a new set of canned policies which can be used with one
or many users.
2018-10-16 12:48:19 -07:00
kannappanr
c7f180ffa9 Add code to translate errInvalidEncryptionParameters to APIErrcode (#6625)
Fixes #6623
2018-10-16 12:27:34 -07:00
kannappanr
b8bd8d6a03 Validate user provided SSE-C key on Head Object API (#6600)
Fixes #6598
2018-10-16 12:24:27 -07:00
Andreas Auernhammer
baec331e84 crypto: add functions for sealing/unsealing the etag for SSE (#6618)
This commit adds two functions for sealing/unsealing the
etag (a.k.a. content MD5) in case of SSE single-part upload.

Sealing the ETag is neccessary in case of SSE-S3 to preserve
the security guarantees. In case of SSE-S3 AWS returns the
content-MD5 of the plaintext object as ETag. However, we
must not store the MD5 of the plaintext for encrypted objects.
Otherwise it becomes possible for an attacker to detect
equal/non-equal encrypted objects. Therefore we encrypt
the ETag before storing on the backend. But we only need
to encrypt the ETag (content-MD5) if the client send it -
otherwise the client cannot verify it anyway.
2018-10-16 10:02:19 -07:00
poornas
557f382477 cache: remove cache space constraint (#6635)
relax cache constraint of requiring 100 times size of object
being cached for better cache utilization.
2018-10-16 11:06:42 +05:30
Harshavardhana
23b166b318 Remove applying custom policies with STS access keys (#6626)
Move away from allowing custom policies, all policies in
STS come from OPA otherwise they fail.
2018-10-15 12:44:03 -07:00
Anis Elleuch
81a481e098 profiling: Fix downloading tracing profiling data (#6599)
pkg/pprof saves tracing profiling data in a different file name
(trace.out) in contrary to other profiling mode.
2018-10-15 11:13:19 -07:00
Anis Elleuch
5b3090dffc encryption: Fix copy from encrypted multipart to single part (#6604)
CopyObject handler forgot to remove multipart encryption flag in metadata
when source is an encrypted multipart object and the target is also encrypted
but single part object.

This PR also simplifies the code to facilitate review.
2018-10-15 11:07:36 -07:00
Harshavardhana
3ef3fefd54 Add ListUsers API to list all configured users in IAM (#6619) 2018-10-13 12:48:43 +05:30
Andreas Auernhammer
28e25eac78 crypto: add helper functions for unsealing object keys (#6609)
This commit adds 3 helper functions for SSE-C and SSE-S3
to simplify object key unsealing in server code.

See #6600
2018-10-12 18:06:38 -07:00
Harshavardhana
b0c9ae7490 Add audit logging for S3 and Web handlers (#6571)
This PR brings an additional logger implementation
called AuditLog which logs to http targets

The intention is to use AuditLog to log all incoming
requests, this is used as a mechanism by external log
collection entities for processing Minio requests.
2018-10-12 12:25:59 -07:00
Harshavardhana
143e7fe300 Add etcd support to support STS on gateway mode (#6531) 2018-10-12 11:32:18 -07:00
Andreas Auernhammer
f09e7ca764 fix travis CI build (#6620)
This commit fixes the Travis CI build by
correcting the golint import path
2018-10-11 14:58:44 -07:00
Mariska Hoogenboom
fae284d6b9 Docs fix for restart issue with orchestrated minio stack (#6606) (#6613) 2018-10-11 14:41:19 +05:30
poornas
83d8e01c81 fix: Close cacheReader if cache entry has expired (#6610)
prevent locking issues

Fixes #6602
2018-10-10 23:01:24 -07:00
poornas
110458cd10 Fix: Disallow requests with SSE-KMS headers (#6587)
Addresses issue #6582. Minio server currently does not
have SSE-KMS support. Reject requests with SSE-KMS headers
with NotImplementedErr
2018-10-09 15:04:53 -07:00
Aditya Manthramurthy
e3eec89d24 Optimize string processing in select (#6593)
Reduce allocations during string concatenation and simplify some
processing code.
2018-10-09 14:02:19 -07:00
Harshavardhana
54ae364def Introduce STS client grants API and OPA policy integration (#6168)
This PR introduces two new features

- AWS STS compatible STS API named AssumeRoleWithClientGrants

```
POST /?Action=AssumeRoleWithClientGrants&Token=<jwt>
```

This API endpoint returns temporary access credentials, access
tokens signature types supported by this API

  - RSA keys
  - ECDSA keys

Fetches the required public key from the JWKS endpoints, provides
them as rsa or ecdsa public keys.

- External policy engine support, in this case OPA policy engine

- Credentials are stored on disks
2018-10-09 14:00:01 -07:00
Aditya Manthramurthy
16a100b597 Fix out-of-bound array access crash in select processing (#6594)
Fix test case.
2018-10-09 09:45:56 -07:00
Anis Elleuch
cbc5d78a09 Handle read/quorum errors when initializing all subsystems (#6585)
- Only require len(disks)/2 to initialize the cluster
- Fix checking of read/write quorm in subsystems init
- Add retry mechanism in policy and notification to avoid aborting in case of read/write quorums errors
2018-10-08 15:47:13 -07:00
Minio Trusted
d8a2975a68 Update yaml files to latest version RELEASE.2018-10-06T00-15-16Z 2018-10-06 00:19:47 +00:00
Anis Elleuch
66d911653f xl: Fix typo in PutObjectPart when part size is 10Mb (#6574)
PutObjectPart forgot to allocate buffer memory when the size
of the uploaded part is exactly equal to blockSizeV1 = 10 Mb.
2018-10-05 17:09:50 -07:00
Anis Elleuch
ca6f795504 xl: check for correct err variable when prepareFile fails in PutObjectPart (#6573)
in xl.PutObjectPart call, prepareFile detected an error when the storage
is exhausted but we were returning the wrong error.

With this commit, users can see the correct error message when their disks
become full.
2018-10-05 17:09:20 -07:00
Eco
2af0f11731 Update readme.md (#6568) 2018-10-05 16:25:22 -07:00
Harshavardhana
c3408f4f04 Send correct bucket notifications from web handlers (#6572)
Upload, Download, DownloadZip were incomplete
2018-10-05 11:20:00 -07:00
Minio Trusted
b92c324254 Update yaml files to latest version RELEASE.2018-10-05T01-03-03Z 2018-10-05 01:08:39 +00:00
Krishna Srinivas
81bee93b8d Move remote disk StorageAPI abstraction from RPC to REST (#6464) 2018-10-04 17:44:06 -07:00
Ashish Kumar Sinha
670f9788e3 Count(*) to give integer value (#6564)
The Max, Min functions were giving float value even when they were integers.  
Resolved max and Min to return integers in that scenario.

Fixes #6472
2018-10-04 17:33:53 -07:00
Anis Elleuch
f187a16962 xl: Require full write quorum in rename() as general rule (#6535)
Simplify the logic of using rename() in xl. Currently, renaming
doesn't require the source object/dir to be existent in at least
read quorum disks, since there is no apparent reason for it
to be as a general rule, this commit will just simplify the
logic to avoid possible inconsistency in the backend in the future.
2018-10-04 17:22:49 -07:00
Anis Elleuch
e031f2b614 xl: Fix typo in PutObject when uploading a 10Mb file (#6567)
PutObject forgot to allocate memory when the size of the uploaded
file is exactly equal to blockSizeV1 = 10 Mb.
2018-10-04 15:37:15 -07:00
Harshavardhana
c2b7b82ef4 Verify bitrot in ReadFile correctly when verifier is set (#6563)
This is a major regression introduced in this commit

ce02ab613d is the first bad commit
commit ce02ab613d
Author: Krishna Srinivas <634494+krishnasrinivas@users.noreply.github.com>
Date:   Mon Aug 6 15:14:08 2018 -0700

    Simplify erasure code by separating bitrot from erasure code (#5959)

:040000 040000 794f58d82a d2201ebfc8 M	cmd

This effects all distributed server deployments since this commit

All the following releases are affected

- RELEASE.2018-09-25T21-34-43Z
- RELEASE.2018-09-12T18-49-56Z
- RELEASE.2018-09-11T01-39-21Z
- RELEASE.2018-09-01T00-38-25Z
- RELEASE.2018-08-25T01-56-38Z
- RELEASE.2018-08-21T00-37-20Z
- RELEASE.2018-08-18T03-49-57Z

Thanks to Anis for reproducing the issue
2018-10-04 10:16:38 -07:00
Harshavardhana
02ad9d6072 Avoid unsolicited response over idle http client (#6562)
Without this PR minio server is writing an erroneous
response to clients on an idle connections which ends
up printing following message

```
Unsolicited response received on idle HTTP channel
```

This PR would avoid sending responses on idle connections
.i.e routine network errors.
2018-10-04 10:13:12 -07:00
Anis Elleuch
ea9408ccbb worm: when enabled, avoid renaming the existing object in tmp directory (#6560)
In XL PutObject & CompleteMultipartUpload, the existing object is renamed
to the temporary directory before checking if worm is enabled or not.

Most of the times, this doesn't cause an issue unless two uploads to the
same location occurs at the same time. Since there is no locking in object
handlers, both uploads will reach XL layer. The second client acquiring
write lock in put object or complete upload in XL will rename the object
to the temporary directory before doing the check and returning the error (wrong!).

This commit fixes then the behavior: no rename to temporary directory if
worm is enabled.
2018-10-03 20:39:24 -07:00
Pontus Leitzler
307765591d Use GetObjectInfo instead of GetObjectNInfo before cache decision (#6553) 2018-10-03 11:02:32 -07:00
Harshavardhana
5d859b2178 gateway/azure: allow putObject to support block based upload (#6552)
Current implementation simply uses all the memory locally
and crashes when a large upload is initiated using Minio
browser UI.

This PR uploads stream in blocks and finally commits the blocks
allowing for low memory footprint while uploading large objects
through Minio browser UI.

This PR also adds ETag compatibility for single PUT operations.

Fixes #6542
Fixes #6550
2018-10-02 23:08:16 -07:00
Harshavardhana
223967fd32 Return always a default heal item upon unexpected error (#6556)
Never return an empty result item even upon error,
choose all the default values and based on the errors
make sure to send right result reply.
2018-10-02 17:13:51 -07:00
Pontus Leitzler
274b35154c Fix go1.11 vet shadow errors (#6555) 2018-10-02 15:21:34 -07:00
Harshavardhana
c05ced08bb Handle Range requests on empty objects (#6557)
Return a proper error on empty objects, S3 returns
416 Invalid Range on objects which are empty.

We should return the same.
2018-10-02 12:48:51 -07:00
Praveen raj Mani
c7722fbb1b Simplify pkg mimedb (#6549)
Content-Type resolution can now use a function `TypeByExtension(extension)` 
to resolve to the respective content-type.
2018-10-02 11:48:17 +05:30
Harshavardhana
f163bed40d Add Vault support for custom CAs directory (#6527) 2018-10-01 13:49:10 -07:00
Harshavardhana
b4772849f9 Heal recursively all entries in config/ prefix (#6545)
This to ensure that we heal all entries in config/
prefix, we will have IAM and STS related files which
are being introduced in #6168 PR

This is a change to ensure that we heal all of them
properly, not just `config.json`
2018-10-01 22:24:26 +05:30
Harshavardhana
839a758a36 Fix a crash due to race between Abort/CompleteMultipart (#6544)
Fixes #6429
2018-10-01 09:50:09 -07:00
Harshavardhana
aebfceeafb Heal backend configuration file (#6532)
Fixes #6461
2018-09-29 13:47:01 +05:30
Anis Elleuch
83d7ec09c1 Disable restarting server after setting a new config (#6521)
Also disable listening to service restart event in tests since
we don't do this anymore.
2018-09-28 12:10:51 -07:00
Harshavardhana
8c29f69b00 Fix racy error communication inside go-routine (#6539)
Use CloseWithError to communicate errors in pipe,
this PR also fixes potential shadowing of error
2018-09-28 13:14:59 +05:30
Praveen raj Mani
ce9d36d954 Add object compression support (#6292)
Add support for streaming (golang/LZ77/snappy) compression.
2018-09-28 09:06:17 +05:30
Anis Elleuch
5c765bc63e Fix one possible data race in admin tests (#6537)
go test shows the following warning:
```
WARNING: DATA RACE
Write at 0x000002909e18 by goroutine 276:
  github.com/minio/minio/cmd.testAdminCmdRunnerSignalService()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-rpc_test.go:44 +0x94

 Previous read at 0x000002909e18 by goroutine 194:
  github.com/minio/minio/cmd.testServiceSignalReceiver()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-handlers_test.go:467 +0x70
```

The reason for this data race is that some admin tests are not waiting for go routines
that they created to be properly exited, which triggers the race detector.
2018-09-27 17:16:30 -07:00
Anis Elleuch
6c7c6bec91 profiler: Download API returns error when all nodes fail (#6525)
When download profiling data API fails to gather profiling data
from all nodes for any reason (including profiler not enabled),
return 400 http code with the appropriate json message.
2018-09-27 10:34:37 -07:00
poornas
ed703c065d Add ObjectOptions to GetObjectNInfo (#6533) 2018-09-27 15:36:45 +05:30
Aditya Manthramurthy
387584356f Remove unused range parsing code and update tests (#6530) 2018-09-27 15:24:07 +05:30
Harshavardhana
1111419d4a Add debugging for mutex, tracing (#6522) 2018-09-27 09:32:05 +05:30
Anis Elleuch
20378821cf madmin: close http response when returning an error (#6526)
httpRespToErrorResponse() usually reads the http response when
the http error code is not expected to parse the json error
response in the http body, however it was never properly closing
the connection. This PR fixes the behavior.
2018-09-26 11:03:35 -07:00
Pontus Leitzler
ce1bfa6de8 Removed unused vendored dependencies (#6520) 2018-09-25 14:51:51 -07:00
Minio Trusted
6c26227081 Update yaml files to latest version RELEASE.2018-09-25T21-34-43Z 2018-09-25 21:39:03 +00:00
Anis Elleuch
aa4e2b1542 Use GetObjectNInfo in CopyObject and CopyObjectPart (#6489) 2018-09-25 12:39:46 -07:00
Harshavardhana
1e5ac39ff3 Return HTTP response error for malformed requests (#6517) 2018-09-25 11:07:26 -07:00
Harshavardhana
ec2295c3dc Quickly build dev docker images using 'make docker' (#6505)
This PR simplifies the process of developer build of local
docker containers using `make docker`.

You need to provide a TAG i.e

```
TAG=y4m4/minio:exp make docker
```
2018-09-25 10:33:25 -07:00
Andreas Auernhammer
8cf7b88cc5 add functions to remove confidential information (#6516)
This commit adds two functions for removing
confidential information - like SSE-C keys -
from HTTP headers / object metadata.

This creates a central point grouping all
headers/entries which must be filtered / removed.

See also https://github.com/minio/minio/pull/6489#discussion_r219797993
of #6489
2018-09-24 21:02:51 +05:30
Harshavardhana
48bfebe442 HEAD on an object should mimic GET without body (#6508)
Add "Range" header support etc.
2018-09-23 22:54:10 +05:30
Pontus Leitzler
df60b3c733 Remove unnecessary contexts passed as data to FatalIf. No need to log an empty context. (#6487) 2018-09-21 16:04:11 -07:00
Aditya Manthramurthy
584cb61bb8 Switch back to GetObjectInfo for HEAD requests (#6513) 2018-09-21 13:48:58 -07:00
kannappanr
c1798cc89a Update GCS storage library to support GetObject API on gzipped objects (#6506)
Fixes #6280
2018-09-21 11:43:10 -07:00
Aditya Manthramurthy
3c8fabd116 Fix cleanup of pipe in GetObjectNInfo handlers (#6509) 2018-09-21 11:42:06 -07:00
Aditya Manthramurthy
36e51d0cee Add GetObjectNInfo to object layer (#6449)
The new call combines GetObjectInfo and GetObject, and returns an
object with a ReadCloser interface.

Also adds a number of end-to-end encryption tests at the handler
level.
2018-09-20 19:22:09 -07:00
Harshavardhana
7d0645fb3a Deprecate domain, browser as config entries (#6498) 2018-09-20 14:56:32 -07:00
Kanagaraj M
7c339e248a fix success alert not shown after object is uploaded (#6500) 2018-09-20 22:54:09 +05:30
Harshavardhana
b62ed5dc90 select API CSV may not be specified (#6493)
This should be present until we support JSON
2018-09-20 15:04:26 +05:30
Harshavardhana
f0641a0406 Avoid changing creds in streaming signature (#6495)
This PR fixes a potential issue where credentials can be
changed in middle of different chunks during data transfer
which can lead to unexpected bugs.
2018-09-19 16:52:05 -07:00
Harshavardhana
3d060f8b64 Peer/rpc should never honor RPC calls without object-layer (#6486)
Fixes #6484
2018-09-19 21:32:56 +05:30
Jay Mundrawala
052a7b8eec Allow minio s3 gateway to use different AWS auth mechanisms (#6422)
Allow minio s3 gateway to use aws environment credentials,
IAM instance credentials, or AWS file credentials.

If AWS_ACCESS_KEY_ID, AWS_SECRET_ACCSES_KEY are set, 
or minio is running on an ec2 instance with IAM instance credentials, 
or there is a file $HOME/.aws/credentials, minio running as an S3
gateway will authenticate with AWS S3 using those one of credentials.

The lookup order:
1. AWS environment varaibles
2. IAM instance credentials
3. $HOME/.aws/credentials
4. minio environment variables

To authenticate with the minio gateway, you will always use the
minio environment variables MINIO_ACCESS_KEY MINIO_SECRET_KEY.
2018-09-19 18:05:30 +05:30
Anis Elleuch
9531cddb06 Add Profiler Admin API (#6463)
Two handlers are added to admin API to enable profiling and disable
profiling of a server in a standalone mode, or all nodes in the
distributed mode.

/minio/admin/profiling/start/{cpu,block,mem}:
  - Start profiling and return starting JSON results, e.g. one
    node is offline.

/minio/admin/profiling/download:
  - Stop the on-going profiling task
  - Stream a zip file which contains all profiling files that can
    be later inspected by go tool pprof
2018-09-18 16:46:35 -07:00
Harshavardhana
6fe9a613c0 Prioritize HTTP requests over Heal (#6468)
Additionally also heal 256 objects at any given
time in parallel.

Fixes #6196
Fixes #6241
2018-09-17 18:28:34 -07:00
Andreas Auernhammer
b729a4e83c Remove brittle TestServerTLSCiphers unit test (#5982)
The test TestServerTLSCiphers seems to fail sometimes for
no obvious reason. Actually the test is not needed
(as unit test) since minio/mint tests the server's TLS ciphers
as part of its security tests.

Fixes #5977
2018-09-17 15:55:09 +05:30
Harshavardhana
a0683d3c1f Send progress only when requested by client in SelectObject (#6467) 2018-09-17 11:52:46 +05:30
Anis Elleuch
66fda7a37f Use retry mechanism when initializing configuration (#6475)
Currently, one node in a cluster can fail to boot with the following error message:

```
ERROR Unable to initialize config system: Storage resources are insufficient for the write operation
```

  This happens when disks are formatted, read quorum is met but write
quorum is not met. In checkServerConfig(), a insufficient read quorum
error is replaced by errConfigNotFound, the code will generate a
new config json and try to save it, but it will fail because write
quorum is not met.

  Replacing read quorum with errConfigNotFound is also wrong because it
can lead, in rare cases, to overwrite the config set by the user.

  So, this commit adds a retry mechanism in configuration initialization
to retry only with read or write quorum errors.

  This commit will also fix the following cases:
 - Read quorum is lost just after the initialization of the object layer.
 - Write quorum not met when upgrading configuration version.
2018-09-15 22:09:51 -07:00
Harshavardhana
a63bc9254d Add 'disk' tag to log output to enhance 'disk not found' errors (#6460) 2018-09-13 21:42:50 -07:00
poornas
14fa0097b0 fix: UploadPart,CopyObjectPart does not need sse-s3 header (#6386)
S3 API spec for UploadPart requires encryption headers to be
specified only for SSE-C
2018-09-13 14:53:03 -07:00
Kanagaraj M
d99c397746 fix public directory listing not working in browser (#6466)
Fixes #6436
2018-09-13 14:43:50 -07:00
Harshavardhana
e3777b1dd9 Combine obtaining resource, host, method into one operation (#6465)
This also adds a reduced timeout for errant connections, to
be quickly closed if they can't even send HTTP headers properly.

Fixes #6459
2018-09-13 18:17:03 +05:30
Minio Trusted
63c03758e6 Update yaml files to latest version RELEASE.2018-09-12T18-49-56Z 2018-09-12 18:55:31 +00:00
Anis Elleuch
ce419c9835 Bump RPC version after ReadFile RPC arguments change (#6457)
ReadFile RPC input argument has been changed in commit a8f5939452959d27674560c6b803daa9,
however, RPC doesn't detect such a change when it calls other nodes with older versions.

Hence, bumping RPC version.

Fixes #6458
2018-09-12 10:45:28 -07:00
Harshavardhana
0c2b708484 for O_RDONLY mode hold shared locks on windows (#6454)
Fixes #6401
2018-09-12 09:29:41 -07:00
Harshavardhana
166e998788 Fix healthcheck for NAS gateway (#6452)
It was expected that in gateway mode, we do not know
the backend types whereas in NAS gateway since its
an extension of FS mode (standalone) this leads to
an issue in LivenessCheckHandler() which would perpetually
return 503, this would affect all kubernetes, openshift
deployments of NAS gateway.
2018-09-11 13:44:10 -07:00
Andreas Auernhammer
267a0a3dfa fix X-Amz-Credential parsing for V4 policy signature (#6451)
This commit fixes an AWS S3 incompatibility issue.
The AccessKeyID may contain one or more `/` which caused
the server to interpret parts of the AccessKeyID as
other `X-Amz-Credential` parameters (like date, region, ...)

This commit fixes this by allowing 5 or more
`X-Amz-Credential` parameter strings and only interpreting
the last 5.

Fixes #6443
2018-09-11 11:17:23 -07:00
Minio Trusted
985fd7d4e7 Update yaml files to latest version RELEASE.2018-09-11T01-39-21Z 2018-09-11 01:43:45 +00:00
Harshavardhana
5479a6e33e Ignore migration if config files don't exist (#6448) 2018-09-10 18:24:57 -07:00
ebozduman
fb4186f6b9 Adds missing info to docs for credentials and domain env. vars. (#6447)
* Adds missing information to documentation for credentials and domain environment variables for distributed minio server startup.
2018-09-10 17:14:40 -07:00
Anis Elleuch
7571582000 Print storage errors during distributed initialization (#6441)
This commit will print connection failures to other disks in other nodes
after 5 retries. It is useful for users to understand why the
distribued cluster fails to boot up.
2018-09-10 16:21:59 -07:00
Harshavardhana
12b4971b70 Rename config.json in config-dir with '.deprecated' extension (#6446)
Fixes #6444
2018-09-10 16:15:47 -07:00
poornas
5c0b98abf0 Add ObjectOptions to ObjectLayer calls (#6382) 2018-09-10 09:42:43 -07:00
Praveen raj Mani
30d4a2cf53 s3select should honour custom record delimiter (#6419)
Allow custom delimiters like `\r\n`, `a`, `\r` etc in input csv and 
replace with `\n`.

Fixes #6403
2018-09-10 21:50:28 +05:30
Anis Elleuch
92bc7caf7a Reword missing credentials error msg (#6418)
Enhance a little bit the error message that is showing
when access & secret keys are not specified in the
environment when running Minio in gateway and server mode.

This commit also removes a redundant check of access/secret keys.
2018-09-09 22:51:48 +05:30
Harshavardhana
19202bae81 Allow backward compatible way to load creds from config.json (#6435)
Print warning message for users to migrate to newer style of distributed
deployment by always setting credentials as ENVs.

Fixes #6434
2018-09-07 11:18:49 -07:00
Praveen raj Mani
e7a4512a90 Redis documentation Fix (#6378)
Fixed few typos and missing `format` field in the example config provided.

Fixes #6356
2018-09-07 07:12:01 -07:00
Kaan Kabalak
ff822e64ca Show pointer when user hovers over folder name (#6412)
To indicate that there is an action tied when the user clicks on the
folder name on the Minio Browser, the default cursor has been replaced
with a pointer on folder name hover. This is a regression from #6193.

Fixes #6411
2018-09-07 14:15:53 +05:30
Annanay Agarwal
7cb87f863e Kafka (sarama) authentication with user/pass (#6291) 2018-09-07 00:01:58 -07:00
Nitish Tiwari
67d8396af4 Fix Manta gateway client creation flow (#6425)
This commit fixes the Manta gateway client creation flow. We now affix
the endpoint scheme with endpoint URL while creating the Manta client
for gateway.

Also add steps in Manta gateway docs on how to run with custom Manta
endpoint.

Fixes #6408
2018-09-07 08:41:42 +05:30
Janko Marohnić
8b0cc376f4 Remove "List Object Parts" from Azure limitations (#6427)
Since https://github.com/minio/minio/pull/5198 has been implemented,
this is not a limitation anymore.
2018-09-06 17:19:51 -07:00
Krishnan Parthasarathi
9e5c4df106 Gateway should honour --address flag while checking port availability (#6428)
Fixes #6426
2018-09-06 16:42:33 -07:00
Andreas Auernhammer
fd8749f42a return Access Denied for invalid SSE keys (#6432)
This commit fixes are regression in the server regarding
handling SSE requests with wrong SSE-C keys.

The server now returns an AWS S3 compatable API error (access denied)
in case of the SSE key does not match the secret key used during upload.

Fixes #6431
2018-09-06 12:31:12 -07:00
Anis Elleuch
5c13765168 postgresql: Disable validation of connectionString field (#6397)
A bug concerning the validation of connectionString is found,
however there is no solution to fix it for now, postgresql API
doesn't help to do that hence disabling validation of that field.
2018-09-06 20:34:52 +05:30
Anis Elleuch
3099af70a3 Add admin get/set config keys API (#6113)
This PR adds two new admin APIs in Minio server and madmin package:
- GetConfigKeys(keys []string) ([]byte, error)
- SetConfigKeys(params map[string]string) (err error)

A key is a path in Minio configuration file, (e.g. notify.webhook.1)

The user will always send a string value when setting it in the config file,
the API will know how to convert the value to the appropriate type. The user
is also able to set a raw json.

Before setting a new config, Minio will validate all fields and try to connect
to notification targets if available.
2018-09-06 20:33:18 +05:30
Harshavardhana
fd1b8491db Drain response body properly for http connection pool (#6415)
Currently Go http connection pool was not being properly
utilized leading to degrading performance as the number
of concurrent requests increased.

As recommended by Go implementation, we have to drain the
response body and close it.
2018-09-05 16:47:14 -07:00
Anis Elleuch
1961f2ef54 xl: Fix removing an empty directory (#6421)
Removing an empty directory is not working because of xl.DeleteObject()
was only checking if the passed prefix is an actual object but it
should also check if it is an empty directory.
2018-09-05 16:38:03 -07:00
Harshavardhana
631c78e655 Bump up soMaxConn backlog for listener to 2048 (#6416)
soMaxConn value is 128 on almost all linux systems,
this value is too low for Minio at times when used
against large concurrent workload e.g: spark applications
this causes a sort of SYN flooding observed by the kernel
to allow for large backlog increase this value to 2048.

With this value we do not see anymore SYN flooding
kernel messages.
2018-09-05 13:16:19 -07:00
Harshavardhana
e0f8b767ba Fail for critical errors early on during prepare storage (#6404) 2018-09-05 10:20:54 -07:00
Harshavardhana
d0d015361c Fix config subsystem to wait on quorum number of formatted disks (#6407) 2018-09-05 20:55:55 +05:30
Krishna Srinivas
81b7e5c7a8 Send length instead of empty buffer for ReadFile() (#6414) 2018-09-04 23:22:05 -07:00
Harshavardhana
9e32cc283f Fix distributed docs to mention homogenous envs (#6405)
Also deprecate old syntax use only ellipses
2018-09-05 08:54:04 +05:30
Krishnan Parthasarathi
1126410e62 Implement ListMultipartUploads, ListObjectParts for GCS gateway (#6377)
ListMultipartUploads implementation is meant for docker-registry
use-case only. It lists only the first upload with a prefix matching
the object being uploaded.
2018-09-04 13:11:33 -07:00
Kanagaraj Mayilsamy
2c46214291 fix browser hang when listobjects is denied (#6385)
When Deny ListObjects policy is set on bucket, the browser
is hanging while trying to list the object. This is now
fixed by showing appropriate error message and an
empty object list.

Fixes #6371
2018-09-04 13:02:02 -07:00
Barnaby Keene
d13bd5b9b5 Remove double backtick that was breaking docs (#6410)
On the documentation site, the double backtick with nothing in between was breaking the page render and making the text itself look quite awkward!
2018-09-04 12:06:57 -07:00
Minio Trusted
c8c70a3750 Update yaml files to latest version RELEASE.2018-09-01T00-38-25Z 2018-09-01 00:43:06 +00:00
ebozduman
882a1a1ccc Stops listing objects for write-only access (#6396) 2018-08-31 13:20:27 -07:00
Harshavardhana
8690d62146 Allow fallback listen if first listener fails (#6380)
On linux listen() uses kernel features TCP_FASTOPEN, DEFER_ACCEPT

Fixes #6379
2018-08-31 13:17:05 -07:00
Anis Elleuch
85117d554f xl: Avoid removing a directory if it is not an object dir (#6395)
DeleteObject should not remove any directory unless the latter
is an empty directory.

Fixes #6394
2018-08-31 13:16:35 -07:00
Harshavardhana
4487f70f08 Revert all GetObjectNInfo related PRs (#6398)
* Revert "Encrypted reader wrapped in NewGetObjectReader should be closed (#6383)"

This reverts commit 53a0bbeb5b.

* Revert "Change SelectAPI to use new GetObjectNInfo API (#6373)"

This reverts commit 5b05df215a.

* Revert "Implement GetObjectNInfo object layer call (#6290)"

This reverts commit e6d740ce09.
2018-08-31 13:10:12 -07:00
Harshavardhana
fb27388101 HTTP headers are case insensitive handle them appropriately (#6390)
An issue was reproduced when minio-js client functional
tests are setting lower case http headers, in our current
master branch we specifically look for canonical host header
which may be not necessarily true for all http clients.
This leads to a perpetual hang on the *net.Conn*.

This PR fixes regression caused by #6206 by handling the
case insensitivity.
2018-08-31 17:00:32 +05:30
Kaan Kabalak
5e7ccc983d UI: Handle policies listed in mc as 'none' by not showing them (#6353) 2018-08-30 15:20:27 -07:00
Bala FA
72fa2b4537 Add RPC counters for HTTP stats. (#6206)
This patch introduces separate counters for HTTP stats for minio
reserved bucket.

Fixes #6158
2018-08-30 14:17:58 +05:30
Praveen raj Mani
5399d91965 Add version for appveyor to use go1.10.4 (#6384) 2018-08-30 14:09:31 +05:30
Harshavardhana
53a0bbeb5b Encrypted reader wrapped in NewGetObjectReader should be closed (#6383) 2018-08-29 19:18:00 -07:00
Harshavardhana
384a862940 Return quorum error based on disks in abortMultipartUpload (#6362)
Fixes #4980
2018-08-29 13:36:19 -07:00
Harshavardhana
029f52880b With no read quorum config should be treated as notFound (#6374)
This will allow the config subsystem to initialize properly
in situations where many servers are coming up in a rolling
fashion.
2018-08-28 14:23:22 -07:00
Harshavardhana
5b05df215a Change SelectAPI to use new GetObjectNInfo API (#6373)
This PR also removes some double checks
2018-08-28 13:08:30 -07:00
Harshavardhana
a13cd7b7c4 Separate build functional tests from Makefile (#6351)
Recently travis seems to have issues with builds after
merge, so this PR is an experiment to fix this.

Failed: https://travis-ci.org/minio/minio/builds/419769285#L2310

Succeeded: https://travis-ci.org/minio/minio/builds/419565606#L2322

This PR was tested to be working properly on my fork to build
on travis, would need to do the same in minio/minio and see
how it goes through.
2018-08-28 13:57:01 +05:30
Anis Elleuch
d524924b80 Fix gateway s3 doc to run custom S3 endpoint (#6369)
To pass a custom S3 endpoint in S3 gateway, the user needs
to specify it as an argument after 'minio gateway s3' and not
as '--address' option since this latter specifies the address
to which the gateway should listen.
2018-08-28 10:09:07 +05:30
Aditya Manthramurthy
e6d740ce09 Implement GetObjectNInfo object layer call (#6290)
This combines calling GetObjectInfo and GetObject while returning a
io.ReadCloser for the object's body. This allows the two operations to
be under a single lock, fixing a race between getting object info and
reading the object body.
2018-08-27 15:28:23 +05:30
Harshavardhana
cea4f82586 Revert "Remove curl after the usage (#6347)" (#6366)
This reverts commit dafa5073cb.

This commit revert also fixes #6365
2018-08-26 23:40:21 -07:00
Minio Trusted
1d6ce115da Update yaml files to latest version RELEASE.2018-08-25T01-56-38Z 2018-08-25 02:04:10 +00:00
Anis Elleuch
06d2dfa31c Fix WORM and BROWSER status calculation (#6360)
One typo introduced in a recent commit miscalculates if worm and browser
are enabled or not. A simple test is also added to detect this issue
in the future if it ever happens again.
2018-08-24 14:36:14 -07:00
poornas
d547873b17 webhandler - display encryption errors properly (#6339)
For encrypted objects, download errors need to be
displayed in web response format instead of xml format.

Fixes #6327
2018-08-24 07:56:24 -07:00
Praveen raj Mani
01721a840a Build fix for go1.11rc1 (#6354)
Vendorized the following packages

- "github.com/rjeczalik/notify"
- "github.com/minio/highwayhash"

Fixes #6315
2018-08-24 13:29:17 +05:30
Krishna Srinivas
52f6d5aafc Rename of structs and methods (#6230)
Rename of ErasureStorage to Erasure (and rename of related variables and methods)
2018-08-23 23:35:37 -07:00
Harshavardhana
2211a5f1b8 Avoid ListenBucket targets to be listed in ServerInfo (#6340)
In current master when you do `mc watch` you can see a
dynamic ARN being listed which exposes the remote IP as well

```
mc watch play/airlines
```

On another terminal
```
mc admin info play
●  play.minio.io:9000
   Uptime : online since 11 hours ago
  Version : 2018-08-22T07:50:45Z
   Region :
 SQS ARNs : arn:minio:sqs::httpclient+51c39c3f-131d-42d9-b212-c5eb1450b9ee+73.222.245.195:33408
    Stats : Incoming 30GiB, Outgoing 7.6GiB
  Storage : Used 7.7GiB
```

SQS ARNs listed as part of ServerInfo should be only external targets,
since listing an ARN here is not useful and it cannot be re-purposed in
any manner.

This PR fixes this issue by filtering out httpclient from the ARN list.

This is a regression introduced in #5294 0e4431725c
2018-08-23 23:31:14 -07:00
Harshavardhana
1ffa6adcd4 Ignore io.EOF returned by ReadFrom for zero byte fs.json (#6346)
Fixes #6256
2018-08-24 11:34:21 +05:30
kannappanr
add57a6938 Add content-length as part of event notification structure (#6341)
Fixes #6321
2018-08-23 14:40:54 -07:00
Manjunath A Kumatagi
dafa5073cb Remove curl after the usage (#6347) 2018-08-23 10:48:29 -07:00
Praveen raj Mani
65e05a06fb Remove notifications Fix (#6082)
Remove all the notifications for an empty rulesMap

Fixes #6053
2018-08-23 22:53:18 +05:30
Dee Koder
5c9354894b Update Issue & PR Templates to have regression information when available (#6342) 2018-08-23 01:41:22 -07:00
Harshavardhana
b01e69e08f Initialize global object layer after all subsystems have initialized (#6333)
This is to ensure that object API operations are not performed
on a server on which subsystems are yet to be initialized.
2018-08-22 23:11:17 -07:00
Raphael Randschau
8601f29d95 select: fix int overflow of math.MaxInt64 on ARM (#6317) 2018-08-22 16:16:04 +05:30
Nitish Tiwari
0aee722e3f Fix Minio browser screenshots in docs (#6334)
Fixes #6308
2018-08-22 13:15:36 +05:30
Harshavardhana
beb6d40ce6 Avoid crash when policy subsystem is not initialized (#6326)
Fixes #6324
2018-08-21 15:38:51 -07:00
poornas
19db921555 CopyObject: fix regression in key rotation (#6331)
After key rotation, metadata was not being replaced with new sealed key.
Regression introduced in commit e71ef905f9
2018-08-21 15:12:00 -07:00
Minio Trusted
68b9e9e7e7 Update yaml files to latest version RELEASE.2018-08-21T00-37-20Z 2018-08-21 00:44:41 +00:00
kannappanr
2d84b02bc4 Check for absence of checksum field and attributes. (#6298)
Fixes #6295
2018-08-20 16:58:47 -07:00
poornas
8b2801bd46 Update documentation to show how to update minio config fields (#6301)
- with recent commit 1fb2e9ef95, config
can no longer be updated by editing config.json. This is because config
has been migrated inside the minio backend. Update documentation on
how to set/get configuration using mc admin config command.
2018-08-20 13:37:10 -07:00
Harshavardhana
7d7e21aebb Merge initConfig logic to ConfigSys (#6312) 2018-08-19 13:57:18 -07:00
Nitish Tiwari
bf14e5ce1b Fix distributed doc as total storage capacity is not displayed anymore (#6309) 2018-08-18 10:01:24 -07:00
Andreas Auernhammer
d531080b7e add SSE-KMS not-implemented error handling (#6234)
This commit adds error handling for SSE-KMS requests to
HEAD, GET, PUT and COPY operations. The server responds
with `not implemented` if a client sends a SSE-KMS
request.
2018-08-17 21:07:19 -07:00
Harshavardhana
a6b8a5487a Fail gateway properly with an error on port conflicts (#6303) 2018-08-17 21:06:36 -07:00
Minio Trusted
6c0d53a1c5 Update yaml files to latest version RELEASE.2018-08-18T03-49-57Z 2018-08-18 03:54:06 +00:00
Harshavardhana
9f14433cbd Ensure that setConfig uses latest functionality (#6302) 2018-08-17 18:51:34 -07:00
Harshavardhana
50a817e3d3 Use new listener which implements enhanced tcp features (#6289)
This package provide customizable TCP net.Listener with various
performance-related options:

 * SO_REUSEPORT. This option allows linear scaling server performance
   on multi-CPU servers.
   See https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/ for details.
 * TCP_DEFER_ACCEPT. This option expects the server reads from the accepted
   connection before writing to them.
 * TCP_FASTOPEN. See https://lwn.net/Articles/508865/ for details.
2018-08-17 18:44:02 -07:00
Harshavardhana
5a4a57700b Add select docs and fix return values for Select API (#6300) 2018-08-17 17:11:39 -07:00
Harshavardhana
3de5a3157f Enhance picking valid xlMeta based on quorum (#6297)
This PR borrows the idea from getFormatXLQuorum()
2018-08-17 14:42:04 -07:00
Kaan Kabalak
50dec08002 Correct link paths in Chinese documentation (#6299) 2018-08-17 13:16:17 -07:00
poornas
e71ef905f9 Add support for SSE-S3 server side encryption with vault (#6192)
Add support for sse-s3 encryption with vault as KMS.

Also refactoring code to make use of headers and functions defined in
crypto package and clean up duplicated code.
2018-08-17 12:52:14 -07:00
junpeng liu
3d197c1449 Modify several translation errors (#6038) 2018-08-17 12:04:09 +05:30
Harshavardhana
65de2d68c0 Allow for proper garbage collection pooling bytes.Buffer (#6266) 2018-08-16 18:37:43 -07:00
Harshavardhana
1103ad2d08 Watch for symlinked certs in container envs (#6282)
Fixes #6278
2018-08-16 18:37:21 -07:00
Harshavardhana
eab947cf42 Make sure to update modTime in erasure metadata (#6296)
This is to ensure that when we update xl.json with new
parts have the latest modtime, in-turn avoids consistency
issues when the disk is offline.
2018-08-16 17:55:01 -07:00
Harshavardhana
0fe9e95250 Validate prefixes on all sets (#6294)
This PR fixes a regression introduced in 8eb838bf91
where hashing technique was used on prefixes to get the right set
to perform the operation, this is not correct since prefixes and
their corresponding keys might hash to a different value depending
on the key length.

For prefixes/directories we should look everywhere to support proper
quorum based listing.

Fixes #6293
2018-08-16 16:49:38 -07:00
kannappanr
c7946ab9ab Remove unnecessary error log messages (#6186) 2018-08-16 12:57:49 -07:00
Harshavardhana
f5df3b4795 Remove select docs (#6287)
Select API is sufficiently documented, this doc is also incomplete.

- https://aws.amazon.com/blogs/aws/s3-glacier-select/
- https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-ruby/
- https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/
- https://aws.amazon.com/blogs/developer/category/storage/s3-select/
2018-08-15 19:47:22 -07:00
Harshavardhana
f26325c988 Support supplying custom drives per set count (#6261) 2018-08-15 16:35:21 -07:00
Arjun Mishra
7c14cdb60e S3 Select API Support for CSV (#6127)
Add support for trivial where clause cases
2018-08-15 03:30:19 -07:00
Harshavardhana
0e02328c98 Migrate config.json from config-dir to backend (#6195)
This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.

In future releases, we will slowly migrate out of the
current architecture.

Fixes #6182
2018-08-15 10:11:47 +05:30
Harshavardhana
380524ae27 Unlock read lock on uploadID upon errors (#6283) 2018-08-14 18:35:30 -07:00
kannappanr
0286e61aee Log disk not found error just once (#6059)
Modified the LogIf function to log only if the error passed
is not on the ignored errors list.

Currently, only disk not found error is added to the list.
Added a new function in logger package called LogAlwaysIf, 
which will print on any error.

Fixes #5997
2018-08-14 13:58:48 -07:00
wd256
ff29aed05d gcs: Translate S3 user-defined metadata prefix to/from GCS custom metadata prefix (#6270) 2018-08-14 11:53:39 -07:00
Harshavardhana
64f2c61813 Implement memory efficient readdir for windows (#6247)
Fixes #6164
2018-08-09 14:52:29 -07:00
Andreas Auernhammer
525c04fd07 crypto: add SSE-KMS HTTP header detection (#6228)
This commit adds support for detecting SSE-KMS headers.
The server should be able to detect SSE-KMS headers to
at least fail such S3 requests with not implemented.
2018-08-09 13:02:57 -07:00
Sanat Mouli
efac90461a Fix bug in Share Object Modal (#6257)
Fixes #6249
2018-08-09 09:45:43 -07:00
Durgadas Kamath
71979376b5 Add s390x support (#6263) 2018-08-08 15:49:12 -07:00
Anis Elleuch
5a1ae862a7 Avoid sending an error after 206 HTTP code (#6264)
When a S3 client sends a GET Object with a range header, 206 http
code is returned indicating success, however the call of the object
layer's GetObject() inside the handler can return an error and will lead
to writing an XML error message, which is obviously wrong since
we already sent 206 http code. So in the case, we just stop sending
data to the S3 client, this latter can still detect if there is no
error when comparing received data with Content-Length header
in the Get Object response.
2018-08-08 15:39:47 -07:00
Anis Elleuch
6df20734f9 Avoid logging the body of the http 206 response (#6258)
When an S3 client issues a GET request with range specified, Minio
server returns some partial data with 206 http code. The latter
is sent in MINIO_HTTP_TRACE output which is incorrect. This PR
fixes the issue.
2018-08-08 12:34:42 -07:00
Sanat Mouli
98bce72295 Fix output path for go-bindata-assetfs (#6253) 2018-08-08 10:46:02 +05:30
Harshavardhana
2f1756489e Add tracing capabilities for internode rpc Servers (#6254) 2018-08-07 15:21:30 +05:30
Harshavardhana
9719640e34 Use sha256-simd instead of crypto/sha256 (#6252) 2018-08-06 18:00:10 -07:00
Krishna Srinivas
ce02ab613d Simplify erasure code by separating bitrot from erasure code (#5959) 2018-08-06 15:14:08 -07:00
Oleg Kovalov
37de2dbd3b simplifying if-else chains to switches (#6208) 2018-08-06 10:26:40 -07:00
Harshavardhana
a82500f162 Support dumb terminals by turning off color (#6246)
ANSI colors do not work on dumb terminals, in situations
when minio is running as a service under systemd.

This PR ensures we turn off color in those situations.
2018-08-06 18:16:49 +05:30
Harshavardhana
2dede2fdc2 Add reliable RemoveAll to handle racy situations (#6227) 2018-08-06 09:45:28 +05:30
Harshavardhana
13fbb96736 Hold locks granularly in nslockMap (#6242)
With benchmarks increases the performance for small files
by almost 4x times the previous releases.
2018-08-06 08:55:25 +05:30
Harshavardhana
eabfcea34e Add granular locking in retryTicker (#6236)
This is to avoid serializing RPC contention on ongoing
parallel operations, the blocking profile indicating
all calls were being serialized through setRetryTicker.
2018-08-03 18:57:00 -07:00
Andreas Auernhammer
a078703214 catch crypto.* errors and add SSE-S3 invalid algorithm err (#6229)
This commit adds the crypto.* errors to the
`toAPIErrorCode` switch. Further this commit adds an S3
API error code returned whenever the client specifes a
SSE-S3 request with an invalid algorithm parameter.
2018-08-03 16:55:45 -07:00
Harshavardhana
bd2b22572f Increase max idle connections from 100 to 4096 (#6244)
This is to be inline with our benchmarking results
2018-08-03 16:29:28 -07:00
poornas
5f69f04909 nas gateway: fix regression in global bucket policy initialization (#6243)
Fixes #6238
globalPolicySys used to be initialized in fs/xl layer. The referenced
commit moved this logic to server/gateway initialization,but a check
to avoid double initialization prevented globalPolicySys to be loaded
from disk for NAS.

fixes regression from commit be1700f595
2018-08-03 15:12:18 -07:00
Minio Trusted
a1a426e523 Update yaml files to latest version RELEASE.2018-08-02T23-11-36Z 2018-08-02 23:17:30 +00:00
Harshavardhana
a091b1a3ee Fix admin API doc formatting (#6235) 2018-08-02 14:21:38 -07:00
Harshavardhana
556a51120c Deprecate ListLocks and ClearLocks (#6233)
No locks are ever left in memory, we also
have a periodic interval of clearing stale locks
anyways. The lock instrumentation was not complete
and was seldom used.

Deprecate this for now and bring it back later if
it is really needed. This also in-turn seems to improve
performance slightly.
2018-08-02 23:09:42 +05:30
poornas
eb391a53c1 check for syscall errors in posix-errors helper functions (#6232) 2018-08-02 10:38:51 +05:30
Harshavardhana
e17e09ea3c Handle POST object upload without filename param (#6221)
POST mime/multipart upload style can have filename value optional
which leads to implementation issues in Go releases in their
standard mime/multipart library.

When `filename` doesn't exist Go doesn't update `form.File` which
we rely on to extract the incoming file data, strangely when `filename`
is not specified this data is buffered in memory and is now part of
`form.Value` instead of `form.File` which creates an inconsistent
behavior.

This PR tries to fix this in our code for the time being, but ideal PR
would be to fix the upstream mime/multipart library to handle the
above situation consistently.
2018-08-01 14:19:11 -07:00
Andreas Auernhammer
76c423392a crypto: add GenerateIV from random IV generation (#6215)
This commit adds a `GenerateIV` function to simplify
the generation of random IVs.

It adds some unit tests for `GenerateIV` in key_test.go
2018-08-01 01:02:07 -07:00
Aarushi Arya
8e6d756e3a Appropriate error message on unsuccessful update. (#6203) 2018-08-01 01:01:37 -07:00
Andreas Auernhammer
a7c9058375 crypto: implement Stringer for S3 and SSEC (#6216)
This commit adds a `fmt.Stringer` implementation for
SSE-S3 and SSE-C. The string representation is the
domain used for object key sealing.
See: `ObjectKey.Seal(...)` and `ObjectKey.Unseal(...)`
2018-07-31 11:15:12 -07:00
Nitish Tiwari
b16e33bcf5 Fix Kubernetes TLS doc to avoid creating CAs dir on read only mount (#6214) 2018-07-31 10:58:34 -07:00
Nitish Tiwari
197af49c99 Fix healthcheck handler to verify gateway backend liveness (#6218)
Fixes #6217
2018-07-31 10:55:34 -07:00
kannappanr
264cc4020f Return 503 instead of 404 if more than half of disks are not found (#6207)
Fixes #6163
2018-07-31 00:23:29 -07:00
Minio Trusted
df88421087 Update yaml files to latest version RELEASE.2018-07-31T02-11-47Z 2018-07-31 02:17:27 +00:00
Harshavardhana
dbd89bbae3 Remove double RLocks for GetBucketInfo (#6209) 2018-07-30 17:38:52 -07:00
Aditya Manthramurthy
224a272cf2 Fix type of bitrot mismatch error (#6205)
The error type `hashMismatchError` is lost when the error is received
from a remote disk.

Fixes #6201
2018-07-29 15:00:37 +05:30
Harshavardhana
ad86454580 Make sure to handle FaultyDisks in listing ops (#6204)
Continuing from PR 157ed65c35

Our posix.go implementation did not handle  I/O errors
properly on the disks, this led to situations where
top-level callers such as ListObjects might return early
without even verifying all the available disks.

This commit tries to address this in Kubernetes, drbd/nbd based
persistent volumes which can disconnect under load and
result in the situations with disks return I/O errors.

This commit also simplifies listing operation, listing
never returns any error. We can avoid this since we pretty
much ignore most of the errors anyways. When objects are
accessed directly we return proper errors.
2018-07-27 15:32:19 -07:00
Andreas Auernhammer
644c2ce326 crypto: add support for parsing/creating SSE-C/SSE-S3 metadata (#6169)
* crypto: add support for parsing SSE-C/SSE-S3 metadata

This commit adds support for detecting and parsing
SSE-C/SSE-S3 object metadata. With the `IsEncrypted`
functions it is possible to determine whether an object
seems to be encrypted. With the `ParseMetadata` functions
it is possible to validate such metadata and extract the
SSE-C/SSE-S3 related values.

It also fixes some naming issues.

* crypto: add functions for creating SSE object metadata

This commit adds functions for creating SSE-S3 and
SSE-C metadata. It also adds a `CreateMultipartMetadata`
for creating multipart metadata.

For all functions unit tests are included.
2018-07-25 13:35:54 -07:00
Harshavardhana
2debe77586 Remove error returned when part sizes are un-equal (#6183)
Since implementing `pwrite` like implementation would
require a more complex code than background append
implementation, it is better to keep the current code
as is and not implement `pwrite` based functionality.

Closes #4881
2018-07-24 21:31:03 -07:00
Harshavardhana
20480ba3f7 Remove references to MINIO_ENDPOINTS (#6200)
MINIO_ENDPOINTS is a special case scenario
we don't need to document it for now.
2018-07-24 17:17:25 -07:00
kannappanr
4f52d22c36 Fix make test failure (#6185)
Fix shadowing errors.
2018-07-24 14:17:58 -07:00
Kaan Kabalak
cbe8df198e Replace pointer with default cursor on filename hover (#6193) 2018-07-24 13:34:20 +05:30
Harshavardhana
157ed65c35 Fix healthcheck handler to check errors in local disks only (#6184)
Healthcheck handler in current implementation was
performing ListBuckets() to check for liveness of Minio
service. ListBuckets() implementation on the other hand
doesn't do quorum based listing and if one of the disks
returned error, an I/O error it would be lead to kubernetes
taking the minio pod down prematurely even if the disk
is not local to that minio server.

The reason is ListBuckets() call cannot be trusted to
provide us the valid information that we need, Minio is a
clustered application which is designed to handle disk
failures. Error on one of the disks doesn't mean the pod
should become fully non-operational.

This PR attempts to fix this by only checking for alive
disks which are local to each setup and also by simply
performing a Stat() operation, if the Stat() returned
error on all disks local to a particular server then
we can let kubernetes safely take it down, until then
we should be operational.
2018-07-23 12:21:25 -07:00
Minio Trusted
869018ad14 Update yaml files to latest version RELEASE.2018-07-23T18-34-49Z 2018-07-23 18:39:08 +00:00
Harshavardhana
5acc2a6db1 S3 gateway signature probe use a unique bucket (#6190)
This fixes an issue because someone is
using `probe-bucket-sign` bucket name in
region 'eu-central-1'
2018-07-23 10:16:56 -07:00
kannappanr
2cd14f567c Do not set Key and BucketName in ErrorResponse, if empty (#6174) 2018-07-23 14:09:09 +05:30
Harshavardhana
f1be356cc6 Do not use parallel deletes to avoid random I/O (#6178)
The current code for deleting 1000 objects simultaneously
causes significant random I/O, which on slower drives
leads to servers disconnecting in a distributed setup.

Simplify this by serially deleting and reducing the
chattiness of this operation.
2018-07-20 21:21:01 -07:00
kannappanr
76ddf4d32f Log x-amz-request-id as log and XML error response (#6173)
Currently, requestid field in logEntry is not populated, as the
requestid field gets set at the very end.
It is now set before regular handler functions. This is also
useful in setting it as part of the XML error response.

Travis build for ppc64le has been quite inconsistent and stays queued
for most of the time. Removing this build as part of Travis.yml for
the time being.
2018-07-20 18:46:32 -07:00
kannappanr
7b91bd71fe Remove ppc64le support from .travis.yml (#6180)
Travis CI for ppc64le stays in queued state.
Removing the support for now.
2018-07-20 18:39:55 -07:00
Harshavardhana
36ab615518 Remove unused functions and constants (#6175) 2018-07-20 23:37:43 +05:30
kannappanr
963a70053b Do not trace HTTP body of PostPolicyBuckethandler (#6177) 2018-07-20 07:37:46 -07:00
Anis Elleuch
9c5e971a58 Add new console/http loggers (#6066)
- Add console target logging, enabled by default.
- Add http target logging, which supports an endpoint
  with basic authentication (username/password are passed
  in the endpoint url itself)
- HTTP target logging is asynchronous and some logs can be
  dropped if channel buffer (10000) is full
2018-07-19 15:55:06 -07:00
Anis Elleuch
b1c9eb0e01 Disable splitting lines in pretty error messages (#6171)
In a small window, UI error tries to split lines for an eye candy
error message. However, since we show some docs.minio.io links in some
error messages, these links are actually broken and not easily selected
in a X terminal. This PR changes the behavior and won't split lines
anymore.
2018-07-19 15:49:02 -07:00
Nitish Tiwari
b8f4f26cf6 Add S3 gateway documentation (#6165)
Fixes #4830
2018-07-19 11:54:38 -07:00
kannappanr
43cc0096fa Add support for deployment ID (#6144)
deployment ID helps in identifying a minio deployment in the case of remote
logging targets.
2018-07-18 20:17:35 -07:00
Anis Elleuch
e8a008f5b5 Better validation of all config file fields (#6090)
Add Validate() to serverConfig to call it at server
startup and in Admin SetConfig handler to minimize
errors scenario after server restart.
2018-07-18 11:22:29 -07:00
Andreas Auernhammer
758a80e39b crypto: add basic functionality for parsing SSE-C headers (#6148)
This commit adds basic support for SSE-C / SSE-C copy.
This includes functions for determining whether SSE-C
is requested by the S3 client and functions for parsing
such HTTP headers.

All S3 SSE-C parsing errors are exported such that callers
can pattern-match to forward the correct error to S3
clients.

Further the SSE-C related internal metadata entry-keys
are added by this commit.
2018-07-18 10:49:26 -07:00
wd256
3ec4738955 gcs: Use Pager to iterate results in ListObjectsV1/V2 (#6162)
Fixes #6052
2018-07-18 21:49:16 +05:30
Andreas Auernhammer
6c93c60424 crypto: add a basic KMS implementation (#6161)
This commit adds a basic KMS implementation for an
operator-specified SSE-S3 master key. The master key
is wrapped as KMS such that using SSE-S3 with master key
and SSE-S3 with KMS can use the same code.

Bindings for a remote / true KMS (like hashicorp vault)
will be added later on.
2018-07-17 22:40:34 -07:00
Krishna Srinivas
0c9f4c9092 formatMetaV1 should be "inherited" by disk format structs (#6134) 2018-07-16 20:26:42 -07:00
Andreas Auernhammer
289d6ce1d7 crypto: update SSE-S3 and SSE-C key derivation (#6152)
This commit updates the key derivation to reflect the
latest change of crypto/doc.go. This includes handling
the insecure legacy KDF.

Since #6064 is fixed, the 3. test case for object key
generation is enabled again.
2018-07-16 07:49:50 -07:00
Rob Girard
2a12e694f3 Changed command line examples (#6149)
Order for server:disk originally provided wouldn't stripe 
wide and may lead to availability issues.

Also added Short-form examples using {1...n} and a 
warning about {1..2} vs {1...3}
2018-07-14 20:48:38 +05:30
Harshavardhana
db26d3c9e2 Fix handling files at leaf attempting disk.ListDir() (#6155)
Return an ignorable error upon readDir() failure on
a file.
2018-07-14 12:11:48 +05:30
Stefan Husch
914c76a801 Implement lock.Open() to fix #5642 (#6150) 2018-07-13 18:37:02 -07:00
Krishnan Parthasarathi
a1ef90be52 gcs: Limit number of objects listed to max-keys (#6133) 2018-07-13 10:27:26 -07:00
Minio Trusted
7c4a41b933 Update yaml files to latest version RELEASE.2018-07-13T00-09-07Z 2018-07-13 00:43:51 +00:00
Nitish Tiwari
2aa18cafc6 Update federation target to etcd/clientv3 (#6119)
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.

Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
2018-07-12 14:12:40 -07:00
Andreas Auernhammer
adf7340394 fix size computation for en/decrypted objects (#6147)
This PR fixes the size calculation for encrypted multipart
objects.
2018-07-12 11:23:32 -07:00
Harshavardhana
b11a8eb3f4 Support multiple values for x-amz-meta header (#6145)
Fixes #5595
2018-07-12 09:40:14 -07:00
Andreas Auernhammer
15771ebe8d Fix decrypted object size and key derivation in CopyObjectPart (#6141)
This commit fixes the size calculation for multipart
objects. The decrypted size of an encrypted multipart
object is the sum of the decrypted part sizes.

Also fixes the key derivation in CopyObjectPart.
Instead of using the same object-encryption-key for each
part now an unique per-part key is derived.

Updates #6139
2018-07-12 21:59:56 +05:30
Praveen raj Mani
44865596db SignatureV4 validation with Metadata in the presignedUrl (#5894)
The `X-Amz-Meta-`/`X-Minio-Meta-` will now be recognized in query string also.

Fixes #5857 #5950
2018-07-10 20:27:10 -07:00
Nitish Tiwari
c9bc7e47b9 Update distributed docs (#6123)
We need to clarify that distributed Minio doesn't strictly need a
fresh drive, instead it just needs a fresh directory on the drive.
2018-07-10 07:32:24 +05:30
Anis Elleuch
be1700f595 Avoid startup abort when a notify target is down (#6126)
Minio server was preventing itself to start when any notification
target is down and not running. The PR changes the behavior by
avoiding startup abort in that case, so the user will still
be able to access Minio server using mc admin commands after
a restart or set config commands.
2018-07-10 07:20:31 +05:30
Minio Trusted
42c5b64e4e Update yaml files to latest version RELEASE.2018-07-10T01-42-11Z 2018-07-10 01:46:03 +00:00
Krishna Srinivas
40ed0d1f5d Support 1GB disk size (#6137)
Pivotal CF by default has 1GB disk option which causes minio to not start
2018-07-09 18:23:49 -07:00
Andreas Auernhammer
b181a693fb fix object rebinding SSE-C security guarantee violation (#6121)
This commit fixes a weakness of the key-encryption-key
derivation for SSE-C encrypted objects. Before this
change the key-encryption-key was not bound to / didn't
depend on the object path. This allows an attacker to
repalce objects - encrypted with the same
client-key - with each other.

This change fixes this issue by updating the
key-encryption-key derivation to include:
 - the domain (in this case SSE-C)
 - a canonical object path representation
 - the encryption & key derivation algorithm

Changing the object path now causes the KDF to derive a
different key-encryption-key such that the object-key
unsealing fails.
Including the domain (SSE-C) and encryption & key
derivation algorithm is not directly neccessary for this
fix. However, both will be included for the SSE-S3 KDF.
So they are included here to avoid updating the KDF
again when we add SSE-S3.

The leagcy KDF 'DARE-SHA256' is only used for existing
objects and never for new objects / key rotation.
2018-07-09 17:18:28 -07:00
Anis Elleuch
4ddc222f46 fix: Propagate bucket policy update in a distributed setup (#6135)
Commit 0d52126023 caused a regression in setting
a new bucket policy in a distributed setup. The reason is that gob is not able
to encode fields declared as interfaces unless we provide GobEncode() and GobDecode()
This PR adds them by using json marshaller and unmarshaller that are already
implemented for Functions interface.
2018-07-09 02:18:48 -07:00
Mike Scarlett
c310cbbe89 Update comments regarding GCS component count (#6131) 2018-07-06 17:07:11 -07:00
Harshavardhana
0ef0d7e685 pkg/certs: On windows watch for directory changes to load certs (#6128)
This PR fixes an issue when configuring Minio TLS on windows
2018-07-05 16:33:37 -07:00
wksw
c62813c887 completed sample config and fix wrong json format (#6096)
missing worm configuration and wrong json format
2018-07-05 13:40:07 +05:30
poornas
1da362538b cache: allow ellipse style entries for MINIO_CACHE_DRIVES (#6088)
Fixes #5863
2018-07-03 16:54:10 -07:00
Krishna Srinivas
e40a5e05e1 Do notification in background to not block S3 client REST calls (#6005) 2018-07-03 11:09:36 -07:00
ebozduman
b0b0fb4c8d Validate Minio config.json file on the client side (#6067) 2018-07-03 11:07:46 -07:00
kannappanr
726e75611e Do not log BucketNotFound errors on minio console (#6114) 2018-07-03 11:04:55 -07:00
Rafael Peria de Sene
317e648c0d Add support for ppc64le (#6116)
Signed-off-by: Rafael Peria de Sene <rpsene@br.ibm.com>
2018-07-03 09:42:19 -07:00
Harshavardhana
80b3e9cb03 use appropriate HTTP status for storage is full (#6117) 2018-07-03 09:40:14 -07:00
Harshavardhana
6c85706c24 Use GetSourceIP for source ip as request params (#6109)
Fixes #6108
2018-07-02 14:40:18 -07:00
Harshavardhana
d7ced9a8b5 Upgrade gjson to bring in new performance improvements (#6106) 2018-07-02 13:31:11 -07:00
Praveen raj Mani
360f3f9335 Checking the existence of the bucket in DeleteObjectHandler (#6085)
Fixes #6077
2018-06-30 22:35:43 -07:00
wd256
25f9b0bc3b Handle ListObjectsV2 start-after parameter in ObjectLayer (#6078) 2018-07-01 09:52:45 +05:30
Harshavardhana
a5453c307f Fix kernel tuning script to ignore write failures (#6107)
Certain SCSI drivers do not allow certain tuning parameters
like nr_requests, max_sectors_kb to be changed, ignore these
errors silently as this script is simply a best effort.

Fixes #6103
2018-06-30 14:55:21 -07:00
Harshavardhana
92a6676a2f Avoid unnecessary logging for policy not found errors (#6104) 2018-06-29 06:30:10 -07:00
Minio Trusted
f53d511798 Update yaml files to latest version RELEASE.2018-06-29T02-11-29Z 2018-06-29 02:14:58 +00:00
Harshavardhana
e5e522fc61 docs: fix all Chinese doc links for the new docs site (#6097)
Additionally fix typos, default to US locale words
2018-06-28 16:02:02 -07:00
Harshavardhana
de251483d1 Avoid ticker timer to simplify disk usage (#6101)
This PR simplifies the code to avoid tracking
any running usage events. This PR also brings
in an upper threshold of upto 1 minute suspend
the usage function after which the usage would
proceed without waiting any longer.
2018-06-28 15:05:45 -07:00
Andreas Auernhammer
805186ab97 new package: cmd/crypto (#6062)
This commit introduces a new crypto package providing
AWS S3 related cryptographic building blocks to implement
SSE-S3 (master key or KMS) and SSE-C.

This change only adds some basic functionallity esp.
related to SSE-S3 and documents the general approach
for SSE-S3 and SSE-C.
2018-06-28 12:47:42 -07:00
Praveen raj Mani
ea76e72054 Incorrect error message for insufficient volume fix (#6099)
Reply back with appropriate error message when the server is spawn
with volume of insufficient size (< 1GiB).

Fixes #5993.
2018-06-28 12:01:05 -07:00
Kaan Kabalak
8bd7a19d50 Refactor login page to get rid of warning on test (#6083) 2018-06-28 10:30:57 -07:00
Harshavardhana
25de775560 disable disk-usage when export is root mount path (#6091)
disk usage crawling is not needed when a tenant
is not sharing the same disk for multiple other
tenants. This PR adds an optimization when we
see a setup uses entire disk, we simply rely on
statvfs() to give us total usage.

This PR also additionally adds low priority
scheduling for usage check routine, such that
other go-routines blocked will be automatically
unblocked and prioritized before usage.
2018-06-27 18:59:38 -07:00
Harshavardhana
abf209b1dd load bucket policies using object layer API (#6084)
This PR fixes an issue during gateway mode
where underlying policies were not translated
into meaningful policies.
2018-06-27 12:29:48 +05:30
Harshavardhana
d9d13c898c docs: Fix admin heal API description (#6073) 2018-06-26 10:53:14 -07:00
Nitish Tiwari
ad79c626c6 Throw 404 for head requests for prefixes without trailing "/" (#5966)
Minio server returns 403 (access denied) for head requests to prefixes
without trailing "/", this is different from S3 behaviour. S3 returns
404 in such cases.

Fixes #6080
2018-06-26 06:54:00 +05:30
Andreas Auernhammer
cd152f404a replace os.Exit with panic for logger.CriticalIf (#6065)
This commit prevents complete server failures caused by
`logger.CriticalIf` calls. Instead of calling `os.Exit(1)`
the function now executes a panic with a special value
indicating that a critical error happend. At the top HTTP
handler layer panics are recovered and if its a critical
error the client gets an InternalServerError status code.

Further this allows unit tests to cover critical-error code
paths.
2018-06-25 13:51:49 -07:00
Praveen raj Mani
5fbdd70de9 Minio handle for list_objects_v2 with a blank ContinuationToken (#5940)
minio will now return an error for a blank continuation token in list_objects_v2,so as in s3.

Fixes #5931
2018-06-25 12:35:43 -07:00
Ashish Kumar Sinha
0bbdd02a57 Updating disk storage for FS/Erasure mode (#6081)
Updating the disk storage stats for FS/Erasure coded backend
2018-06-25 10:46:48 -07:00
Annanay Agarwal
78abe5234e Add functionality to make cache limit configurable (#5991) 2018-06-25 10:24:12 -07:00
Minio Trusted
f46ee54194 Update yaml files to latest version RELEASE.2018-06-22T23-48-46Z 2018-06-22 23:52:10 +00:00
Harshavardhana
6005dbf01f Fix broken doc links (#6068) 2018-06-22 09:14:41 +05:30
Nitish Tiwari
eb0e56ccf6 Update content-language field for Azure gateway (#6061)
Fixes https://github.com/minio/minio-go/issues/996
2018-06-21 09:46:45 -07:00
Dee Koder
c91abe6c4b Update with absolute path for images (#6060) 2018-06-21 08:29:12 +05:30
Aditya Manthramurthy
670b538dde Fix server crash when no system certificates are found (#6055) 2018-06-19 13:38:22 -07:00
Nitish Tiwari
186000328e Update StorageClass from Objectinfo (#6054)
See #6051
2018-06-19 11:22:08 -07:00
Harshavardhana
2575f4198a Auto-probe backend signature support (#6044) 2018-06-18 15:27:14 -07:00
Harshavardhana
05a64dee95 Do not leave stale entries on etcd on a failed update operation (#6048) 2018-06-18 12:05:17 -07:00
kannappanr
577d10674d Cleanup stack trace in error logs (#6045)
Add compile time GOROOT path to the list of prefix
of file paths to be removed.

Add webhandler function names to the slice that
stores function names to terminate logging.
2018-06-18 12:04:46 -07:00
kannappanr
81ee79b042 Do not log browser login failures (#6049) 2018-06-18 11:49:49 -07:00
kannappanr
d94500ae26 Makefile: Create bin directory if it doesn't exist (#6050) 2018-06-18 11:45:28 -07:00
kannappanr
001d9a4ae7 Remove unwanted volume access denied log (#6042)
Fixes #6039
2018-06-18 11:43:41 -07:00
Harshavardhana
c3a5146422 Presign content-sha256 should look for query and header (#6046) 2018-06-15 14:21:17 -07:00
Nitish Tiwari
36c39d04da Update vendorized minio-go to support start-after param (#6043)
Fixes #6032
2018-06-14 15:08:02 -07:00
Harshavardhana
28d526bc68 Change CriticalIf to FatalIf for proper error message (#6040)
During startup until the object layer is initialized
logger is disabled to provide for a cleaner UI error
message. CriticalIf is disabled, use FatalIf instead.

Also never call os.Exit(1) on running servers where
you can return error to client in handlers.
2018-06-14 10:17:07 -07:00
Harshavardhana
05f96f3956 Avoid allocating larger buffer if the incoming body is small (#6035) 2018-06-13 11:55:12 -07:00
Harshavardhana
cb9ee1584a Fix TestHealStartNStatusHandler sporadic failure (#6015)
Fixes #5818
2018-06-12 16:36:31 -07:00
Andreas Auernhammer
9f4c120731 limit memory allocations during multiple object deletion (#6033)
This commit limits the amount of memory allocated by the
S3 Multi-Object-Delete-API. The server used to allocate as
many bytes as provided by the client using Content-Length.

S3 specifies that the S3 Multi-Object-Delete-API can delete
at most 1000 objects using a single request.
(See: https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html)
Since the maximum S3 object name is limited to 1024 bytes the
XML body sent by the client can only contain up to 1000 * 1024
bytes (excluding XML format overhead).

This commit limits the size of the parsed XML for the S3
Multi-Object-Delete-API to 2 MB. This fixes a DoS
vulnerability since (auth.) clients, MitM-adversaries
(without TLS) and un-auth. users accessing buckets allowing
multi-delete by policy can kill the server.

This behavior is similar to the AWS-S3 implementation.
2018-06-11 10:51:38 -07:00
Harshavardhana
12a916091e Convert federation doc as quickstart guide (#6030) 2018-06-09 14:00:11 +05:30
Minio Trusted
842092f8de Update yaml files to latest version RELEASE.2018-06-09T03-43-35Z 2018-06-09 03:47:19 +00:00
Harshavardhana
371349787f Remove region requirement for Healing (#6031) 2018-06-08 17:54:57 -07:00
Nitish Tiwari
3dc13323e5 Use random host from among multiple hosts to create requests
Also use hosts passed to Minio startup command to populate IP
addresses if MINIO_PUBLIC_IPS is not set.
2018-06-08 10:22:01 -07:00
Nitish Tiwari
6ce7265c8c Add support for CopyObject across regions and multiple Minio IPs
This PR adds CopyObject support for objects residing in buckets
in different Minio instances (where Minio instances are part of
a federated setup).

Also, added support for multiple Minio domain IPs. This is required
for distributed deployments, where one deployment may have multiple
nodes, each with a different public IP.
2018-06-08 10:22:01 -07:00
Nitish Tiwari
f30c95a301 Add etcd handling for web-handler methods 2018-06-08 10:22:01 -07:00
Harshavardhana
481390d51a Converge etcd functionality as part of quick.Config 2018-06-08 10:22:01 -07:00
Nitish
6df1e4a529 Add functionality to add old buckets to etcd on startup
Buckets already present on a Minio server before it joins a
bucket federated deployment will now be added to etcd during
startup. In case of a bucket name collision, admin is informed
via Minio server console message.

Added configuration migration for configuration stored in etcd
backend.

Also, environment variables are updated and ListBucket path style
request is no longer forwarded.
2018-06-08 10:22:01 -07:00
Harshavardhana
853ea371ce Bring etcd support for bucket DNS federation
- Supports centralized `config.json`
- Supports centralized `bucket` service records
  for client lookups
- implement a new proxy forwarder
2018-06-08 10:22:01 -07:00
Minio Trusted
7872c192ec Update yaml files to latest version RELEASE.2018-06-08T03-49-38Z 2018-06-08 03:52:34 +00:00
Kaan Kabalak
39919708d6 Fix regression issue where users couldn't click into folders (#6029)
This commit fixes the issue introduced in #6023

The clicks on folder names direct the users inside the folders; while
the clicks on file names don't download the files, and will open file
previews on a separate modal in the future. Additionally, when a file
has been selected using the checkbox, it can now be downloaded without
being inside a zip file.

Fixes #6026
2018-06-07 20:43:51 -07:00
Minio Trusted
7e12c3e8b9 Update yaml files to latest version RELEASE.2018-06-07T19-10-07Z 2018-06-07 19:13:18 +00:00
Kaan Kabalak
94ec6f374e NewUX: On filename click, select file instead of downloading (#6023)
This commit has been done according to @abperiasamy's feedback as we
are going to reserve the click on file icon to open the preview modal
in the future.

Also, when the user now selects a single file, the file itself is
downloaded instead of a .zip file containing the file.

Fixes #6019
2018-06-07 12:02:02 -07:00
Harshavardhana
617a6d8e47 Add request tracing support for other REST APIs (#6013) 2018-06-07 10:41:13 -07:00
Praveen raj Mani
c0cfe21c00 Ignore region in the case of admin API (#5919)
Admin API is not an S3 API and hence it is not required
to honor server region while validating admin API calls.

Fixes #2411
2018-06-07 10:37:31 -07:00
Bala FA
6a8bfcef1c remove separate file for posix utils. (#5948) 2018-06-07 12:31:40 +05:30
Harshavardhana
6138cae8e7 Persist MINIO_WORM as part of config.json (#6022) 2018-06-06 18:10:51 -07:00
Krishna Srinivas
113570b514 Refresh in-memory bucket policy cache every 5 minutes (#6007) 2018-06-06 12:52:56 -07:00
Bala FA
6a53dd1701 Implement HTTP POST based RPC (#5840)
Added support for new RPC support using HTTP POST.  RPC's 
arguments and reply are Gob encoded and sent as HTTP 
request/response body.

This patch also removes Go RPC based implementation.
2018-06-06 14:21:56 +05:30
Harshavardhana
9d41051e91 Relax runtime requirement do not fail, print warning (#6017) 2018-06-06 09:25:12 +05:30
Harshavardhana
3143454982 Deprecate and remove configurable disk usage check (#6016) 2018-06-05 18:53:44 -07:00
Harshavardhana
eafc15cd47 Fix presigned URL for access key with special characters (#6012)
Fixes #6011
2018-06-05 10:48:51 -07:00
Harshavardhana
6fb0604502 Allow usage check to be configurable (#6006) 2018-06-04 18:35:41 -07:00
Harshavardhana
df1b33013f Fix byte pool usage, use only one pool for all sets. (#5990) 2018-06-01 16:41:23 -07:00
Harshavardhana
537fd8c821 Bring nicer ui error if creds are missing in Gateway (#6003) 2018-06-01 15:45:11 -07:00
Krishna Srinivas
69b9d6fbee Add user-agent for azure blob requests (#6002) 2018-06-01 15:44:48 -07:00
Harshavardhana
c22b9d5d4d Fix s3cmd issue with ACL handling (#5999)
With the implementation of dummy GET ACL handlers,
tools like s3cmd perform few operations which causes
the ACL call to be invoked. Make sure that in our
router configuration GET?acl comes before actual
GET call to facilitate this dummy call.
2018-05-31 19:43:50 -07:00
Harshavardhana
c247e603d2 terminate disk-usage routine in tests (#5998)
tests were written in the manner by editing internal
variables of fsObjects to mimic certain behavior from
APIs, but this is racy when an active go-routine is
reading from the same variable.

Make sure to terminate the go-routine if possible for
these tests.
2018-05-31 14:03:46 -07:00
Harshavardhana
74328c3061 Implement auto cert reloading (#5963) 2018-05-31 12:30:15 -07:00
Harshavardhana
487ecedc51 fix admin info peer to point to first endpoint (#5996)
The current problem is that when you invoke

```
mc admin info myminio | head -1
●  localhost:9000
```

This output is incorrect as the expected output should be
```
mc admin info myminio | head -1
●  192.168.1.17:9000
```
2018-05-31 08:52:21 +05:30
Andreas Auernhammer
9fb94e6aa8 ensure authenticated request bodies for Admin-API (#5984)
This commit adds a check to the server's admin-API such that it only
accepts Admin-API requests with authenticated bodies. Further this
commit updates the `madmin` package to always add the
`X-Amz-Content-Sha256` header.

This change improves the Admin-API security since the server does not
accept unauthenticated request bodies anymore.

After this commit `mc` must be updated to the new `madmin` api because
requests over TLS connections will fail.
2018-05-30 14:49:03 -07:00
Harshavardhana
5282639f3c Add prometheus endpoint to support total Used storageInfo (#5988)
Since we deprecated Total/Free we don't need to update
prometheus with those metrics. This PR also adds support
for caching implementation.
2018-05-30 11:30:14 -07:00
Arjun Mishra
dd0db526d9 Remove quotes for XSS Protection Header (#5992) 2018-05-30 14:59:27 +05:30
Yaroslav Skopets
a50cc7e937 Add Prometheus metrics for Minio gateway (#5987) 2018-05-30 10:13:46 +05:30
Praveen raj Mani
7ac0fccb6e updated pika API changes in rabbitMQ notification doc (#5980)
The pika python package used for listening on rabbitMQ events
has breaking change. 'type' param changed to 'exchange_type'
in newer releases.
2018-05-29 00:35:28 -07:00
Minio Trusted
3cdf601cf7 Update yaml files to latest version RELEASE.2018-05-25T19-49-13Z 2018-05-25 19:53:08 +00:00
Nitish Tiwari
5afd856355 Update Kubernetes TLS doc with info for distributed setups (#5971)
Also, add details on how to create wildcard self-signed certificates
using openssl
2018-05-23 20:41:25 -07:00
Harshavardhana
000e360196 Deprecate showing drive capacity and total free (#5976)
This addresses a situation that we shouldn't be
displaying Total/Free anymore, instead we should simply
show the total usage.
2018-05-23 17:30:25 -07:00
Harshavardhana
e6ec645035 Implement support for calculating disk usage per tenant (#5969)
Fixes #5961
2018-05-23 15:41:29 +05:30
Acid Chicken (硫酸鶏)
483fe4bed5 Fix typo (#5960) 2018-05-22 08:09:30 +05:30
Harshavardhana
71c4ff9d10 Fix tabs/indents in help message (#5962) 2018-05-21 11:11:57 -07:00
Harshavardhana
ac58283001 Support endpoints through env (#5952) 2018-05-18 17:51:03 -07:00
Andreas Auernhammer
9c8b7306f5 security: fix write-to-RAM DoS vulnerability (#5957)
This commit fixes a DoS vulnerability for certain APIs using
signature V4 by verifying the content-md5 and/or content-sha56 of
the request body in a streaming mode.

The issue was caused by reading the entire body of the request into
memory to verify the content-md5 or content-sha56 checksum if present.

The vulnerability could be exploited by either replaying a V4 request
(in the 15 min time frame) or sending a V4 presigned request with a
large body.
2018-05-18 11:27:25 -07:00
rawipfel
1cf381f1b0 handle Kubernetes read-only secrets (#5951) 2018-05-18 10:31:11 +05:30
kannappanr
ffa6b45d00 Remove treewalk abort log message (#5953)
Log doesn't add any value and can be removed.

Fixes #5943
2018-05-17 17:26:16 -07:00
Bala FA
4eb788df79 rename checkPathValid() to getValidPath() (#5949) 2018-05-17 07:27:07 -07:00
Minio Trusted
f9e8ac429e Update yaml files to latest version RELEASE.2018-05-16T23-35-33Z 2018-05-16 23:39:52 +00:00
Harshavardhana
64288d6eb5 fix: export policy.Resource for remote RPC policy updates (#5941)
This PR fixes a regression where the remote bucket policies
were not updated due to policy.Resource not having exported
fields, gob fails with unexported fields while marshalling.
2018-05-16 16:17:00 -07:00
Nitish Tiwari
41496e1406 Fix broken healthcheck link (#5935) 2018-05-16 14:43:25 -07:00
Nitish Tiwari
e6ebcc4cb6 Remove redundant prometheus data points (#5934)
Removed field minio_http_requests_total as it was redundant with
minio_http_requests_duration_seconds_count

Also removed field minio_server_start_time_seconds as it was
redundant with process_start_time_seconds
2018-05-15 12:23:43 -07:00
Nitish Tiwari
9cab0f25e0 Add top level metrics document to summarize monitoring endpoints (#5923)
Minio server supports healthcheck and prometheus related
unauthenticated endpoints. This document summarizes this
information in a single place and add links for more detailed
documentation if needed.
2018-05-15 12:23:21 -07:00
Minio Trusted
5c21e89559 Update yaml files to latest version RELEASE.2018-05-11T00-29-24Z 2018-05-11 00:34:18 +00:00
Harshavardhana
c872c30ea3 fix: introduce isLeafDir in healing to fix the crash (#5920)
This PR also supports healing directories.

Fixes #5917
2018-05-10 16:53:42 -07:00
Harshavardhana
5b74f918d4 Implement a dummy GetBucketACL for legacy reasons (#5906)
GetBucketACL call returns empty for all GET in ACL requests, 
the primary purpose of this PR is to provide legacy API support 
for legacy applications.

Fixes #5706
2018-05-10 09:32:26 +05:30
Minio Trusted
518f856900 Update yaml files to latest version RELEASE.2018-05-10T00-00-42Z 2018-05-10 00:03:30 +00:00
Krishna Srinivas
cc8178cdc4 Log errors only once for event notification errors (#5905) 2018-05-09 15:59:45 -07:00
Harshavardhana
4886bfbc72 fix: Avoid more crashes due to concurrent map usage (#5912)
This PR fixes another situation where a crash occurs
thanks to @krishnasrinivas for reproducing this

Fixes #5897
2018-05-09 15:11:51 -07:00
Anis Elleuch
c525424179 Return detailed msg for notification errors (#5913) 2018-05-09 15:11:24 -07:00
Anis Elleuch
c1f1fa3cc8 More fixes for error messages (#5911) 2018-05-09 13:29:37 -07:00
Harshavardhana
98f81ced86 fix: Avoid concurrent map writes in go-routines (#5898)
Fixes #5897
2018-05-09 11:25:38 -07:00
Anis Elleuch
2c0020e9ee Fix typo in prometheus scrape url (#5909) 2018-05-09 11:20:03 -07:00
Praveen raj Mani
3971483e18 Remove trailing/leading spaces in Certificates (#5885)
Fixes:  #5632
2018-05-09 14:26:38 +05:30
Ashish Kumar Sinha
deb685c5b5 Enhancements in Minio Prometheus exporter (#5848)
Standardized Minio collectors based on Prometheus 
recommendations.
2018-05-09 01:38:27 -07:00
Krishna Srinivas
bb34bd91f1 Fix unnecessary log messages to avoid flooding the logs (#5900) 2018-05-09 01:38:27 -07:00
Krishna Srinivas
9de8fefa90 Fix azure list-objects not to list minio.sys.tmp/ (#5895) 2018-05-09 01:38:27 -07:00
Anis Elleuch
6d5f2a4391 Better support of empty directories (#5890)
Better support of HEAD and listing of zero sized objects with trailing
slash (a.k.a empty directory). For that, isLeafDir function is added
to indicate if the specified object is an empty directory or not. Each
backend (xl, fs) has the responsibility to store that information.
Currently, in both of XL & FS, an empty directory is represented by
an empty directory in the backend.

isLeafDir() checks if the given path is an empty directory or not,
since dir listing is costly if the latter contains too many objects,
readDirN() is added in this PR to list only N number of entries.
In isLeadDir(), we will only list one entry to check if a directory
is empty or not.
2018-05-09 01:38:21 -07:00
Anis Elleuch
32700fca52 Enhance fatal errors printing of common issues seen by users (#5878) 2018-05-08 19:04:36 -07:00
Harshavardhana
54cd29b51c Add new S3 APIs which are not implemented (#5899)
Fixes #5896
2018-05-08 21:45:10 +05:30
Harshavardhana
c98d8cb1c7 fs: fix a regression allow reading of existing files (#5889) 2018-05-07 17:00:44 -07:00
Minio Trusted
2ecc976950 Update yaml files to latest version RELEASE.2018-05-04T23-13-12Z 2018-05-04 23:16:52 +00:00
Andreas Auernhammer
c5a00e513c fix DoS vulnerability in request authentication (#5887)
This commit fixes a DoS vulnerability in the
request authentication. The root cause is an 'unlimited'
read-into-RAM from the request body.

Since this read happens before the request authentication
is verified the vulnerability can be exploit without any
access privileges.

This commit limits the size of the request body to 3 MB.
This is about the same size as AWS. The limit seems to be
between 1.6 and 3.2 MB - depending on the AWS machine which
is handling the request.
2018-05-04 11:16:14 -07:00
Anis Elleuch
9439dfef64 Use defer style to stop tickers to avoid current/possible misuse (#5883)
This commit ensures that all tickers are stopped using defer ticker.Stop()
style. This will also fix one bug seen when a client starts to listen to
event notifications and that case will result a leak in tickers.
2018-05-04 10:43:20 -07:00
Krishna Srinivas
0f746a14a3 Do not use crypto.SHA3_256 as placeholder for HighwayHash256 (#5847) 2018-05-04 10:42:22 -07:00
Harshavardhana
d69ba7d085 fs: fix logging by not logging common scenarios (#5882)
Fixes #5881
2018-05-03 08:29:38 +05:30
Krishna Srinivas
d6df9b16ac Return NoSuchKey for anonReqs with s3:ListBucket policy (#5876) 2018-05-02 12:13:27 +05:30
Krishna Srinivas
0ad0969d61 Remove s3:ListObjects policy action to be in sync with AWS-S3 (#5875) 2018-05-02 12:13:17 +05:30
wd256
7f207156ec Copy an object without user-defined metadata for Azure gateway (#5872) 2018-05-02 10:20:00 +05:30
Andreas Auernhammer
ca8520fb31 vendor x/sys/cpu and update x/crypto/blake2b (#5870)
This change updates the vendor'd x/crypto/blake2b package and
adds x/sys/cpu.

This change fixes an issue for Go1.11 affecting x/crypto/blake2b
since G1.11 removes the runtime-internal functions `suports_avx()` and
`supports_avx2()`.
2018-05-01 11:16:56 +05:30
Harshavardhana
5f9041571f Heal only when atleast one of the disk is unformatted (#5866)
Current healing has an issue when disks are healed
even when they are offline without knowing if disk
is unformatted. This can lead to issues of pre-maturely
removing the disk from the set just because it was
temporarily offline.

There is an increasing number of `mc admin heal` usage
on a cron or regular basis. It is possible that if healing
code saw disk is offline it might prematurely take it down,
this causes availability issues.

Fixes #5826
2018-05-01 09:07:39 +05:30
cbenien
e43cd0d5c8 Increased MaxIdleConnsPerHost to prevent excessive re-connections and TIME_WAIT when more than 100 clients are using minio (#5860) 2018-04-30 11:59:41 -07:00
Egor Panfilov
4f68e0109c Fix output of AMQP example (#5867) 2018-04-30 08:39:07 +05:30
Minio Trusted
97c0c1abfd Update yaml files to latest version RELEASE.2018-04-27T23-33-52Z 2018-04-27 23:37:02 +00:00
Krishna Srinivas
9816264eed Support for ListObjectsV1 style marker for Azure gateway (#5856)
fixes #4948
2018-04-27 16:08:46 -07:00
Krishna Srinivas
11b8e292a7 Disable logging during unit tests (#5861) 2018-04-27 15:03:19 -07:00
Harshavardhana
b6ca39ea48 Support migrating inconsistent bucket policies (#5855)
Previously we used allow bucket policies without
`Version` field to be set to any given value, but
this behavior is inconsistent with AWS S3.

PR #5790 addressed this by making bucket policies
stricter and cleaner, but this causes a breaking
change causing any existing policies perhaps without
`Version` field or the field to be empty to fail upon
server startup.

This PR brings a code to migrate under these scenarios
as a one time operation.
2018-04-27 15:02:54 -07:00
Elson Rodriguez
1bd7eb979c Add example on how to deploy minio without a backing store. (#5810)
Many on-premise clusters do not have a PV abstraction, this example 
shows a way to deploy minio effectively in such environments.
2018-04-27 20:58:23 +05:30
poornas
0dc3d7ac18 fix error checks when cache is offline/missing. (#5850) 2018-04-26 10:39:05 +05:30
Krishna Srinivas
9aace6d36d Continue healing other objects even if objects without quorum exist (#5851)
fixes #5815
2018-04-25 11:56:39 -07:00
Krishna Srinivas
6831177394 Do not log errFileNotFound error (#5853) 2018-04-25 11:46:49 -07:00
kannappanr
fe126de98b Regenerate fs.json if it is corrupted in FS mode (#5778)
Also return a default e-tag for pre-existing objects.
Fixes #5712
2018-04-24 17:36:43 -07:00
Bala FA
0d52126023 Enhance policy handling to support SSE and WORM (#5790)
- remove old bucket policy handling
- add new policy handling
- add new policy handling unit tests

This patch brings support to bucket policy to have more control not
limiting to anonymous.  Bucket owner controls to allow/deny any rest
API.

For example server side encryption can be controlled by allowing
PUT/GET objects with encryptions including bucket owner.
2018-04-24 15:53:30 -07:00
Andreas Auernhammer
21a3c0f482 disable elliptic curves P-384 and P-521 for TLS. (#5845)
This change disables the non-constant-time implementations of P-384 and P-521.
As a consequence a client using just these curves cannot connect to the server.
This should be no real issues because (all) clients at least support P-256.

Further this change also rejects ECDSA private keys of P-384 and P-521.
While non-constant-time implementations for the ECDHE exchange don't expose an
obvious vulnerability, using P-384 or P-521 keys for the ECDSA signature may allow
pratical timing attacks.

Fixes #5844
2018-04-24 15:47:30 -07:00
Benjamin
c733fe87ce change MAINTAINER to LABEL (#5843) 2018-04-24 09:57:48 -07:00
Harshavardhana
ccdb7bc286 Fix s3 compatibility fixes for getBucketLocation,headBucket,deleteBucket (#5842)
- getBucketLocation
- headBucket
- deleteBucket

Should return 404 or NoSuchBucket even for invalid bucket names, invalid
bucket names are only validated during MakeBucket operation
2018-04-24 08:57:33 +05:30
Harshavardhana
954142a98f Cleanup and make a safer code (#5794) 2018-04-21 20:51:53 -07:00
Bala FA
76cc65531c use package name correctly (#5827) 2018-04-21 19:23:54 -07:00
ebozduman
f16bfda2f2 Remove panic() and handle it appropriately (#5807)
This is an effort to remove panic from the source. 
Add a new call called CriticialIf, that calls LogIf and exits. 
Replace panics with one of CriticalIf, FatalIf and a return of error.
2018-04-19 17:24:43 -07:00
wd256
846f3e8f59 Disable "chunked" uploading by the GCS client for objects smaller than the chunk size. (#5835)
By disabling chunked uploading when the object size is less than the chunk size,
memory is not allocated unnecessarily.
2018-04-19 16:15:14 -07:00
Minio Trusted
75cc2ce9d8 Update yaml files to latest version RELEASE.2018-04-19T22-54-58Z 2018-04-19 22:58:31 +00:00
Andreas Auernhammer
5a16671f72 do not set response header X-Frame-Options for S3 requests (#5838)
This change removes the X-Frame-Options header - It should
not be set for S3 requests since it can break CORS.

Fixes #5813
2018-04-19 11:46:37 -07:00
Ashish Kumar Sinha
9ebb72aa99 Introduce new unauthenticated endpoint /metric (#5723) (#5829)
/metric exposes Promethus compatible data for scraping metrics

Fixes: #5723
2018-04-18 16:01:42 -07:00
wd256
9f31da5d57 Fix PutObject/CopyObject with metadata for GCS gateway (#5828)
Make sure to apply standard headers such as Content-Type,
Content-Disposition and Content-Language to the correct
GCS object attributes during object upload and copy operations.

Fixes: #5800
2018-04-18 21:40:48 +05:30
Harshavardhana
0e9c73cbcf update browser ui-assets (#5831) 2018-04-17 09:31:55 +05:30
Harshavardhana
c332b4fb5e fix shadowing issue in logger (#5830) 2018-04-16 13:36:45 -07:00
Rushan
ed5f2d899d Browser: Disable input validation styles on Firefox (#5822) 2018-04-16 10:47:03 -07:00
Harshavardhana
adf9a9d300 Remove all unused variables and functions (#5823) 2018-04-15 19:26:04 +05:30
Harshavardhana
97a8d856b6 Vendorize new changes from minio-go repo (#5821)
- When policy is empty delete the bucket policy (#966) (04/13/18) <Harshavardhana>
- Add tests to check if ListObjects/V2 returns expected StorageClass (#963) (04/10/18) <Nitish Tiwari>
- Update get/setBucketPolicy methods to use files instead of pkg/policy (#959) (04/10/18) <Nitish Tiwari>
- avoid unnecessary stat call during single copy (#962) (04/06/18) <Andreas Auernhammer>
- avoid sending SSE-S3 header during GET requests. (#965) (04/05/18) <Andreas Auernhammer>
- Fix stream SSE uploads with S3 encrypt type (#960) (04/02/18) <Jesús Espino>
- Fix xml parsing error for RemoveObjects API (#949) (03/29/18) <poornas>
- Allow to upload empty files in stream based uploads (#958) (03/26/18) <Jesús Espino>
- Add missing doneCh in the example for removeobjects (#955) (03/26/18) <Alexandr Korsak>
- tests: Remove partial related tests (#957) (03/26/18) <Anis Elleuch>
- Add transport connection broken error to retry list (#956) (03/19/18) <poornas>
- [refactor]: simplify client encryption examples (#952) (03/19/18) <Andreas Auernhammer>
- Add tests for putObjectContentLanguage (#950) (03/15/18) <Harshavardhana>
- Add putObject/getObject() client side encryption examples (#948) (03/13/18) <Harshavardhana>
2018-04-15 11:39:48 +05:30
Nitish Tiwari
638f01f9e4 Generalize loadConfig method to avoid reading from disk (#5819)
As we move to multiple config backends like local disk and etcd,
config file should not be read from the disk, instead the quick
package should load and verify for duplicate entries.
2018-04-13 15:14:19 -07:00
Harshavardhana
57b8db2088 Cleanup help to bring concise self-help message (#5788) 2018-04-13 11:57:05 -07:00
Harshavardhana
bcd54632f5 Fix broken 'minio update' command on Windows (#5816) 2018-04-13 11:51:03 -07:00
Nitish Tiwari
42f775af4f Add cache config for configuration comparison (#5812) 2018-04-13 23:45:19 +05:30
Minio Trusted
75f35b926d Update yaml files to latest version RELEASE.2018-04-12T23-41-09Z 2018-04-12 23:45:16 +00:00
Andreas Auernhammer
f60765ac93 add some security HTTP headers (#5814)
This change adds some security headers like Content-Security-Policy.
It does not set the HSTS header because Content-Security-Policy prevents
mixed HTTP and HTTPS content and the server does not use cookies.
However it is a header which could be added later on.

It also moves some header added by #5805 from a vendored file
to a generic handler.

Fixes ##5813
2018-04-12 15:57:41 -07:00
Harshavardhana
1f07545e2a Improve init messages for distributed setup (#5786)
Fixes #5531
2018-04-12 15:43:38 -07:00
rwagner_inf
e39de65367 Add security HTTP Headers (#5805)
Some HTTP security headers in Minio.
To avoid problems with XSS and Clickjacking attacks.

X-Frame-Options
X-Frame-Options response header improve the protection
of web applications against Clickjacking. It declares a
policy communicated from a host to the client browser
on whether the browser must not display the transmitted
content in frames of other web pages.

X-XSS-Protection
This header enables the Cross-site scripting (XSS) filter in your browser.
2018-04-12 14:30:06 -07:00
Harshavardhana
bd8d6e3c4e Return bucketNameInvalid properly in B2 gateway (#5787)
Fixes #5758
2018-04-12 11:20:21 +05:30
Harshavardhana
4a874dfbc1 Ignore prefix renames when dest directory is not empty (#5798)
Also make sure to not modify the underlying errors from
layers, we should return the error as is and one object
layer should translate the errors.

Fixes #5797
2018-04-11 17:15:42 -07:00
Bala FA
650c6ee8fb event/target/webhook: treat all 2xx as success. (#5792)
Fixes #5769
2018-04-10 17:45:54 -07:00
kannappanr
57a3d9c16c Modify fatalIf, startup and update message logging code (#5780)
Use a common logging framework to log fatalIf, startup, Info and Update
messages.
2018-04-10 09:37:14 -07:00
kannappanr
cef992a395 Remove error package and cause functions (#5784) 2018-04-10 09:36:37 -07:00
Harshavardhana
217fb470a7 Add a check to check if disk is writable (#5662)
This check is a pre-emptive check to return
error early before we attempt to use the disk
for any other operations later.

refer #5645
2018-04-10 09:26:09 +05:30
Harshavardhana
eb0deabd73 Fail when TLS is configured and HTTP endpoints are provided (#5771) 2018-04-10 07:40:23 +05:30
dingjs
a706c21f70 Tranlsate disk-caching/README.md,nas.md,large-bucket/README.md to Chinese. (#5775) 2018-04-09 17:41:16 -07:00
Harshavardhana
41c31d10ca Enable s3,gcs,manta,oss,b2 gateways as production ready (#5795) 2018-04-09 16:33:48 -07:00
Harshavardhana
1d31ad499f Make sure to re-load reference format after HealFormat (#5772)
This PR introduces ReloadFormat API call at objectlayer
to facilitate this. Previously we repurposed HealFormat
but we never ended up updating our reference format on
peers.

Fixes #5700
2018-04-09 22:55:41 +05:30
Krishna Srinivas
ae8e863ff4 disk.String() represents just path and not URL when disk is a local disk (#5785) 2018-04-06 16:59:31 -07:00
Andreas Auernhammer
da9f0e324e return AWS S3 compatible error for invalid but equal keys during key rotation (#5783)
This change let the server return the S3 error for a key rotation
if the source key is not valid but equal to the destination key.

This change also fixes the SSE-C error messages since AWS returns error messages
ending with a '.'.

Fixes #5625
2018-04-06 14:15:23 -07:00
Harshavardhana
73f7a98590 Enable go1.10.1 version (#5666) 2018-04-06 11:07:39 -07:00
Andreas Auernhammer
2f20d90465 set storage class in FS mode (#5779)
This change sets the storage class of the object-info if a storage
class was specified during PUT. The server now replies with the
storage class which was set during uploading the object in FS mode.

Fixes #5777
2018-04-06 20:50:02 +05:30
Dee Koder
c30f75de28 Update screenshots. Use png instead of jpgs. (#5770) 2018-04-06 13:51:25 +05:30
Harshavardhana
861550446a Allow upto 15 minutes of timeskew for RPC calls as well (#5774)
Default installations of cloned VMs on VMware like env
might experience serious problems with time skewing,
allow for a higher value instead of 3 seconds we are
moving to 15 minutes just like API level skew.

Access to internet and configuring ntp might not be possible,
in such situations providing atleast a 15 minute skew could
cater for majority of situations.
2018-04-05 23:55:31 -07:00
kannappanr
f8a3fd0c2a Create logger package and rename errorIf to LogIf (#5678)
Removing message from error logging
Replace errors.Trace with LogIf
2018-04-05 15:04:40 -07:00
kannappanr
91fd8ffeb7 Move http package inside cmd (#5776) 2018-04-05 14:19:27 -07:00
Nitish Tiwari
9eb94fe8c8 Fix StorageClass field in ListObject/ListObjectV2 response (#5766)
Fixes: #5754
2018-04-05 10:56:28 -07:00
Harshavardhana
30ab71c9d9 Add missing healthcheck router for gateway (#5764) 2018-04-05 07:37:54 +05:30
Kaan Kabalak
38ce19badd Rename minio-limitations.md to minio-limits.md in Chinese translation (#5753)
Applies the change in #5541 for Chinese docs
2018-04-04 19:00:19 -07:00
Minio Trusted
c4a70fdee9 Update yaml files to latest version RELEASE.2018-04-04T05-20-54Z 2018-04-04 05:25:19 +00:00
Harshavardhana
85a57d2021 Make sure to close the disk connections (#5752)
Since we do not re-use storageDisks after moving
the connections to object layer we should close them
appropriately otherwise we have a lot of connection
leaks and these can compound as the time goes by.

This PR also refactors the initialization code to
re-use storageDisks for given set of endpoints until
we have confirmed a valid reference format.
2018-04-04 10:28:48 +05:30
Nitish Tiwari
d67e423a32 Add firewall-cmd entry for firewall configuration (#5755) 2018-04-03 17:15:24 -07:00
Edward Vielmetti
e76fe435c4 "account_id" for "accound_id" (#5757) 2018-04-03 17:06:59 -07:00
Vladimir Cores
1a71004c91 Multi files selection and upload from Browser (#5711) 2018-04-03 16:44:37 -07:00
Edward Vielmetti
1e50951a4a typo fix (#5756) 2018-04-03 21:00:35 +05:30
Harshavardhana
2c5f2e9669 Stop deleting 'format.json' upon unsuccessful save (#5747)
An issue was reproduced when there a no more inodes
available on an existing setup of 4 disks, now we
took one of the disks and reformatted it to relinquish
inodes. Now we attempt to bring the fresh disk back
into setup and perform a heal - at this point creating
new `format.json` fails on existing disks since they
do not have more inodes available.

At this point due to quorum failure, we end up deleting
existing `format.json` as well, this PR removes the code
which deletes existing `format.json` as there is no need
to delete them.
2018-04-03 10:48:06 +05:30
Harshavardhana
b052bfa79e Add necessary safe headers under CORS (#5750)
Fixes #5748
2018-04-02 17:51:32 -07:00
Harshavardhana
8eb838bf91 Extend quorum based listing for prefixes (#5749)
Previous PR 2afd196c83 fixed
the issue of quorum based listing for regular objects, this
PR continues on this idea by extending this support to
object directory prefixes as well.

Fixes #5733
2018-04-02 17:26:34 -07:00
poornas
98315b8e60 Update disk cache documentation (#5743) 2018-04-02 10:25:08 +05:30
kannappanr
b87cc3d643 Get proper GOPATH in trim function (#5744)
Set GOPATH string to empty in build-constants.go
Check for both compile time GOPATH and default GOPATH
while trimming the file path in the stack trace.
Fixes #5741
2018-03-30 19:13:25 -07:00
Harshavardhana
4d02f9bccf fix gateway docs comprehensively (#5740) 2018-03-30 14:16:06 +05:30
Minio Trusted
cf8d12d169 Update yaml files to latest version RELEASE.2018-03-30T00-38-44Z 2018-03-30 00:41:38 +00:00
Krishna Srinivas
804a4f9c15 Fix backend format for disk-cache - not to use FS format.json (#5732) 2018-03-29 14:38:26 -07:00
Nitish Tiwari
328076f773 Fix help message to be uniform across env vars (#5735) 2018-03-30 01:59:21 +05:30
Nitish Tiwari
22e3ace36e Apply storageClass data while config migration (#5737)
Fixes #5736
2018-03-29 13:21:58 -07:00
Harshavardhana
ef61b36c5a Fix PUT bucket notification deadlocks (#5734)
This PR fixes two different variant of deadlocks in
notification.

- holding write lock on the bucket competing with read lock
- holding competing locks on read/save notification config
2018-03-29 12:00:20 -07:00
Harshavardhana
35b3913d22 Fix block id length upto 8bytes only for azure gateway. (#5731)
This PR also reverts commit 2f9354b17e
to bring back 8 byte block id requirement for azure gateway.
2018-03-29 09:54:47 -07:00
Harshavardhana
228c8f05f4 Improve some wording in disk caching docs (#5730) 2018-03-29 10:12:14 +05:30
poornas
af024a9c69 Remove deadcode related to multipart cleanup for fs (#5716)
The cleanup code is no longer needed as we moved to lockfree 
multipart backend for fs
2018-03-29 08:26:52 +05:30
Minio Trusted
5a7ab7b06e Update yaml files to latest version RELEASE.2018-03-28T23-45-53Z 2018-03-28 23:48:23 +00:00
poornas
5006e28367 Add disk-cache and worm to user-agent (#5729) 2018-03-28 16:26:27 -07:00
poornas
a3e806ed61 Add disk based edge caching support. (#5182)
This PR adds disk based edge caching support for minio server.

Cache settings can be configured in config.json to take list of disk drives,
cache expiry in days and file patterns to exclude from cache or via environment
variables MINIO_CACHE_DRIVES, MINIO_CACHE_EXCLUDE and MINIO_CACHE_EXPIRY

Design assumes that Atime support is enabled and the list of cache drives is
fixed.
 - Objects are cached on both GET and PUT/POST operations.
 - Expiry is used as hint to evict older entries from cache, or if 80% of cache
   capacity is filled.
 - When object storage backend is down, GET, LIST and HEAD operations fetch
   object seamlessly from cache.

Current Limitations
 - Bucket policies are not cached, so anonymous operations are not supported in
   offline mode.
 - Objects are distributed using deterministic hashing among list of cache
   drives specified.If one or more drives go offline, or cache drive
   configuration is altered - performance could degrade to linear lookup.

Fixes #4026
2018-03-28 14:14:06 -07:00
Harshavardhana
2f9354b17e Use uuid.New() for uploadID in azure gateway (#5721) 2018-03-28 11:17:25 -07:00
Harshavardhana
6e9c853312 After healing re-load disks with the new format (#5718)
This PR also fixes correct calculation of drive states
before and after healing of objects.

Fixes #5700
Fixes #5708
2018-03-28 06:41:39 +05:30
poornas
76d1e8bbcd change fs.json format to include checksum fields (#5685) 2018-03-27 17:23:10 -07:00
Bala FA
3ebe61abdf Quick support to server level WORM (#5602)
This is a trival fix to support server level WORM.  The feature comes
with an environment variable `MINIO_WORM`.

Usage:
```
$ export MINIO_WORM=on
$ minio server endpoint
```
2018-03-27 16:44:45 -07:00
Takeshi Watanabe
2182c1a4f7 Use paho configuration API instead of setting it directly with struct initializer. (#5707) 2018-03-26 16:45:21 -07:00
Nitish Tiwari
58291acb8d Fix object deletion where quorum is not available (#5709)
Object deletion should not be possible if quorum is not
available. This PR updates deleteObject() to check for
quorum errors before proceeding with object deletion.

Fixes #5535
2018-03-26 16:39:28 -07:00
Harshavardhana
de44be86d0 Use readQuorum instead of writeQuorum to check bucket exists (#5715)
Fixes #5708
Fixes #5700
2018-03-26 16:36:57 -07:00
Kanagaraj M
19451e374a revert browser newux changes (#5714) 2018-03-26 12:49:12 -07:00
Harshavardhana
35e64573fa Add missing env doc for MINIO_DOMAIN (#5705) 2018-03-26 09:11:39 -07:00
kannappanr
7c747a9643 Return complete Location URL in CompleteMultipartUpload (#5692)
Remove getLocation function.
Fixes #5687
2018-03-23 13:46:57 -07:00
Kanagaraj M
1c91c7b54d added eslint prettier check to travis (#5701)
- "yarn eslint" will throw the error if there are any formatting issues
- "yarn format" will format the js file based on the project's
prettier configuration.
2018-03-23 11:41:10 -07:00
Harshavardhana
020cc8b669 Remove old unmaintained docs (#5699)
- removes backend/{fs,xl} files.
- removes FreeBSD doc since we already have FreeNAS doc.
- removes sets/ directory which is redundant with
  large-bucket directory.
- removes browser/ directory which attempts
  to explain about browser RPC calls, since
  its an internal API we do not need to document it.
2018-03-24 00:07:04 +05:30
Nitish Tiwari
850caaa1be Add iptables/ufw command example to open relevant ports for Minio (#5604)
fixes #5022
2018-03-24 00:00:47 +05:30
Kanagaraj M
4ca0ec4586 added yarn test to travis.yml (#5696) 2018-03-23 14:36:20 +05:30
Kanagaraj M
85154508b0 updated favicon.ico and ui-assets.go (#5698) 2018-03-23 14:06:18 +05:30
Rushan
de84f41bec Fix the styling of upload abort button (#5697) 2018-03-23 13:04:39 +05:30
Kaan Kabalak
3805a055ab Fix New UX related bugs for Alerts, BucketList and Abort Upload modal (#5694)
- Center text on Alert component appearing on top
- Change "Upload" to "Continue" on New UX Abort Upload modal
- Fix issue where long bucket names were overflowing instead of
continuing on the next line
- Fix word wrap issue similar to the last one for the alerts that appear
on top
- Help fix the cluttering issue on #5690 as the right margin has been
increased in alerts

Fixes #5689, #5691 and #5695
2018-03-23 12:19:27 +05:30
Harshavardhana
17409517ec Fix docs to expand on new limits on a distributed setup (#5677) 2018-03-22 16:24:14 -07:00
Nitish Tiwari
cf67f1838e Add unit tests for endpoint-ellipses.go (#5681) 2018-03-22 16:23:54 -07:00
Harshavardhana
787f158f92 update browser assets for newux 2018-03-22 13:30:39 -07:00
Kanagaraj M
c0e45f9098 Fix failing unit tests in browser (#5688)
* format js files using prettier

Used the following command to format the files
prettier --write "browser/app/js/**/*.js"

* fix failing unit tests in browser
2018-03-22 12:25:56 -07:00
Kanagaraj M
cb3818be27 adding local pagination to bucket list (#5684)
* adding local pagination to bucket list

When there are more than 5000 buckets, browser ui
becomes unresponsive since react needs to create
5000 elements which takes browser resources.
So we show only 100 buckets for the first time,
and load more buckets when the user is scrolling down.

* move inline styles to less file
2018-03-22 23:28:48 +05:30
Rushan
1459c4be1e Browser: Update UI with new components and elements (#5671) 2018-03-21 11:38:57 -07:00
Kaan Kabalak
384b4fdf28 Tweak fetchBuckets to comply with the changes made to the routes (#5592)
This commit removes the argument in the fetchBuckets function that was
introduced in #5580. Due to the adjustment made in #5584, we no longer
need to differentiate between where the function has been called from.
2018-03-21 11:38:56 -07:00
Kanagaraj M
6a15b89b9a refactor Dropzone component and added tests (#5578) 2018-03-21 11:38:56 -07:00
Kanagaraj M
54e5ee6535 removed duplicate route components from browser (#5584)
All routes '/', '/:bucket/', '/:bucket/*' render the same
component. Instead we could just have a single route like following which
combines all the above routes

'/:bucket?/*'

bucket is optional here, so it can cover '/'
2018-03-21 11:38:56 -07:00
Kaan Kabalak
a6adef0bdf Refactor bucket delete and bucket policy (#5580)
This commit adds the bucket delete and bucket policy functionalities
to the browser.

Part of rewriting the browser code to follow best practices and
guidelines of React (issues #5409 and #5410)

The backend code has been modified by @krishnasrinivas to prevent
issue #4498 from occuring. The relevant changes have been made to the
code according to the latest commit and the unit tests in the backend.
This commit also addresses issue #5449.
2018-03-21 11:38:56 -07:00
Kanagaraj M
416841869a allow non-loggedin users to access public bucket (#5570)
* conditionally render main action buttons

- Make bucket action will be available only for loggedIn users
- File upload button will be avaialble for loggedIn users
  and non-loggedIn users if the prefix is writable

* select the bucket and prefix from the url

When the url contains bucket and prefix, it will be selected
by default instead of the first bucket from the list.

* show BucketSearch only for LoggedIn users

* allow non-LoggedIn users to access public bucket

* removed unused Router imports

* fix test case failures in BucketList.test.js

* remove dupicate minioBrowserPrefix from url

since history is already initialized with minioBrowserPrefix,
no need to use it in push or replace

* remove unused match from App component

* remove unused minioBrowserPrefix imports
2018-03-21 11:30:50 -07:00
Kanagaraj M
bb0adea494 Refactor browser dropdown links and components (#5562)
* refactor browser links and about modal

Moved about modal to separate component and added unit tests.

* refactor change password modal component

* added unit tests for ChangePasswordModal

* fix logout function in browser dropdown

* remove older unused BrowserDropdown component

* remove unused variables from BrowserDropdown component

* show BrowserDropdown and StorageInfo only for LoggedIn users

Non-loggedIn users will see a 'Login' button
2018-03-21 11:30:50 -07:00
Kanagaraj M
6a42727e00 Refactor download object and bulk action components (#5546) 2018-03-21 11:30:50 -07:00
Kanagaraj M
da4558a8f7 Refactor delete object and share object components (#5537) 2018-03-21 11:30:50 -07:00
Kaan Kabalak
566ac78b8b Add missing unit tests for PR #5499 (#5527)
Part of #5410
2018-03-21 11:30:50 -07:00
Kanagaraj M
44f8f7059c Refactor make bucket and upload components (#5521) 2018-03-21 11:30:50 -07:00
Kanagaraj M
9bfa07ecf5 re-organize components and actions by feature (#5518)
Now the files grouped based on the features instead of
the previous approach of grouping by type.
2018-03-21 11:30:50 -07:00
Kaan Kabalak
ead6337eab Add missing unit tests to JavaScript and React components (#5505)
Add the missing unit tests login, bucket listing, and bucket searching functionalities.
2018-03-21 11:30:50 -07:00
Kanagaraj M
feb726dd98 Refactor of components (#5499) 2018-03-21 11:30:50 -07:00
Kanagaraj M
9ab6a8035f upgrading react and refactoring components (#5409) (#5444)
- upgraded react from v16.2.0
- upgraded react-router to v4.2.0 and re-writen the routes
- using prettier to format the code
- added jest to unit test components/reducers/selectors

This provides a skeleton to start of with. Only basic unit test
cases are added, remaining needs to be added.

In terms of functionality, it provides login, listing and searching
buckets. Remaining functionalities will be added in upcoming patches.
2018-03-21 11:30:50 -07:00
Kanagaraj M
76be54551f support for sourcemaps in browser code (#5451)
changed from 'eval' to 'cheap-module-eval-source-map' to 
generate source map of the original code instead of transpiled code.
2018-03-21 11:30:50 -07:00
Kaan Kabalak
0da377f3e9 Upgrade webpack from v1 to v3 (#5448)
- Changed webpack.config.js and webpack.production.config.js
according to the migration guidelines
- Updated packages that had unmet peer dependencies after the 
webpack update

Fixes #5398
2018-03-21 11:30:50 -07:00
Kaan Kabalak
9cc77e51f5 Change name of 'Settings' dropdown link to 'Change Password' (#5394) 2018-03-21 11:30:50 -07:00
Kaan Kabalak
bbbbb8d3e2 Center dropdown toggle correctly across browsers (#5395)
This commit also fixes the shifting issue for Firefox when the dropdown
toggle is clicked.
2018-03-21 11:30:50 -07:00
Harshavardhana
565dc8b283 Do not check for GOPATH env for builds (#5628)
Since go1.8 GOPATH is not required to set prior, as
it defaults to "${HOME}/go" we only need to check if
go tool detected GOPATH correctly. If yes then we
proceed if not we fail.
2018-03-20 12:05:49 -07:00
Harshavardhana
c726145baf Fix regression in removing notification (#5673)
fixes a regression introduced in 0e4431725c
when removing a previously applied notification configuration.

event.ParseConfig() was stricter in terms of handling notification
configuration, we need to allow when notification configuration is
sent empty, this is the way to remove notification configuration.
2018-03-20 12:02:56 -07:00
Harshavardhana
b4ae2bd2f5 Remove snap reference from the repo (#5676)
snap builds are not well tested, since
some releases have stopped working. Remove
their reference to keep it simple.

Fixes #5674
Fixes #5472
2018-03-20 09:33:02 +05:30
Minio Trusted
fb3159277c Update yaml files to latest version RELEASE.2018-03-19T19-22-06Z 2018-03-19 19:25:23 +00:00
Andreas Auernhammer
a4d2e2b428 [refactor] simplify en/decrypted size computation (#5658)
This commit replaces the en/decrypted size computation
with functions from the `sio` package.

Fixes #5657
2018-03-19 22:18:12 +05:30
Harshavardhana
2938e332ba Fix format migration regression (#5668)
Migration regression got introduced in 9083bc152e
adding more unit tests to catch this scenario, we need to fix this by
re-writing the formats after the migration to 'V3'.

This bug only happens when a user is migrating directly from V1 to V3,
not from V1 to V2 and V2 to V3.

Added additional unit tests to cover these situations as well.

Fixes #5667
2018-03-19 21:43:00 +05:30
Minio Trusted
049090126e Update yaml files to latest version RELEASE.2018-03-16T22-52-12Z 2018-03-16 22:55:04 +00:00
Harshavardhana
f23944aed7 Fix heal bucket deadlock after replacing disks (#5661)
Fixes #5659
2018-03-16 15:09:31 -07:00
Harshavardhana
3145462ad2 Return InvalidDigest when md5 sent by client is invalid (#5654)
This is to ensure proper compatibility with AWS S3, handle
special cases where

- Content-Md5 is set to empty
- Content-Md5 is set to invalid
2018-03-16 11:22:34 -07:00
Krishna Srinivas
9ede179a21 Use context.Background() instead of nil
Rename Context[Get|Set] -> [Get|Set]Context
2018-03-15 16:28:25 -07:00
Ersan
33fe42df8a Adds context support for nas, oss and s3 2018-03-15 16:28:25 -07:00
kannappanr
bdb1a90720 Add context support for gateway b2, manta, gcs 2018-03-15 16:28:25 -07:00
Krishna Srinivas
e452377b24 Add context to the object-interface methods.
Make necessary changes to xl fs azure sia
2018-03-15 16:28:25 -07:00
Krishna Srinivas
9083bc152e Flat multipart backend implementation for Erasure backend (#5447) 2018-03-15 13:55:23 -07:00
Bala FA
0e4431725c make notification as separate package (#5294)
* Remove old notification files

* Add net package

* Add event package

* Modify minio to take new notification system
2018-03-15 13:03:41 -07:00
Nitish Tiwari
abffa00b76 Update healthcheck related examples and add head support (#5650)
- Add head method for healthcheck endpoint. Some platforms/users
may use the HTTP Head method to check for health status.

- Add liveness and readiness probe examples in Kubernetes yaml
example docs. Note that readiness probe not added to StatefulSet
example due to https://github.com/kubernetes/kubernetes/issues/27114
2018-03-15 09:55:02 +05:30
Nitish Tiwari
1c0c3369c9 Cleanup large bucket support documents (#5641) 2018-03-14 15:22:17 -07:00
Harshavardhana
4af89543cf Update minio-go dependencies to latest 5.0.0 release (#5640)
With following changes

- Add SSE and refactor encryption API (#942) <Andreas Auernhammer>
- add copyObject test changing metadata and preserving etag (#944) <Harshavardhana>
- Add SSE-C tests for multipart, copy, get range operations (#941) <Harshavardhana>
- Removing conditional check for notificationInfoCh in api-notication (#940) <Matthew Magaldi>
- Honor prefix parameter in ListBucketPolicies API (#929) <kannappanr>
- test for empty objects uploaded with SSE-C headers (#927) <kannappanr>
- Encryption headers should also be set during initMultipart (#930) <Harshavardhana>
- Add support for Content-Language metadata header (#928) <kannappanr>
- Fix check for duplicate notification configuration entries (#917) <kannappanr>
- allow OS to cleanup sockets in TIME_WAIT (#925) <Harshavardhana>
- Sign V2: Fix signature calculation in virtual host style (#921) <A. Elleuch>
- bucket policy: Support json string in Principal field (#919) <A. Elleuch>
- Fix copyobject failure for empty files (#918) <kannappanr>
- Add new constructor NewWithOptions to SDK (#915) <poornas>
- Support redirect headers to sign again with new Host header. (#829) <Harshavardhana>
- Fail in PutObject if invalid user metadata is passed <Harshavadhana>
- PutObjectOptions Header: Don't include invalid header <Isaac Hess>
- increase max retry count to 10 (#913) <poornas>
- Add new regions for Paris and China west. (#905) <Harshavardhana>
- fix s3signer to use req.Host header (#899) <Bartłomiej Nogaś>
2018-03-14 19:38:29 +05:30
Harshavardhana
5dc5e4928c Support content-language header (#5648) 2018-03-14 15:27:32 +05:30
tparker00
bbd7f8a4af Updated healthcheck file (#5647)
Healthcheck is no longer checking for a file so the -I in the curl check will fail.
2018-03-14 13:21:56 +05:30
Nitish Tiwari
6882467db8 Update volume in Dockerfiles (#5646)
All sample commands in Minio docs use the volume path as /data.
Updated the dockerfiles to follow the same pattern.

Fixes #5635
2018-03-14 06:59:27 +05:30
Yueh-Ting Chen
e8d8360d00 fix spelling (#5643) 2018-03-13 19:23:37 +05:30
Alexander Stuart-Kregor
5053f96274 Swap max_hw_sectors_kb for max_sectors_kb. Fixes #5633. (#5638) 2018-03-13 18:47:49 +05:30
Minio Trusted
16000255f8 Update yaml files to latest version RELEASE.2018-03-12T21-25-28Z 2018-03-12 21:28:21 +00:00
Nitish Tiwari
4a4d1d1b82 Add Minio TLS configuration doc for Kubernetes deployment (#5617) 2018-03-12 14:22:23 -07:00
Harshavardhana
29ef7d29e4 Fix deadlock in in-place CopyObject decryption/encryption (#5637)
In-place decryption/encryption already holds write
locks on them, attempting to acquire a read lock would
fail.
2018-03-12 13:52:38 -07:00
Nitish Tiwari
574b667c56 Remove madmin docs from top level docs directory (#5636)
madmin package is well documented in its source directory here
https://github.com/minio/minio/tree/master/pkg/madmin.

Hence, keeping another copy is not required as it makes it difficult
to maintain.
2018-03-12 11:51:58 -07:00
Nitish Tiwari
10b01ac836 Add healthcheck endpoints (#5543)
This PR adds readiness and liveness endpoints to probe Minio server
instance health. Endpoints can only be accessed without authentication
and the paths are /minio/health/live and /minio/health/ready for
liveness and readiness respectively.

The new healthcheck liveness endpoint is used for Docker healthcheck
now.

Fixes #5357
Fixes #5514
2018-03-12 11:46:53 +05:30
Harshavardhana
d90985b6d8 Return authHeaderMalformed for an incorrect region in signature (#5618) 2018-03-09 18:18:57 -08:00
Harshavardhana
7aaf01eb74 Save ETag when updating metadata (#5626)
Fixes #5622
2018-03-09 10:50:39 -08:00
Nitish Tiwari
ba0c7544ea Cleanup orchestration documents (#5623)
- Remove hostPort from Kubernetes deployment example docs. Initially
hostPort was added to ensure Minio pods are allocated to separate
machines, but as per latest Kubernetes documents this is not
recommended approach (ref: https://kubernetes.io/docs/concepts/
configuration/overview/#services). To define pod allocations,
Affinity and Anti-Affinity concepts are the recommended approach.
(ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node)

- Add Minio release tag to Docker-Compose example file.
2018-03-09 15:21:41 +05:30
kannappanr
380e0ddb57 Remove unwanted errorIf calls (#5621)
Remove errorIf call with a errSignatureMismatch error
2018-03-09 00:51:05 -08:00
Andreas Auernhammer
889dd387f1 [doc] fix openssl command for ECDSA key generation (#5616)
This change fixes the command for generating ECDSA private keys.
The current command produces private key files which cannot be parsed
by the server.

Fixes #5614
2018-03-08 15:06:42 -08:00
Dee Koder
973ff2fabd Fix mqtt example py code which was not working (#5619) 2018-03-08 16:46:52 +05:30
poornas
247c1bb5ef Pass location in MakeBucketWithLocation call (#5605)
fixes regression for gateways
2018-03-08 12:56:20 +05:30
Harshavardhana
27258b9c54 Ensure to load only regular files for CAs (#5612)
In kubernetes statefulset like environments when secrets
are mounted to pods they have sub-directories, we should
ideally be only looking for regular files here and skip
all others.
2018-03-07 22:16:28 +05:30
Harshavardhana
b325593b47 SSE-C CopyObject key-rotation doesn't need metadata REPLACE value (#5611)
Fix a compatibility issue with AWS S3 where to do key rotation
we need to replace an existing object's metadata. In such a
scenario "REPLACE" metadata directive is not necessary.
2018-03-06 16:04:48 -08:00
Dee Koder
a34901af77 Add missing wget command for easy cut copy paste. (#5609)
We should keep this for easy one click copy paste on remote servers 
rather than 2 things for users to deal with.
2018-03-06 13:36:48 +05:30
Anis Elleuch
cac10bcbf7 SSE-C: Add support in Bucket Post Policy (#5607)
* SSE-C: Add support in Bucket Post Policy

* Rename isSSECustomerRequest & isSSECopyCustomerRequest to hasSSECustomerHeader hasSSECopyCustomerHeader
2018-03-05 08:02:56 -08:00
Aditya Manthramurthy
ea8973b7d7 Return bit-rot verified data instead of re-reading from disk (#5568)
- Data from disk was being read after bitrot verification to return
  data for GetObject. Strictly speaking this does not guarantee bitrot
  protection, as disks may return bad data even temporarily.

- This fix reads data from disk, verifies data for bitrot and then
  returns data to the client directly.
2018-03-04 14:16:45 -08:00
Harshavardhana
52eea7b9c1 Support SSE-C multipart source objects in CopyObject (#5603)
Current code didn't implement the logic to support
decrypting encrypted multiple parts, this PR fixes
by supporting copying encrypted multipart objects.
2018-03-02 17:24:02 -08:00
Harshavardhana
e4f6877c8b Handle incoming proxy requests ip, scheme (#5591)
This PR implements functions to get the right ip, scheme
from the incoming proxied requests.
2018-03-02 15:23:04 -08:00
Harshavardhana
d71b1d25f8 Make sure to filter out internal metadata (#5601)
Currently we reply back `X-Minio-Internal` values
back to the client for an encrypted object, we should
filter these out and only reply AWS compatible headers.
2018-03-01 16:15:53 -08:00
Anis Elleuch
5f37988db5 Use toAPIErrorCode in HeadObject handler when decrypting request fails (#5600) 2018-03-01 16:01:56 -08:00
Andreas Auernhammer
4e5237b02a vendor update github.com/minio/sio (#5599)
This change updates the sio library and adds DARE 2.0 support
to the server.
2018-03-01 15:28:34 -08:00
Harshavardhana
1b7b8f14c9 Set appropriate encryption headers in HEAD object response (#5596)
Currently we don't set two SSE-C specific headers fix it
for AWS S3 compatibility.
2018-03-01 14:16:40 -08:00
Anis Elleuch
120b061966 Add multipart support in SSE-C encryption (#5576)
*) Add Put/Get support of multipart in encryption
*) Add GET Range support for encryption
*) Add CopyPart encrypted support
*) Support decrypting of large single PUT object
2018-03-01 11:37:57 -08:00
kannappanr
d32f90fe95 Honor global flags irrespective of the position. (#5486)
Flags like `json, config-dir, quiet` are now honored even if they are
between minio and gateway in the cli, like, `minio --json gateway s3`.

Fixes #5403
2018-02-28 20:13:33 -08:00
Harshavardhana
6faa1ef11a Fix shadowing issue reported by go vet (#5590) 2018-02-28 14:30:00 -08:00
Harshavardhana
9af254a82f Remove stable sort usage when not needed (#5586)
Stable sort is needed when we are sorting based on two or more
distinct elements. When equal elements are indistinguishable,
such as with integers, or more generally, any data where the
entire element is the key like `PartNumber`, stability is not
an issue.
2018-02-28 08:33:00 +05:30
Harshavardhana
6b3db7556a Fix gofmt issues reported for simplification (#5581)
added travis to catch this issue
2018-02-26 23:52:03 +05:30
Harshavardhana
5204a754db Move rpc version to 2.0.0 to align with backend migration (#5575)
Fixes #5574
2018-02-23 16:58:37 -08:00
Harshavardhana
7cc678c653 Support encryption for CopyObject, GET-Range requests (#5544)
- Implement CopyObject encryption support
- Handle Range GETs for encrypted objects

Fixes #5193
2018-02-23 15:07:21 -08:00
poornas
b7536570f8 update docs for NAS gateway (#5569) 2018-02-22 09:15:19 +05:30
Harshavardhana
e09d97abaf Fix docs in admin API (#5559) 2018-02-21 12:00:46 -08:00
Krishnan Parthasarathi
e5e3d17216 Do not close *lock.LockedFile on failure (#5565) 2018-02-21 11:28:24 -08:00
Harshavardhana
0ea54c9858 Change CopyObject{Part} to single srcInfo argument (#5553)
Refactor such that metadata and etag are
combined to a single argument `srcInfo`.

This is a precursor change for #5544 making
it easier for us to provide encryption/decryption
functions.
2018-02-21 14:18:47 +05:30
Krishna Srinivas
a00e052606 Provide more descriptive error during erasure init (#5282)
fixes #5239
2018-02-20 18:42:09 -08:00
Anis Elleuch
d2d49f6c6c xl: Avoid removing directory content in Delete API (#5548)
Delete & Multi Delete API should not try to remove the directory content.
The only permitted case is with zero size object with a trailing slash
in its name.
2018-02-20 15:33:26 -08:00
Harshavardhana
db9e83de62 Avoid significant connections in TIME_WAIT (#5555)
MaxIdleConns limits the total number of connections
kept in the pool for re-use. In addition, MaxIdleConnsPerHost
limits the number for a single host. Since minio gateways
usually connect to the same host, setting `MaxIdleConns = 100`
won't really have much of an impact since the idle connection
pool is limited to 2 anyway.

Now, with the pool set to a limit of 2, and when using
the client heavily from 2+ goroutines, the `http.Transport`
will open a connection, use it, then try to return it to
the idle-pool which often fails since there's a limit of 2.
So it's going to close the connection and new ones will be
opened on demand again, many of which get closed soon after
being used. Since those connections/sockets don't disappear
from the OS immediately, use `MaxIdleConnsPerHost = 100`
which fixes this problem.
2018-02-20 12:23:37 -08:00
poornas
25107c2e11 Add NAS gateway support (#5516) 2018-02-20 12:21:12 -08:00
Anis Elleuch
926e480156 posix.RenameFile(): Allow overwriting an empty directory (#5551)
Overwriting files is allowed, but since the introduction of
the object directory, we will aslo need to allow overwriting
an empty directory. Putting twice the same object directory
won't fail with 403 error anymore.
2018-02-20 12:20:18 -08:00
Harshavardhana
b2b5056163 gateway/gcs: Remove unused storageEndpoint (#5556) 2018-02-20 15:07:31 +05:30
A. Elleuch
1e7e41fada tests: Fix failed notify webhook test (#5528)
TestNewWebHookNotify wasn't passing in my local machine. The reason is
that the test expects the POST handler (as a webhook endpoint) is always
running on port 80, which is not always the case.
2018-02-17 19:06:43 -08:00
Harshavardhana
03923947c4 Fix delete bucket policies properly (#5540)
There was bug in previous PR where deleteBucketMetadata()
was never called, fix it correctly.
2018-02-16 20:16:48 -08:00
Harshavardhana
d12bdd50ee Rename minio-limitations.md to minio-limits.md (#5541) 2018-02-16 09:35:02 +05:30
Harshavardhana
fb96779a8a Add large bucket support for erasure coded backend (#5160)
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.

This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.

Some design details and restrictions:

- Objects are distributed using consistent ordering
  to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
  properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
  requirement, you can start with multiple
  such sets statically.
- Static sets set of disks and cannot be
  changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
  changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
  slower since List happens on all servers,
  and is merged at this sets layer.

Fixes #5465
Fixes #5464
Fixes #5461
Fixes #5460
Fixes #5459
Fixes #5458
Fixes #5460
Fixes #5488
Fixes #5489
Fixes #5497
Fixes #5496
2018-02-15 17:45:57 -08:00
Harshavardhana
dd80256151 Directory HEADs with encryption headers shouldn't return errors (#5539)
Since we do not encrypt directories we don't need to send
errors with encryption headers when the directory doesn't
have encryption metadata.

Continuation PR from 4ca10479b5
2018-02-15 14:18:28 -08:00
Harshavardhana
22897de4c7 fail when endpoints point to same path locally (#5523) 2018-02-15 14:38:17 +05:30
Harshavardhana
e22438c8cd Cleanup banners and remove fossa/snap (#5530) 2018-02-15 09:45:42 +05:30
Harshavardhana
c59f1e3a80 revamp minio build messages (#5519) 2018-02-14 10:29:19 +05:30
Harshavardhana
994fe53669 Avoid shadowing ignored errors listAllBuckets() (#5524)
It can happen such that one of the disks that was down would
return 'errDiskNotFound' but the err is preserved due to
loop shadowing which leads to issues when healing the bucket.
2018-02-13 17:03:50 -08:00
Andreas Auernhammer
4ca10479b5 [SSE-C]: avoid encrypting empty objects. (#5525)
This change adds an object size check such that the server does not
encrypt empty objects (typically folders) for SSE-C. The server still
returns SSE-C headers but the object is not encrypted since there is no
point to encrypt such objects.

Fixes #5493
2018-02-13 15:43:46 -08:00
Harshavardhana
91101b11bb Converge repeated code to common deleteBucketMetadata() (#5508) 2018-02-12 18:34:30 -08:00
Harshavardhana
8de6cf4124 update dsync implementation to fix a regression (#5513)
Currently minio master requires 4 servers, we
have decided to run on a minimum of 2 servers
instead - fixes a regression from previous
releases where 3 server setups were supported.
2018-02-12 15:16:12 +05:30
poornas
4f73fd9487 Unify gateway and object layer. (#5487)
* Unify gateway and object layer. Bring bucket policies into
object layer.
2018-02-09 15:19:30 -08:00
Minio Trusted
a7f6e14370 Update yaml files to latest version RELEASE.2018-02-09T22-40-05Z 2018-02-09 22:43:57 +00:00
dingjs
289457568c Translate #5365 to Chinese. (#5439) 2018-02-08 17:00:03 -08:00
Harshavardhana
fd3897d0c3 Move to go1.9.4 with recent security release (#5502) 2018-02-08 14:33:22 +05:30
Krishna Srinivas
047b7aff0c Seek to offset 0 after Truncate() (#5375) 2018-02-06 15:37:48 -08:00
Harshavardhana
1164fc60f3 Bring semantic versioning to provide for rolling upgrades (#5495)
This PR brings semver capabilities in our RPC layer to
ensure that we can upgrade the servers in rolling fashion
while keeping I/O in progress. This is only a framework change
the functionality remains the same as such and we do not
have any special API changes for now. But in future when
we bring in API changes we will be able to upgrade servers
without a downtime.

Additional change in this PR is to not abort when serverVersions
mismatch in a distributed cluster, instead wait for the quorum
treat the situation as if the server is down. This allows
for administrator to properly upgrade all the servers in the cluster.

Fixes #5393
2018-02-06 15:07:17 -08:00
kannappanr
48218272cc Document object name limitations on Windows (#5491)
Fixes #5161
2018-02-03 19:57:40 +05:30
Harshavardhana
0c880bb852 Deprecate and remove in-memory object caching (#5481)
in-memory caching cannot be cleanly implemented
without the access to GC which Go doesn't naturally
provide. At times we have seen that object caching
is more of an hindrance rather than a boon for
our use cases.

Removing it completely from our implementation
  related to #5160 and #5182
2018-02-02 10:17:13 -08:00
Harshavardhana
1ebbc2ce88 Make sure to convert the disk errors to object errors (#5480)
Fixes a bug introduced in the directory support PR, with
this fix s3fs works properly.
2018-02-02 14:04:15 +05:30
A. Elleuch
da2faa19a1 Reduce Minio access key minimum length to 3 (#5478)
This is a generic minimum value. The current reason is to support
Azure blob storage accounts name whose length is less than 5. 3 is the
minimum length for Azure.
2018-02-02 09:13:30 +05:30
Krishna Srinivas
2afd196c83 Quorum based listing for XL (#5475)
fixes #5380
2018-02-01 10:47:49 -08:00
Krishna Srinivas
b606ba3f81 fs.json file should be closed in CompleteMultipartUpload (#5482) 2018-02-01 15:27:12 +05:30
Harshavardhana
3316dbc037 simplify storage class validation (#5470)
Check if the storage class is set in an
non XL setup instead of relying on `globalEndpoints`
value. Also converge the checks for both SS
and RRS parity configuration.

This PR also removes redundant `tt.name` in all
test cases, since each testcase doesn't need to
be numbered explicitly they are numbered implicitly.
2018-02-01 13:00:07 +05:30
Harshavardhana
033cfb5cef Remove stale code from minio server (#5479) 2018-01-31 18:28:28 -08:00
Krishna Srinivas
3b2486ebaf Lock free multipart backend implementation for FS (#5401) 2018-01-31 13:17:24 -08:00
Aditya Manthramurthy
018813b98f Fix configuration handling bugs: (#5473)
* Update the GetConfig admin API to use the latest version of
  configuration, along with fixes to the corresponding RPCs.
* Remove mutex inside the configuration struct, and inside
  notification struct.
* Use global config mutex where needed.
* Add `serverConfig.ConfigDiff()` that provides a more granular diff
  of what is different between two configurations.
2018-01-31 08:15:54 -08:00
ebozduman
e608e05cda Removes capitalization of error causes (#5468) 2018-01-30 21:42:15 -08:00
Harshavardhana
3ea28e9771 Support creating directories on erasure coded backend (#5443)
This PR continues from #5049 where we started supporting
directories for erasure coded backend
2018-01-30 08:13:13 +05:30
Krishna Srinivas
45c35b3544 Autocorrect user provided Azure endpoint (#5417)
fixes #5373
2018-01-29 10:30:08 -08:00
Andreas Auernhammer
09a9002f12 add documentation about PKCS-8 encrypted RSA keys (#5454)
This change adds documentation about PKCS-8 vs PKCS-1 pitfalls. It 
also provides a command to convert encrypted PKCS-8 RSA keys to 
encrypted PKCS-1 RSA keys.

Fixes #5453
2018-01-27 09:30:02 +05:30
Aditya Manthramurthy
5cdcc73bd5 Admin API auth and heal related fixes (#5445)
- Fetch region for auth from global state
- Fix SHA256 handling for empty body in heal API
2018-01-25 19:24:00 +05:30
poornas
2dd117f647 fix testcases to init nslock properly (#5429) 2018-01-24 09:04:09 +05:30
Harshavardhana
2d19663fef Update dockerfile go version to 1.9.2 (#5441) 2018-01-23 17:19:19 +05:30
Aditya Manthramurthy
254b05e314 Fix locking in some admin APIs: (#5438)
- read lock for get config
- write lock for update creds
- write lock for format file
2018-01-22 18:09:12 -08:00
Aditya Manthramurthy
a003de72bf Update madmin doc (fixes #5432) (#5433) 2018-01-22 16:10:43 -08:00
Aditya Manthramurthy
a337ea4d11 Move admin APIs to new path and add redesigned heal APIs (#5351)
- Changes related to moving admin APIs
   - admin APIs now have an endpoint under /minio/admin
   - admin APIs are now versioned - a new API to server the version is
     added at "GET /minio/admin/version" and all API operations have the
     path prefix /minio/admin/v1/<operation>
   - new service stop API added
   - credentials change API is moved to /minio/admin/v1/config/credential
   - credentials change API and configuration get/set API now require TLS
     so that credentials are protected
   - all API requests now receive JSON
   - heal APIs are disabled as they will be changed substantially

- Heal API changes
   Heal API is now provided at a single endpoint with the ability for a
   client to start a heal sequence on all the data in the server, a
   single bucket, or under a prefix within a bucket.

   When a heal sequence is started, the server returns a unique token
   that needs to be used for subsequent 'status' requests to fetch heal
   results.

   On each status request from the client, the server returns heal result
   records that it has accumulated since the previous status request. The
   server accumulates upto 1000 records and pauses healing further
   objects until the client requests for status. If the client does not
   request any further records for a long time, the server aborts the
   heal sequence automatically.

   A heal result record is returned for each entity healed on the server,
   such as system metadata, object metadata, buckets and objects, and has
   information about the before and after states on each disk.

   A client may request to force restart a heal sequence - this causes
   the running heal sequence to be aborted at the next safe spot and
   starts a new heal sequence.
2018-01-22 14:54:55 -08:00
Harshavardhana
f3f09ed14e Fix a bug in dsync initialization and communication (#5428)
In current implementation we used as many dsync clients
as per number of endpoints(along with path) which is not
the expected implementation. The implementation of Dsync
was expected to be just for the endpoint Host alone such
that if you have 4 servers and each with 4 disks we need
to only have 4 dsync clients and 4 dsync servers. But
we currently had 8 clients, servers which in-fact is
unexpected and should be avoided.

This PR brings the implementation back to its original
intention. This issue was found #5160
2018-01-22 10:25:10 -08:00
Harshavardhana
bb73c84b10 Add notification structure link (#5426)
Fixes #4545
2018-01-20 09:23:09 +05:30
Harshavardhana
e19eddd759 Remove requirement for custom RPCClient (#5405)
This change is a simplification over existing
code since it is not required to have a separate
RPCClient structure instead keep authRPCClient can
do the same job.

There is no code which directly uses netRPCClient(),
keeping authRPCClient is better and simpler. This
simplication also allows for removal of multiple
levels of locking code per object.

Observed in #5160
2018-01-19 16:38:47 -08:00
Andreas Auernhammer
7f99cc9768 add HighwayHash256 support (#5359)
This change adds the HighwayHash256 PRF as bitrot protection / detection
algorithm. Since HighwayHash256 requires a 256 bit we generate a random
key from the first 100 decimals of π - See nothing-up-my-sleeve-numbers.
This key is fixed forever and tied to the HighwayHash256 bitrot algorithm.

Fixes #5358
2018-01-19 10:18:21 -08:00
Aditya Manthramurthy
2760409656 Remove dead code and associated dead code warning (#5424) 2018-01-19 10:16:21 -08:00
fossabot
1f13235cbd Add license scan report and status (#5430) 2018-01-19 13:16:59 +05:30
poornas
dd5a3289dd fix: listobjects return empty response for invalid prefix/marker (#5425)
Currently minio server returns a NotImplemented error when marker
is not common with prefix. Instead, return an empty ListObjectsResponse
2018-01-18 14:39:39 -08:00
Harshavardhana
b6e4f053a3 Fix lock rpc server maintenance loop go-routine leak (#5423)
The problem was after the globalServiceDoneCh receives a
message, we cleanly stop the ticker as expected. But the
go-routine where the `select` loop is running is never
returned from. The stage at which point this may occur
i.e server is being restarted, doesn't seriously affect
servers usage. But any build up like this on server has
consequences as the new functionality would come in future.
2018-01-18 14:39:24 -08:00
Minio Trusted
1c3f55ff64 Update yaml files to latest version RELEASE.2018-01-18T20-33-21Z 2018-01-18 20:41:53 +00:00
Nitish Tiwari
e2d5a87b26 Fix free and total space reported in startup banner (#5419)
With storage class support, the free and total space
reported in Minio XL startup banner should be based on
totalDisks - standardClassParityDisks, instead of totalDisks/2.

fixes #5416
2018-01-17 11:25:51 -08:00
Andreas Auernhammer
d0a43af616 replace all "crypto/sha256" with "github.com/minio/sha256-simd" (#5391)
This change replaces all imports of "crypto/sha256" with
"github.com/minio/sha256-simd". The sha256-simd package
is faster on ARM64 (NEON instructions) and can take advantage
of AVX-512 in certain scenarios.

Fixes #5374
2018-01-17 10:54:31 -08:00
Paul Stack
a020a70484 gateway/manta: Bump manta dependencies (#5414)
Internally, triton-go, what manta minio is built on, changed it's internal
error handling. This means we no longer need to unwrap specific error types

This doesn't change any manta minio functionality - it just changes how errors are
handled internally and adds a wrapper for a 404 error
2018-01-17 10:38:39 -08:00
Andreas Auernhammer
3f09c17bfe fix authentication bypass against Admin-API (#5412)
This change fixes an authentication bypass attack against the
minio Admin-API. Therefore the Admin-API rejects now all types of
requests except valid signature V2 and signature V4 requests - this
includes signature V2/V4 pre-signed requests.

Fixes #5411
2018-01-17 10:36:25 -08:00
ebozduman
24d9d7e5fa Removes logrus package and refactors logging messages (#5293)
This fix removes logrus package dependency and refactors the console
logging as the only logging mechanism by removing file logging support.
It rearranges the log message format and adds stack trace information
whenever trace information is not available in the error structure.
It also adds `--json` flag support for server logging.
When minio server is started with `--json` flag, all log messages are
displayed in json format, with no start-up and informational log
messages.
Fixes #5265 #5220 #5197
2018-01-17 07:24:46 -08:00
kannappanr
85580fe0d6 Document pre-existing data in fs mode (#5365)
Document that pre-existing data in fs mode is safe and 
accessible

Fixes #5050
2018-01-17 20:44:31 +05:30
Nitish Tiwari
8a1dc10c60 Update storage class related documents (#5399)
- Add storage usage info in storage class doc
- Update distributed & erasure code doc with info on storage class
2018-01-17 14:52:42 +05:30
Krishnan Parthasarathi
17301fe45d Don't delete lock ops entry during state change (#5388)
lock ops entry is removed in deleteLockEntryForOps, it shouldn't be removed
in status*To* functions.
2018-01-16 12:00:12 -08:00
Aditya Manthramurthy
aa7e5c71e9 Remove upload healing related dead code (#5404) 2018-01-15 18:20:39 -08:00
Kaan Kabalak
78a641fc6a Fix multi-file dropzone upload issue causing bucket listing duplication (#5392)
This commit fixes the order of the functions inside the selectPrefix
function due to the fact that, as multiple files were being uploaded,
the resetObjects function (that clears the object list) ran repeatedly
for each of these objects, right before the appendObjects function (that
appends the objects being uploaded to the current list of objects) also
ran for all of these objects. This caused all the objects in the bucket
to be repeated in the list for the number of objects that were dragged
into the dropzone.
2018-01-13 22:45:20 +05:30
Harshavardhana
12f67d47f1 Fix a possible race during PutObject() (#5376)
Under any concurrent removeObjects in progress
might have removed the parents of the same prefix
for which there is an ongoing putObject request.
An inconsistent situation may arise as explained
below even under sufficient locking.

PutObject is almost successful at the last stage when
a temporary file is renamed to its actual namespace
at `a/b/c/object1`. Concurrently a RemoveObject is
also in progress at the same prefix for an `a/b/c/object2`.

To create the object1 at location `a/b/c` PutObject has
to create all the parents recursively.

```
a/b/c - os.MkdirAll loops through has now created
        'a/' and 'b/' about to create 'c/'
a/b/c/object2 - at this point 'c/' and 'object2'
        are deleted about to delete b/
```

Now for os.MkdirAll loop the expected situation is
that top level parent 'a/b/' exists which it created
, such that it can create 'c/' - since removeObject
and putObject do not compete for lock due to holding
locks at different resources. removeObject proceeds
to delete parent 'b/' since 'c/' is not yet present,
once deleted 'os.MkdirAll' would receive an error as
syscall.ENOENT which would fail the putObject request.

This PR tries to address this issue by implementing
a safer/guarded approach where we would retry an operation
such as `os.MkdirAll` and `os.Rename` if both operations
observe syscall.ENOENT.

Fixes #5254
2018-01-13 22:43:02 +05:30
poornas
0bb6247056 Move nslocking from s3 layer to object layer (#5382)
Fixes #5350
2018-01-13 10:04:52 +05:30
Andreas Auernhammer
dd202a1a5f restrict TLS cipher suites of the server (#5245)
This change restircts the supported cipher suites of the minio server.
The server only supports AEAD ciphers (Chacha20Poly1305 and 
AES-GCM)

The supported cipher suites are:
 - tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
 - tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
 - tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 - tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
 - tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 - tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Fixes #5244 and #5291
2018-01-13 09:12:11 +05:30
Nitish Tiwari
ede504400f Add validation of xlMeta ErasureInfo field (#5389) 2018-01-12 18:16:30 +05:30
Harshavardhana
4b2d04c86f Add chroot environment doc for minio (#5366)
Fixes #4659
2018-01-12 07:55:40 +05:30
Nitish Tiwari
42633748db Update madmin package to return storage class parity (#5387)
After the addition of Storage Class support, readQuorum
and writeQuorum are decided on a per object basis, instead
of deployment wide static quorums.

This PR updates madmin api to remove readQuorum/writeQuorum
and add Standard storage class and reduced redundancy storage
class parity as return values. Since these parity values are
used to decide the quorum for each object.

Fixes #5378
2018-01-12 07:52:52 +05:30
Aditya Manthramurthy
cd22feecf8 Remove healing of incomplete multipart uploads (#5390)
Since the server performs automatic clean-up of multipart uploads that
have not been resumed for more than a couple of weeks, it was decided
to remove functionality to heal multipart uploads.
2018-01-11 15:07:43 -08:00
kannappanr
20584dc08f Remove unnecessary errors printed on the console (#5386)
Some of the errors printed on server console can be
removed as those error message is unnecessary.

Fixes #5385
2018-01-11 11:42:05 -08:00
Aditya Manthramurthy
8e4eb591c1 Update error response when heal is not implemented (#5383) 2018-01-11 10:21:41 -08:00
Nitish Tiwari
1b721d76b1 Assume standard storage class if not set in metadata (#5370)
If STANDARD storage class is set before starting up Minio server, 
but x-amz-storage-class metadata field is not set in a PutObject 
request, Minio server defaults to N/2 data and N/2 parity disks.

This PR changes the behaviour to use data and parity disks set in
STANDARD storage class, even if x-amz-storage-class metadata 
field is not present in PutObject requests.
2018-01-11 14:58:12 +05:30
Aditya Manthramurthy
f413224b24 Fix config set handler (#5384)
- Return error when the config JSON has duplicate keys (fixes #5286)

- Limit size of configuration file provided to 256KiB - this prevents
  another form of DoS
2018-01-11 12:36:36 +05:30
Harshavardhana
b526cd7e55 Remove requirement for issued at JWT claims (#5364)
Remove the requirement for IssuedAt claims from JWT
for now, since we do not currently have a way to provide
a leeway window for validating the claims. Expiry does
the same checks as IssuedAt with an expiry window.

We do not need it right now since we have clock skew check
in our RPC layer to handle this correctly.

rpc-common.go
```
func isRequestTimeAllowed(requestTime time.Time) bool {
        // Check whether request time is within acceptable skew time.
        utcNow := UTCNow()
        return !(requestTime.Sub(utcNow) > rpcSkewTimeAllowed ||
                utcNow.Sub(requestTime) > rpcSkewTimeAllowed)
}
```

Once the PR upstream is merged https://github.com/dgrijalva/jwt-go/pull/139
We can bring in support for leeway later.

Fixes #5237
2018-01-10 10:34:00 -08:00
Aditya Manthramurthy
3f8379d07d Update Elasticsearch documentation with authentication information (#5381)
- Add documentation to show how to supply credential to access a
  secured elasticsearch server.

Fixes #5329
2018-01-10 09:20:42 +05:30
Harshavardhana
7350543f24 Allow x-amz-content-sha256 to be optional for PutObject() (#5340)
x-amz-content-sha256 can be optional for any AWS signature v4
requests, make sure to skip sha256 calculation when payload
checksum is not set.

Here is the overall expected behavior

** Signed request **
- X-Amz-Content-Sha256 is set to 'empty' or some 'value'  or its
  not 'UNSIGNED-PAYLOAD'- use it to validate the incoming payload.
- X-Amz-Content-Sha256 is set to 'UNSIGNED-PAYLOAD' - skip checksum verification
- X-Amz-Content-Sha256 is not set we use emptySHA256

** Presigned request **
- X-Amz-Content-Sha256 is set to 'empty' or some 'value'  or its
  not 'UNSIGNED-PAYLOAD'- use it to validate the incoming payload
- X-Amz-Content-Sha256 is set to 'UNSIGNED-PAYLOAD' - skip checksum verification
- X-Amz-Content-Sha256 is not set we use 'UNSIGNED-PAYLOAD'

Fixes #5339
2018-01-09 12:49:50 +05:30
Nitish Tiwari
56bde5df31 Refactor storage class parsing for Gateway mode (#5331)
This PR updates the behaviour to print relevant error message
if storage class is set in config.json for gateway

This PR also fixes the case where storage class set via
environment variables is not parsed properly into config.json.
2018-01-08 22:26:13 -08:00
Paul Trunk
bd9cdcf379 Add custom secret names for Docker (#5355) 2018-01-09 10:46:25 +05:30
Krishna Srinivas
7c72d14027 Separate the codebase for XL and FS format.json related code (#5317) 2018-01-08 14:30:55 -08:00
Harshavardhana
ccd9767b7a Remove incomplete older chinese translation doc (#5368) 2018-01-08 12:48:59 +05:30
Harshavardhana
dae8193bd4 Remove duplicate http constants (#5367) 2018-01-08 10:17:48 +05:30
kannappanr
1de3bd6911 Save http trace to a file (#5300)
Save http trace to a file instead of displaying it onto the console.
the environment variable MINIO_HTTP_TRACE will be a filepath instead
of a boolean.

This to handle the scenario where both json and http tracing are
turned on. In that case, both http trace and json output are displayed
on the screen making the json not parsable. Loging this trace onto
a file helps us avoid that scenario.

Fixes #5263
2018-01-05 11:24:31 -08:00
Kaan Kabalak
de2ce5acb4 Correct color code of Excel file icon (#5361)
Fixes #5352
2018-01-05 16:19:41 +05:30
Paul Stack
a1a98617ca gateway/manta: Add support for RBAC (#5332)
Manta has the ability to allow users to authenticate with a 
username other than the main account. We want to expose 
this functionality to minio manta gateway.
2018-01-05 13:30:29 +05:30
Andreas Auernhammer
b85c75996d add support for encrypted TLS private keys (#5308)
This change adds support for password-protected private keys.
If the private key is encrypted the server tries to decrypt
the key with the password provided by the env variable 
MINIO_CERT_PASSWD.

Fixes #5302
2018-01-05 13:18:08 +05:30
Harshavardhana
cc2497f52f Exitcode with '1' when update is available (#5354)
--quiet should simply update the binary without any prompt.

Fixes #5347
2018-01-04 21:26:43 +05:30
Nitish Tiwari
1e5fb4b79a Fix storage class related issues (#5338)
- Update startup banner to print storage class in capitals. This
makes it easier to identify different storage classes available.

- Update response metadata to not send STANDARD storage class.
This is in accordance with AWS S3 behaviour.

- Update minio-go library to bring in storage class related
changes. This is needed to make transparent translation of
storage class headers for Minio S3 Gateway.
2018-01-04 11:44:45 +05:30
kannappanr
6f7c6fc560 Honor browser enabled config value in startup message (#5313)
Currently, browser access information is displayed without checking
if browser enabled flag is turned off in config.json. Fixing it to
hide the information if the flag is turned off.

Fixes #5312
2018-01-04 11:00:52 +05:30
Harshavardhana
c0721164be Automatically set goroutines based on shardSize (#5346)
Update reedsolomon library to enable feature to automatically
set number of go-routines based on the input shard size,
since shard size is sort of a constant in Minio for
objects > 10MiB (default blocksize)

klauspost reported around 15-20% improvement in performance
numbers on older systems such as AVX and SSE3

```
name                  old speed      new speed      delta
Encode10x2x10000-8    5.45GB/s ± 1%  6.22GB/s ± 1%  +14.20%    (p=0.000 n=9+9)
Encode100x20x10000-8  1.44GB/s ± 1%  1.64GB/s ± 1%  +13.77%  (p=0.000 n=10+10)
Encode17x3x1M-8       10.0GB/s ± 5%  12.0GB/s ± 1%  +19.88%  (p=0.000 n=10+10)
Encode10x4x16M-8      7.81GB/s ± 5%  8.56GB/s ± 5%   +9.58%   (p=0.000 n=10+9)
Encode5x2x1M-8        15.3GB/s ± 2%  19.6GB/s ± 2%  +28.57%   (p=0.000 n=9+10)
Encode10x2x1M-8       12.2GB/s ± 5%  15.0GB/s ± 5%  +22.45%  (p=0.000 n=10+10)
Encode10x4x1M-8       7.84GB/s ± 1%  9.03GB/s ± 1%  +15.19%    (p=0.000 n=9+9)
Encode50x20x1M-8      1.73GB/s ± 4%  2.09GB/s ± 4%  +20.59%   (p=0.000 n=10+9)
Encode17x3x16M-8      10.6GB/s ± 1%  11.7GB/s ± 4%  +10.12%   (p=0.000 n=8+10)
```
2018-01-03 13:47:22 -08:00
Minio Trusted
b1fb550d5c Update yaml files to latest version RELEASE.2018-01-02T23-07-00Z 2018-01-02 23:11:15 +00:00
Andreas Auernhammer
a6318dbdaf fix timing oracle attack against signature V2/V4 verification (#5335)
This change replaces the non-constant time comparison of
request signatures with a constant time implementation. This
prevents a timing attack which can be used to learn a valid 
signature for a request without knowing the secret key.

Fixes #5334
2018-01-02 12:00:02 +05:30
Harshavardhana
e39d7ddb0f Fix PostPolicy form tests without hardcoded dates (#5337)
Fixes #5336
2018-01-01 07:28:10 +05:30
Kaan Kabalak
659f724f4c Integrate existing remove bucket functionality from newux to current UI (#5289)
This commit takes the existing remove bucket functionality written by
brendanashworth, integrates it to the current UI with a dropdown for
each bucket, and fixes small issues that were present, like the dropdown
not disappearing after the user clicks on 'Delete' for certain buckets.
This feature only deletes a bucket that is empty (that has no objects).

Fixes #4166
2017-12-29 18:45:44 +05:30
Nitish Tiwari
baaf67d82e Update config.json guide with details of version 22 (#5328)
Fixes #5296
2017-12-28 23:04:44 +05:30
A. Elleuch
2244adff07 fix: Better printing of XL config init error (#5284) 2017-12-28 23:02:48 +05:30
Nitish Tiwari
e3d841ffd1 Fix config.json parsing to fetch correct storage class (#5327) 2017-12-28 14:19:45 +05:30
Minio Trusted
751632d79e Update yaml files to lastest version RELEASE.2017-12-28T01-21-00Z 2017-12-28 01:26:09 +00:00
Nitish Tiwari
545a9e4a82 Fix storage class related issues (#5322)
- Add storage class metadata validation for request header
- Change storage class header values to be consistent with AWS S3
- Refactor internal method to take only the reqd argument
2017-12-27 10:06:16 +05:30
Harshavardhana
f25ec31565 Set maxResources appropriately for gateway like server (#5321) 2017-12-24 20:09:30 +05:30
Matthieu Paret
374feda237 add HTTPStats to madmin (#5299) 2017-12-22 17:47:30 -08:00
A. Elleuch
6ef0161835 fix: Restore empty files when healing (#5257)
HealFile() does not process the case when an empty file is lost in
some disks. Since, Reedsolomon erasure doesn't handle restoring empty
data, HealFile will create empty files similarly to CreateFile().
2017-12-22 14:57:57 -08:00
Nitish Tiwari
1a3dbbc9dd Add x-amz-storage-class support (#5295)
This adds configurable data and parity options on a per object
basis. To use variable parity

- Users can set environment variables to cofigure variable
parity

- Then add header x-amz-storage-class to putobject requests
with relevant storage class values

Fixes #4997
2017-12-22 16:58:13 +05:30
Aditya Manthramurthy
f1355da72e Add base64 encoded MD5 output for Hash Reader (#5315)
- Use it to send the Content-MD5 header correctly encoded to S3
  Gateway

- Fixes a bug in PutObject (including anonymous PutObject) and
  PutObjectPart with S3 Gateway found when testing with Mint.
2017-12-21 17:27:33 -08:00
Krishnan Parthasarathi
bbe521ffec ReInitDisk RPC handler should use retryStorage (#5310) 2017-12-21 12:28:01 +05:30
Paul Stack
7d75d61621 Add Support for Manta Object Storage as a Gateway (#5025)
Manta is an Object Storage by [Joyent](https://www.joyent.com/)

This PR adds initial support for Manta. It is intended as non-production 
ready so that feedback can be obtained.
2017-12-20 13:37:56 +05:30
Harshavardhana
1f77708a30 Limit number of connections upto system maxlimit (#5109) 2017-12-20 13:30:14 +05:30
Timon Wong
84fc78d60f Implement Alibaba Cloud OSS gateway support (#5103) 2017-12-19 13:55:17 +05:30
poornas
a182fe8c15 update steps to make changes to config.json (#5292) 2017-12-17 21:00:12 -08:00
Harshavardhana
819d1e80c6 Add more delays on distributed startup for slow network (#5240)
Refer #5237
2017-12-16 08:25:29 -08:00
Kaan Kabalak
ffdf115bf2 Fix mobile issue where checkbox was seen over sidebar on list item hover (#5288)
Fixes #5287
2017-12-15 12:36:05 -08:00
Harshavardhana
eb7c690ea9 Support in-place upgrades of new minio binary and releases. (#4961)
This PR allows 'minio update' to not only shows update banner
but also allows for in-place upgrades.

Updates are done safely by validating the downloaded
sha256 of the binary.

Fixes #4781
2017-12-15 12:33:42 -08:00
Nitish Tiwari
8c08571cd9 Update Kubernetes example yaml files (#5278)
Removed the non production ready Kubernetes constructs that are not needed
for standard Minio deployment. General cleanup of the documents.
2017-12-12 10:29:00 +05:30
kannappanr
2853fa1882 Remove logger field info from docs (#5281)
Logger field is removed from the docs,
as it has been removed from the config file.
2017-12-11 13:20:05 -08:00
Harshavardhana
1672c73988 Fix snap docs with instructions to run (#5266) 2017-12-07 08:38:58 +05:30
Harshavardhana
5476415016 Remove unused deps for cloudresourcemanager (#5274) 2017-12-06 16:41:56 -08:00
Kaan Kabalak
67ac74471d Fix error message on browser window resize when user has no buckets (#5275)
This commit handles the case where the list of buckets is empty on the
listObjects function of actions.js

Fixes #5267
2017-12-06 15:52:23 -08:00
kannappanr
a1c1a18dc5 Remove "logger" field from config.json (#5268)
File logging removed as part of improvement to server logging.

config.json format updated to version 21.

Fixes #5176
2017-12-06 12:48:29 +05:30
Harshavardhana
eb2894233c Convert gateways into respective packages (#5200)
- Make azure gateway a package
- Make b2 gateway a package
- Make gcs gateway a package
- Make s3 gateway a package
- Make sia gateway a package
2017-12-05 17:58:09 -08:00
Nitish Tiwari
52e382b697 Update go version in snap file (#5256) 2017-12-04 15:49:31 -08:00
techknowlogick
0d435e11b1 Change container name in b2 docs (#5259) 2017-12-04 22:34:00 +05:30
Harshavardhana
2755a0b763 Check if SSL is configured to validate input arguments (#5252)
This PR handles following situations

- secure endpoints provided, server should fail to start
  if TLS is not configured

- insecure endpoints provided, server starts ignoring
  if TLS is configured or not.

Fixes #5251
2017-12-04 12:17:12 +05:30
Aditya Manthramurthy
043e030a4a Add CopyObjectPart support to gateway S3 (#5213)
- Adds a metadata argument to the CopyObjectPart API to facilitate
  implementing encryption for copying APIs too.

- Update vendored minio-go - this version implements the
  CopyObjectPart client API for use with the S3 gateway.

Fixes #4885
2017-12-02 08:33:59 +05:30
Harshavardhana
490c30f853 erasure: Support cleaning up of stale multipart objects (#5250)
Just like our single directory/disk setup, this PR brings
the functionality to cleanup stale multipart objects
older > 2 weeks.
2017-11-30 18:11:42 -08:00
Harshavardhana
59749a2b85 erasure: Remove prefix based listing support on ListMultipartUploads (#5248)
Previously we have removed this support under FS on #4996,
deprecate this on erasure coded backend as well to simplify
our multipart support.
2017-11-30 15:58:46 -08:00
Michael Lynch
fc3cf97b81 Removing isValidObjectName from Sia gateway (#5243)
This check incorrectly rejects most valid filenames. The only filenames Sia
forbids are leading forward slashes and path traversal characters, but it's
better to simply allow Sia to reject invalid names on its own rather than try
to anticipate errors from Sia:

https://github.com/NebulousLabs/Sia/blob/master/doc/api/Renter.md#path-parameters-4
2017-11-30 14:43:21 -08:00
Harshavardhana
d45a8784fc Fix hash order to generate more even distribution (#5247)
The problem in existing code was the following line

```
start := int(keyCrc%uint32(cardinality)) | 1
```

A given a value of N cardinality the ending result
because of the the bitwise '|' would lead to always
higher affinity to odd sequences.

As can be seen from the test cases that this can
lead to many objects being allocated the same set
of disks or atleast the first disk is an odd disk
always.  This introduces a performance problem
for majority of the objects under concurrent load.

Remove `| 1` to provide a more cleaner distribution
and the new code will be.
```
start := int(keyCrc % uint32(cardinality))
```

Thanks to Krishna Srinivas for pointing out the bitwise
situation here.
2017-11-30 12:57:03 -08:00
Nitish Tiwari
6d7319380c Add Transparent Hugepage information (#5246)
Fixes #5242
2017-11-30 12:18:00 -08:00
Krishna Srinivas
14e6c5ec08 Simplify the steps to make changes to config.json (#5186)
This change introduces following simplified steps to follow 
during config migration.

```
 // Steps to move from version N to version N+1
 // 1. Add new struct serverConfigVN+1 in config-versions.go
 // 2. Set configCurrentVersion to "N+1"
 // 3. Set serverConfigCurrent to serverConfigVN+1
 // 4. Add new migration function (ex. func migrateVNToVN+1()) in config-migrate.go
 // 5. Call migrateVNToVN+1() from migrateConfig() in config-migrate.go
 // 6. Make changes in config-current_test.go for any test change
```
2017-11-29 13:12:47 -08:00
A. Elleuch
98d07210e7 fix: Ignore logging some tcp routine errors (#5097) 2017-11-28 13:51:17 -08:00
Nitish Tiwari
6923630389 Update bucket notification docs to mention events supported (#5235)
Fixes #4898
2017-11-28 15:41:26 +05:30
Nitish Tiwari
0c73c81919 Cleanup TLS setup document (#5231)
Fixes #4959
2017-11-28 15:15:50 +05:30
Harshavardhana
a46b640da3 gateway/sia: Support proper {make,get}Bucket operations (#5229)
Current implementation we faked the makeBucket operations
to allow for s3 clients to behave properly. But instead
we can create a placeholder zero byte file instead, which
is a hexadecimal representation of the bucket name itself.
2017-11-28 13:40:44 +05:30
Krishna Srinivas
71f9d2beff Increase maximum size of PUT request to 5TB (#5241)
fixes #5148
2017-11-28 12:59:02 +05:30
Michael Lynch
cf414a6053 Fixing Sia file uploads (#5233)
The Sia gateway had a bug with uploading that prevented the user's uploads
from reaching the Sia backend. The PutObject function called fsRemoveFile at
the end of the function, which didn't give the Sia backend enough time to
upload the file to the Sia network.

This adds a goroutine that watches the file upload progress and doesn't delete
the file until the upload reaches 100% complete.

Note that this solution has the limitation where if the minio process dies in
the middle of upload, it will leave orphaned files in the SIA_TEMP directory
that the user will need to remove manually.
2017-11-28 12:25:15 +05:30
Harshavardhana
05b395e81d Add more unit tests for azure/gcs/b2 gateway (#5236)
Also adds a blazer SDK update exposing
error response headers.
2017-11-27 18:29:22 -08:00
Paul Nicholls
6a2d7ae808 gateway/azure: ListParts return an empty list if no parts uploaded yet (#5230)
This makes azure ListParts implementation behave more like S3 
by returning an empty list rather than an error when no parts have
been uploaded yet.
2017-11-27 17:42:27 -08:00
Eduardo Reyes
685afb6749 Fixes last item in the list menu being blocked. (#5234)
This commit fixes an issue where the last item's menu on a list of files
that scrolls gets blocked by the floating add button.

The fix is simply add the same padding that we use for the responsive
view, since it works just fine in responsive.
2017-11-27 14:02:57 -08:00
Harshavardhana
8efa82126b Convert errors tracer into a separate package (#5221) 2017-11-25 11:58:29 -08:00
Frank Wessels
6e6aeb6a9e Updated version of klauspost/reedsolomon using proper AVX2 instructions as well a providing support for Cauchy matrices. (#5215) 2017-11-24 14:23:20 -08:00
Arnaud de Mouhy
c9e00ae0a5 Add support for dynamic address in docker healthcheck (#5228) 2017-11-24 14:11:26 +05:30
Nitish Tiwari
08e0698b7e Update docs to latest Minio server release (#5227) 2017-11-23 10:19:53 +05:30
Harshavardhana
135a6a7bb4 Add chinese translation docs. (#5224) 2017-11-22 15:15:40 -08:00
Dee Koder
8b4d7048f8 fixed typo variable under domain heading. (#5223) 2017-11-22 12:37:00 -08:00
David G
f4d4ea5c36 Implement Sia Gateway (#5114) 2017-11-22 12:12:10 -08:00
Aditya Manthramurthy
d1a6c32d80 Improve make and make install messages (#5207) 2017-11-21 16:22:01 -08:00
Krishna Srinivas
bbd05a8f1c gateway-gcs: Close the writer with error in case of any errors. (#5217)
fixes #5216
2017-11-21 14:45:37 -08:00
Krishna Srinivas
4393afb7e2 Remove checkGCSProjectID() as it needs extra permission setting (#5210)
fixes #5209
2017-11-21 10:43:39 -08:00
Krishna Srinivas
1a53734477 Rename UserDefined to UserMetadata for events (#5206)
fixes #5165
2017-11-20 15:32:25 -08:00
Andreas Auernhammer
e95c0bb913 return AWS compliant error if SSE-C key is wrong (#5203)
This PR changes the behavior of DecryptRequest.
Instead of returning `object-tampered` if the client provided
key is wrong DecryptRequest will return `access-denied`.

This is AWS S3 behavior.

Fixes #5202
2017-11-20 14:04:10 -08:00
Krishna Srinivas
fce556b8a0 Support for ListObjectParts in azure-gateway (#5198)
fixes #5169
2017-11-20 14:03:20 -08:00
Andreas Auernhammer
b97f99766f add benchmarks for erasure backend (#5084)
This change adds benchmarks for erasure read/write in different setups.
2017-11-17 14:57:04 -08:00
Nitish Tiwari
f7b6f7b22f Update getObjectInfo to stat for objects with trailing / (#5179)
Apache Spark sends getObject requests with trailing "/".
This PR updates the getObjectInfo to stat for files
even if they are sent with trailing "/".

Fixes #2965
2017-11-16 16:00:27 -08:00
Krishnan Parthasarathi
2a0a62b78d Return ErrContentSHA256Mismatch when sha256sum is invalid (#5188) 2017-11-16 11:13:04 -08:00
Krishnan Parthasarathi
67f66c40c1 Fix ListenBucketNotification deadlock (#5028)
Previously ListenBucketNotificationHandler could deadlock with
PutObjectHandler's eventNotify call when a client closes its
connection. This change removes the cyclic dependency between the
channel and map of ARN to channels by using a separate done channel to
signal that the client has quit.
2017-11-16 10:56:06 -08:00
Krishna Srinivas
5a2bdf6959 Handle Path validation inside the PostPolicy handler (#5192) 2017-11-15 14:10:45 -08:00
silenceshell
51e78a3e20 fix a typo (#5187) 2017-11-15 15:12:14 +05:30
Harshavardhana
9eb52ec7c7 Remove release scripts for minio. (#5181)
Use `GOOS=<osname> go build`  to build minio for any platform of choice.
2017-11-14 17:05:40 -08:00
Harshavardhana
0827a2747b api: CopyObject should return NotImplemented for now (#5183)
Commit ca6b4773ed introduces SSE-C
support for HEAD, GET, PUT operations but since we do not
implement CopyObject() we should return NotImplemented.
2017-11-14 16:57:19 -08:00
Krishna Srinivas
e7a724de0d Virtual host style S3 requests (#5095) 2017-11-14 16:56:24 -08:00
Krishna Srinivas
d57d57ddf5 Update ui-assets.go (#5185) 2017-11-14 16:22:35 -08:00
Harshavardhana
a4d6195244 Add public data-types for easier external loading (#5170)
This change brings public data-types such that
we can ask projects to implement gateway projects
externally than maintaining in our repo.

All publicly exported structs are maintained in object-api-datatypes.go

completePart --> CompletePart
uploadMetadata --> MultipartInfo

All other exported errors are at object-api-errors.go
2017-11-14 13:55:10 +05:30
Krishna Srinivas
7d3eaf79ff Set Minio user-agent for GCS calls (#5154) 2017-11-13 19:06:51 -08:00
kannappanr
b63c37b28e Return MethodNotAllowed error in PostPolicyBucketHandler if URL contains object name (#5142)
S3 spec requires that MethodNotAllowed error be return if object name is part
of the URL.

Fix postpolicy related unit tests to not set object name as part of target URL.

Fixes #5141
2017-11-13 16:30:20 -08:00
Harshavardhana
8d59f35523 Add GetInfo() support for solaris (#5174)
Fixes #5173
2017-11-13 12:54:38 -08:00
kannappanr
f460eceb6d Check for value > 7 days in X-Amz-Expires header. (#5163)
Add a check to see if the X-Amz-Expires header in the presigned URL is less than 7 days.

Fixes #5162
2017-11-13 12:54:03 -08:00
Harshavardhana
d10679866c Fix minio distributed setup to properly work on windows (#5152)
On windows having a preceding "/" will cause problems, if the
command line already has C:/<export-folder/ in it. Final resulting
path on windows might become C:/C:/ this will cause problems
of starting minio server properly in distributed mode on windows.
As a special case make sure to trim off the separator.

NOTE: It is also perfectly fine for windows users to have a path
without C:/ since at that point we treat it as relative path
and obtain the full filesystem path as well. Providing C:/
style is necessary to provide paths other than C:/,
such as F:/, D:/ etc.

Another additional benefit here is that this style also
supports providing UNC paths as well.

Fixes #5136
2017-11-12 08:09:53 +05:30
Andreas Auernhammer
a79a7e570c replace SSE-C key derivation scheme (#5168)
This chnage replaces the current SSE-C key derivation scheme. The 'old'
scheme derives an unique object encryption key from the client provided key.
This key derivation was not invertible. That means that a client cannot change
its key without changing the object encryption key.
AWS S3 allows users to update there SSE-C keys by executing a SSE-C COPY with
source == destination. AWS probably updates just the metadata (which is a very
cheap operation). The old key derivation scheme would require a complete copy
of the object because the minio server would not be able to derive the same
object encryption key from a different client provided key (without breaking
the crypto. hash function).

This change makes the key derivation invertible.
2017-11-10 17:21:23 -08:00
Harshavardhana
16ecaac4fc Help message should prioritize gateway after server (#5153)
Currently gateway is listed as a command after {version, update}
which is incorrect, fix it.
2017-11-08 13:38:53 -08:00
Andreas Auernhammer
ca6b4773ed add SSE-C support for HEAD, GET, PUT (#4894)
This change adds server-side-encryption support for HEAD, GET and PUT
operations. This PR only addresses single-part PUTs and GETs without
HTTP ranges.

Further this change adds the concept of reserved object metadata which is required
to make encrypted objects tamper-proof and provide API compatibility to AWS S3.
This PR adds the following reserved metadata entries:
- X-Minio-Internal-Server-Side-Encryption-Iv          ('guarantees' tamper-proof property)
- X-Minio-Internal-Server-Side-Encryption-Kdf         (makes Key-MAC computation negotiable in future)
- X-Minio-Internal-Server-Side-Encryption-Key-Mac     (provides AWS S3 API compatibility)

The prefix `X-Minio_Internal` specifies an internal metadata entry which must not
send to clients. All client requests containing a metadata key starting with `X-Minio-Internal`
must also rejected. This is implemented by a generic-handler.

This PR implements SSE-C separated from client-side-encryption (CSE). This cannot decrypt
server-side-encrypted objects on the client-side. However, clients can encrypted the same object
with CSE and SSE-C.

This PR does not address:
 - SSE-C Copy and Copy part
 - SSE-C GET with HTTP ranges
 - SSE-C multipart PUT
 - SSE-C Gateway

Each point must be addressed in a separate PR.

Added to vendor dir:
 - x/crypto/chacha20poly1305
 - x/crypto/poly1305
 - github.com/minio/sio
2017-11-07 15:18:59 -08:00
Krishna Srinivas
7e7ae29d89 browser: Remove hardcoding of minioBrowserPrefix=/minio (#5048)
This enable reverse proxy of minio-browser. Fixes #5040
2017-11-06 15:59:37 -08:00
Krishnan Parthasarathi
7d18f00116 Make GCS multipart upload failures S3-compatible (#5138) 2017-11-06 10:09:21 -08:00
Harshavardhana
719f8c258a fix content-sha256 verification for presigned PUT (#5137)
It is possible that x-amz-content-sha256 is set through
the query params in case of presigned PUT calls, make sure
that we validate the incoming x-amz-content-sha256 properly.

Current code simply just allows this without honoring the
set x-amz-content-sha256, fix it.
2017-11-05 16:32:19 +05:30
Harshavardhana
dcdb07433a Support conditions for ListMultipartUploads and ListParts (#5130) 2017-11-02 11:39:48 -07:00
kannappanr
26e9f78a86 Display help when access/secret key is not set (#5132)
Display help message, when access and secret keys are not set in
any of the gateway.

Fixes #5124
2017-11-01 11:45:27 -07:00
Nitish Tiwari
34a1b58a75 Remove redirectHeaders method (#5120)
As of go 1.8, headers are copied on redirect, so we no longer need
to do this manually.

See https://github.com/golang/go/issues/4800 and
https://go-review.googlesource.com/c/go/+/28930 for more context on go
behaviour.

Fixes #5042
2017-11-01 12:43:13 +05:30
Bala FA
32c6b62932 move credentials as separate package (#5115) 2017-10-31 11:54:32 -07:00
Harshavardhana
8d584bd819 Remove unused value from retry-storage (#5129) 2017-10-29 15:43:16 +05:30
Nitish Tiwari
031767791d Update GitHub PR template to include mint tests (#4970)
Mint is the functional testing platform for Minio server. So, it is
important that each PR to Minio server repository is checked for
addition/updating of Mint test cases.
2017-10-28 07:14:19 -07:00
Nitish Tiwari
3b917067d9 Update sample yaml files to latest release RELEASE.2017-10-27T18-59-02Z (#5127) 2017-10-28 07:11:53 -07:00
Harshavardhana
203ac8edaa Bucket policies should use minio-go/pkg/policy instead. (#5090) 2017-10-27 16:14:06 -07:00
Harshavardhana
8bbfb1b714 Allow event notifications to work without region (#5119)
Fixes #5101
2017-10-27 15:09:55 -07:00
Harshavardhana
b4ddccc2f7 browser: Return a more descriptive error for HTTP 500 (#5112) 2017-10-27 15:09:14 -07:00
Timon Wong
6400f506da Simplify gateway backend registration (#5111) 2017-10-27 15:07:46 -07:00
Frank Wessels
7195ac7f14 Add space to error message (#5108) 2017-10-27 15:07:14 -07:00
Krishna Srinivas
03df692ae2 Support for bosh/pcf user-agent when querying for updates. (#5116) 2017-10-26 18:53:45 -07:00
kannappanr
a011fe8450 "0" offset is ignored in GetObject method in Azure Gateway code (#5118)
In GetObject method, Check if startoffset is a non-negative value.
Ignore check for startOffset > and check for only length > 0.

Fixes minio/mint#191
2017-10-26 18:01:46 -07:00
kannappanr
95d97c2d6d GCS gateway to return error in getBucketPolicy, when no policy is set (#5117)
Return NoSuchBucketPolicy error when there is no policy set.
Fixes minio/mint#199
2017-10-26 18:01:00 -07:00
Bala FA
bc8b936d4b convert ETag properly for all gateways (#5099)
Previously ID/ETag from backend service is used as is which causes
failure on s3cmd like tools where those tools use ETag as checksum to
validate data.  This is fixed by prepending "-1".

Refer minio/mint#193 minio/mint#201
2017-10-26 10:17:07 -07:00
Aditya Manthramurthy
d23ded0d83 Use retryableStorage after healing format.json (#5105)
- Previously networkStorage was being used and this lead to errors
  when listing with a down server/disk

Fixes #5089
2017-10-26 09:52:23 -07:00
Julien Maitrehenry
db3fed2279 Fix s3MetaToAzureProperties Content-Md5 key (#5068) 2017-10-25 11:00:07 -07:00
A. Elleuch
866dffcd62 log: Store http request/responses in a log file (#4804)
When MINIO_TRACE_DIR is provided, create a new log file and store all
HTTP requests + responses data, body are excluded to reduce memory
consumption. MINIO_HTTP_TRACE=1 enables logging. Use non mem
consuming http req/resp recorders, the maximum is about 32k per request.
This logs to STDOUT, body logging is disabled for PutObject PutObjectPart
GetObject.
2017-10-25 10:59:53 -07:00
Harshavardhana
5eb210dd2e Set etag properly to calculated value if available (#5106)
Fixes #5100
2017-10-24 12:25:42 -07:00
kula
758d5458f0 Update documenation to reflect correct default region of '' (#5101)
Ever since commit 5db1e9f3dd the
default region as been '', instead of 'us-east-1'. Update
documentation to reflect this, in particular, documentation about
notifications.
2017-10-23 11:57:40 -07:00
asubmani
8a40da3fd0 Update azure.md (#5086) 2017-10-23 14:38:38 +05:30
Harshavardhana
1d8a8c63db Simplify data verification with HashReader. (#5071)
Verify() was being called by caller after the data
has been successfully read after io.EOF. This disconnection
opens a race under concurrent access to such an object.
Verification is not necessary outside of Read() call,
we can simply just do checksum verification right inside
Read() call at io.EOF.

This approach simplifies the usage.
2017-10-22 11:00:34 +05:30
Krishna Srinivas
65a817fe8c Move packages between dependencies and devDependencies (#5094)
This change is needed to generate the right Open Source Disclosure File.
2017-10-21 12:21:32 +05:30
Julien Maitrehenry
1256b0b818 Fix multipart upload etag on azure gateway (#5055) 2017-10-20 14:00:18 -07:00
Krishna Srinivas
7e05b826fa Figure out projectID for GCS automatically from credentials.json (#5029)
fixes #5027
2017-10-20 13:59:12 -07:00
Harshavardhana
d82a1da511 Fix notification unmarshalling, unmarshal only when size is > 0 (#5087)
Fixes #5085
2017-10-20 13:57:57 -07:00
Frank Wessels
f598f4fd1b Fix typo in comment (#5088) 2017-10-20 15:08:15 +05:30
Krishna Srinivas
e65d8fdc7d Remove unused packages from vendor.json (#5079) 2017-10-19 15:14:10 +05:30
A. Elleuch
b919462610 fix: Avoid teeing data into a null cache buffer (#5070)
In some cases, Cache manager returns ErrCacheFull error when creating a
new cache buffer but the code still sends object data to nil cache buffer data.
2017-10-18 14:42:10 -07:00
Nitish Tiwari
8287ab091c Ignore file not found error for multipart-uploads (#5065)
Dont print the error errFileNotFound, as it is expected that concurrent
complete-multipart-uploads or abort-multipart-uploads would have deleted
the file, and the file may not be found

Fixes: https://github.com/minio/minio/issues/5056
2017-10-18 14:26:20 -07:00
Harshavardhana
f25bec6bf1 Do not attempt to generate URLToken for anonymous downloads (#5078)
This is a regression since last release - fixes #5076
2017-10-18 11:14:27 +05:30
Krishna Srinivas
75865efb0e fs: All parts except the last part should be of the same size (#5045)
fixes #4881
2017-10-17 12:01:28 -07:00
Harshavardhana
3d2d63f71e Fix gateway docs remove redundant files (#5072) 2017-10-17 08:12:44 +05:30
Harshavardhana
53e133e844 Remove NotSupported error redundant with NotImplemented (#5074) 2017-10-17 08:11:06 +05:30
Harshavardhana
b2cbade477 Support creating empty directories. (#5049)
Every so often we get requirements for creating
directories/prefixes and we end up rejecting
such requirements. This PR implements this and
allows empty directories without any new file
addition to backend.

Existing lower APIs themselves are leveraged to provide
this behavior. Only FS backend supports this for
the time being as desired.
2017-10-16 17:20:54 -07:00
Harshavardhana
0c0d1e4150 Implement backblaze-b2 gateway support (#5002)
Fixes https://github.com/minio/minio/issues/4072
2017-10-13 16:26:16 +05:30
Harshavardhana
3d0dced23c Remove go1.9 specific code for windows (#5033)
Following fix https://go-review.googlesource.com/#/c/41834/ has
been merged upstream and released with go1.9.
2017-10-13 15:31:15 +05:30
Nitish Tiwari
ad53c5d859 Remove body from POST request in webhook (#5067)
When webhook notification is configured, Minio server tries to lookup the
webhook endpoint by making a POST request with body set as releasetag.
We can remove the body from the POST request as the POST body does not
add any specific value.

This discussion on IETF group says empty POSTs are okay
http://lists.w3.org/Archives/Public/ietf-http-wg/2010JulSep/0272.html

Fixes: https://github.com/minio/minio/issues/5066
2017-10-13 13:29:01 +05:30
Harshavardhana
45463b1d6b Fix azure metadata handling for all incoming PUT requests (#5038)
s3cmd cli fails when trying to upload a file to azure gateway.
Previous fixes in azure to handle client side encryption alone
did not completely address the problem.

We need to possibilly convert all the x-amz-meta-<name>
, i.e specifically <name> should be converted into a
C# identifier as mentioned in the docs for `put-blob`.

https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob

```
s3cmd put README.md s3://myanis/
upload: 'README.md' -> 's3://myanis/README.md'  [1 of 1]
 4598 of 4598   100% in    0s    47.24 kB/s  done
upload: 'README.md' -> 's3://myanis/README.md'  [1 of 1]
 4598 of 4598   100% in    0s    50.47 kB/s  done
ERROR: S3 error: 400 (InvalidArgument): Your metadata headers are not supported.
```

There is a separate issue with s3cmd after this fix is applied where
the ETag is wronly validated https://github.com/s3tools/s3cmd/issues/880
But that is an upstream s3cmd problem which wrongly interprets ETag
to be md5sum of the content that was uploaded.
2017-10-12 12:16:24 -07:00
Nitish Tiwari
d5895d3243 Update docker swarm doc to use docker volume prune (#5053)
docker prune command is an improvement over previous two step process
of removal of unused volumes in Swarm.
2017-10-11 10:17:57 -07:00
Krishna Srinivas
db1edfe487 Fix data race bug in the testcase TestHTTPListenerAcceptParallel (#5043) 2017-10-11 10:17:37 -07:00
Bala FA
9c16f73334 pkg/http: use port 65432 than 9000 for unit tests (#5021)
Fixes #5014
2017-10-11 10:16:38 -07:00
Nitish Tiwari
f50bec2987 Update Minio version in docs and example yaml files (#5054) 2017-10-10 17:35:00 -07:00
Harshavardhana
e2bff5cdc0 Move Dockerfile to go 1.9.1 (#5047)
Additionally add Dockerfile.dev to build local development images.
2017-10-10 14:17:16 -07:00
Julien Maitrehenry
02a5f1e96a Add b2s method on pkg/disk/type_bsd.go (#5036) 2017-10-10 02:27:28 -07:00
Harshavardhana
4deefa3695 tests: Remove dependency on check.v1 (#5034)
This PR addresses a long standing dependency on
`gopkg.in/check.v1` project used for our tests.
All tests are re-written to use the go default
testing framework instead.

There was no reason for us to use an external
package where Go tools are sufficient for this.
2017-10-10 02:14:42 -07:00
Bala FA
d28b3d8801 Move to go1.9.1 as default environment. (#5041) 2017-10-09 22:23:59 -07:00
Harshavardhana
099b5293a9 Move gateway unsupported functions into a common struct. (#5009)
This is done to avoid repeated declaration of not-implemented
functions for each gateway. It also avoids a possible bug in go
https://github.com/golang/go/issues/18468 which is triggered on
our multiple PRs already.
2017-10-09 16:41:35 -07:00
Aditya Manthramurthy
4a0a491ca1 Refactor update check code (#5020)
- Add release-time conversion helpers
- Split GetCurrentReleaseTime() into two simpler functions.
- Avoid appending strings when assembling user-agent string.
- Reorder release info URLs to check the newer URLs earlier.
- Remove trivial low-level functions created solely for the purpose of
  writing tests.
- Remove some unnecessary tests.
2017-10-09 16:12:13 -07:00
Harshavardhana
db6b6e9518 S3 peers should be initialized properly (#5024)
Fixes #4991
2017-10-08 20:23:42 -07:00
Bala FA
d8a11c8f4b fix build failure for go1.9 (#4872) 2017-10-06 17:00:15 -07:00
Harshavardhana
0b546ddfd4 Return errors in PutObject()/PutObjectPart() if input size is -1. (#5015)
Amazon S3 API expects all incoming stream has a content-length
set it was superflous for us to support object layer which supports
unknown sized stream as well, this PR removes such requirements
and explicitly error out if input stream is less than zero.
2017-10-06 09:38:01 -07:00
Krishna Srinivas
d1712a46a7 Enable ListMultipartUploads and ListObjectParts for FS (#4996)
* Enable ListMultipartUploads and ListObjectParts for FS.
Previously we had disabled ListMultipartUploads and ListObjectParts
to see if any clients break. Docker registry broke. This patch
enables ListMultipartUploads and ListObjectParts, however
ListMultipartUploads with prefix based listing is not
supported (which is not used by docker registry anyway).
i.e ListMultipartUploads will need exact object name.
2017-10-05 16:08:25 -07:00
Harshavardhana
c46c2a6dd6 Fix healthcheck issue with Docker containers (#5011)
Fixes #5010
2017-10-05 13:52:57 -07:00
Bala FA
88938340b3 remove all dead codes (#5019)
Fixes #5012
2017-10-05 12:25:45 -07:00
Krishnan Parthasarathi
13a7033505 Translate s3 gateway errors at object layer (#5006) 2017-10-05 12:24:45 -07:00
Bala FA
670e3e0c8b verify: add minio gateway s3 setup for verification tests (#4976) 2017-10-04 11:16:39 -07:00
Harshavardhana
89d528a4ed Allow CopyObject() in S3 gateway to support metadata (#5000)
Fixes #4924
2017-10-03 10:38:25 -07:00
A. Elleuch
53f3d2fd65 Push max threads to little less than kernel limit (#5001)
Let Minio server use more threads than allowed by golang runtime. This
is important to better deal with high load.
2017-10-03 10:37:45 -07:00
A. Elleuch
a4f26aec00 fix: List buckets response should return UTC modtime (#5004) 2017-10-03 10:34:51 -07:00
Bala FA
60cc6184d2 azure: handle list objects properly (#4953)
When removing `minio.sys.tmp` prefixed entries and objects/prefixes is
empty, populate till we get all valid entries.
2017-09-29 12:08:23 -07:00
poornas
ce2d185211 Add maxKeys validation for azure and gcs gateway (#4999)
Gateway implementation of ListObjectsV1 does not validate maxKeys range.
Raise an InvalidArgument when maxKeys is negative so that ListObjects
call is compatible with S3 on all gateways.
2017-09-29 12:07:44 -07:00
Aditya Manthramurthy
b05351c420 Fix CopyObject with metadata for Azure gateway (#4986) 2017-09-29 10:58:40 -07:00
Harshavardhana
b415c600e1 Add bucketName checks for azure and s3 gateway in GetBucketInfo. (#4992)
Gateway interface implementations of GetBucketInfo() under
azure and s3 gateway did not perform any bucketname input
validation resulting in incorrect responses when the tests
are expecting InvalidBucketName.

Fixes #4983
2017-09-28 19:37:09 -07:00
Aditya Manthramurthy
4c9fae90ff Optimize healObject by eliminating extra data passes (#4949) 2017-09-28 15:57:19 -07:00
Aditya Manthramurthy
94670a387e Update Azure SDK (#4985) 2017-09-28 15:23:46 -07:00
Nitish Tiwari
789270af3c Vendorize latest minio-go (#4989)
As minio-go behavior is fixed to treat empty byte arrays and nil byte
arrays in the same manner. These changes are needed in minio to
address the PutObject failure for S3 Gateway.

Fixes: https://github.com/minio/minio/issues/4974,
https://github.com/minio/minio-java/issues/615
2017-09-28 08:10:38 -07:00
fangyuxiang
a5fbe1e16c fs: optimize multipart clean work (#4944) 2017-09-28 08:09:28 -07:00
Bala FA
3c836b5f34 tests: remove test cases not applicable for docker. (#4951)
When running `make test` in docker, two test cases cause hanging.
This Patch fixes the problem by removing those test cases.

Thanks to @ws141 for identifying the problem.
2017-09-27 13:51:26 -07:00
Andreas Auernhammer
02af37a394 optimize memory allocs during reconstruct (#4964)
The reedsolomon library now avoids allocations during reconstruction.
This change exploits that to reduce memory allocs and GC preasure during
healing and reading.
2017-09-27 10:29:42 -07:00
Harshavardhana
4879cd73f8 api: MakeBucket() should honor regions properly. (#4969)
Fixes #4967
2017-09-26 20:13:06 -07:00
Aditya Manthramurthy
b5dc4b5873 Fix CopyObject with metadata for GCS Gateway (#4971) 2017-09-26 11:04:42 -07:00
Harshavardhana
6dcfaa877c Fix signature v2 handling for resource names (#4965)
Previously we were wrongly adding `?` as part
of the resource name, add a test case to check
if this is handled properly.

Thanks to @kannappanr for reproducing this.

Without this change presigned URL generated with following
command would fail with signature mismatch.
```
aws s3 presign s3://testbucket/functional-tests.sh
```
2017-09-26 11:00:07 -07:00
Tamer Fahmy
0bf981278e Provide the correct free block size volume/disk information (#4943)
On *NIX platforms the statfs(2) system call returns a struct containing both the
free blocks in the filesystem (Statfs_t.Bfree) and the free blocks available to
the unprivileged or non-superuser (Statfs_t.Bavail).

The `Bfree` and `Bavail` fields (with `Bfree >= Bavail`) will be set to
different values on e.g. filesystems such as ext4 that reserve a certain
percentage of the filesystem blocks which may only be allocated by admnistrative
privileged processes.

The calculations for the `Total` disk space need to subtract the difference
between the `Bfree` and `Bavail` fields for it to correctly show the total
available storage space available for unprivileged users.

This implicitly fixes a bug where the `Used = Total - Free` calculation yielded
different (and also incorrect) results for identical contents stored when only
the sizes of the disks or backing volumes differed. (as can be witnessed in the
`Used:` value displayed in the Minio browser)

See:
- https://wiki.archlinux.org/index.php/ext4#Reserved_blocks
- http://man7.org/linux/man-pages/man2/statfs.2.html
- https://man.openbsd.org/statfs
- http://lingrok.org/xref/coreutils/src/df.c#893
2017-09-25 18:46:19 -07:00
Harshavardhana
d3eb5815d9 Avoid DDOS in PutObject() when objectName is '/' and size '0' (#4962)
It can happen that an incoming PutObject() request might
have inputs of following form eg:-

 - bucketName is 'testbucket'
 - objectName is '/'

bucketName exists and was previously created but there
are no other objects in this bucket. In a situation like
this parentDirIsObject() goes into an infinite loop.

Verifying that if '/' is an object fails on both backends
but the resulting `path.Dir('/')` returns `'/'` this causes
the closure to loop onto itself.

Fixes #4940
2017-09-25 14:47:58 -07:00
Andreas Auernhammer
7e6b5bdbb7 remove ReadFileWithVerify from StorageAPI (#4947)
This change removes the ReadFileWithVerify function from the
StorageAPI. The ReadFile was basically a redirection to ReadFileWithVerify.
This change removes the redirection and moves the logic of
ReadFileWithVerify directly into ReadFile.
This removes a lot of unnecessary code in all StorageAPI implementations.

Fixes #4946

* review: fix doc and typos
2017-09-25 11:32:56 -07:00
Harshavardhana
4cadb33da2 api/PostPolicy: Allow location header fully qualified URL (#4926)
req.Host is used to construct the final object location.

Fixes #4910
2017-09-24 16:43:21 -07:00
Harshavardhana
c3ff402fcb Fix signature v2 and presigned query unescaping. (#4936)
Simplifies the testing code by using s3signer
package from minio-go library.

Fixes #4927
2017-09-24 14:20:12 -07:00
Davor Kapsa
aceaa1ec48 travis: update go versions (#4945) 2017-09-22 14:05:07 -07:00
Harshavardhana
330f79b40e Remove pre go1.8 code and cleanup (#4933)
We don't need certain go1.7.x custom code anymore, since
we have migrated to go1.8
2017-09-22 14:03:31 -07:00
Bala FA
6d7df1d1cb Use mc functional tests for verifying build (#4934) 2017-09-20 13:22:05 -07:00
Aditya Manthramurthy
3c0d3f7510 Fix bug in ErasureStorage.HealFile (#4913) 2017-09-20 09:50:27 -07:00
Krishnan Parthasarathi
f7ae3be586 Removing 100MB part and 1TB limitation (#4939)
With https://github.com/minio/minio/pull/4869 maximum size of a single
multipart upload part in not restricted to 100MB. 1TB maximum object
size limitation is no longer applicable too.
2017-09-20 09:48:48 -07:00
Bala FA
70fec0a53f azure: add stateless gateway support (#4874)
Previously init multipart upload stores metadata of an object which is
used for complete multipart.  This patch makes azure gateway to store
metadata information of init multipart object in azure in the name of
'minio.sys.tmp/multipart/v1/<UPLOAD-ID>/meta.json' and uses this
information on complete multipart.
2017-09-19 16:08:08 -07:00
Nitish Tiwari
fba1669966 Update docker-compose sample file to use local volume driver so data (#4937)
persists even after Minio container is stopped/deleted
2017-09-19 12:43:45 -07:00
Harshavardhana
71201273e2 Remove reference to go1.8 issue from homebrew (#4931) 2017-09-19 12:41:19 -07:00
Andreas Auernhammer
79ba4d3f33 refactor ObjectLayer PutObject and PutObjectPart (#4925)
This change refactor the ObjectLayer PutObject and PutObjectPart
functions. Instead of passing an io.Reader and a size to PUT operations
ObejectLayer expects an HashReader.
A HashReader verifies the MD5 sum (and SHA256 sum if required) of the object.
This change updates all all PutObject(Part) calls and removes unnecessary code
in all ObjectLayer implementations.

Fixes #4923
2017-09-19 12:40:27 -07:00
Harshavardhana
f8024cadbb [security] rpc: Do not transfer access/secret key. (#4857)
This is an improvement upon existing implementation
by avoiding transfer of access and secret keys over
the network. This change only exchanges JWT tokens
generated by an rpc client. Even if the JWT can be
traced over the network on a non-TLS connection, this
change makes sure that we never really expose the
secret key over the network.
2017-09-19 12:37:56 -07:00
Evan
f680b8482f Add a snapcraft build badge. (#4805) 2017-09-19 12:36:35 -07:00
Nitish Tiwari
a56f7e2f23 Refactor CONTRIBUTING.md to make it more reader friendly (#4918) 2017-09-18 11:02:45 -07:00
Nitish Tiwari
6d5d49bfb1 Update CLI examples to be in sync with examples used on Minio website (#4920) 2017-09-14 19:17:42 -07:00
fangyuxiang
8e4842b665 fs: multipart clean only trigger once (#4915) 2017-09-14 19:17:26 -07:00
ebozduman
b74ef6d5f4 Fixes the if condition when uploads.json file cannot be found (#4883) 2017-09-14 16:09:12 -07:00
fangyuxiang
9925640da8 fs: multipart clean doesn't work when object name has '/' (#4919) 2017-09-14 16:00:57 -07:00
Andrej Pregl
f45e0a44b8 Change average from int to int64 in order to support 32-bit systems. (#4921) 2017-09-14 10:23:23 -07:00
Krishna Srinivas
3e632a49ee In gateway mode "continuation-token" will not contain "prefix" (#4911)
fixes #4900
2017-09-13 17:27:19 -07:00
Bala FA
1bb3a03099 build: add verify check on make test (#4901)
This patch adds basic tests for FS/XL/Distribute setups.
2017-09-12 16:56:33 -07:00
Krishna Srinivas
42b3795304 Set NextContinuationToken in ListObjectsV2 response for gateway (#4908)
fixes #4900
2017-09-12 16:19:58 -07:00
Bala FA
302fcb3b17 azure: handle encryption headers and azure InvalidMetadata error (#4893)
Previously minio gateway returns invalid bucket name error for invalid
meta data.  This is fixed by returning BadRequest with 'Unsupported
metadata' in response.

Fixes #4891
2017-09-12 16:14:41 -07:00
Harshavardhana
b9fc4150f6 Fix preInit logic when mixed disk situations exist. (#4904)
When servers are started simultaneously across multiple
nodes or simulating a local setup, it can happen such
that one of the servers in setup reaches a following
situation where it observes

 - Some servers are formatted
 - Some servers are unformatted
 - Some servers are offline

Current state machine doesn't handle this correctly, to fix
this situation where we have unformatted, formatted and
disks offline we do not decisively know the course of
action. So we wait for the offline disks to change their state.

Once the offline disks change their state to either one of these
states we can decisively move forward.

  - nil (formatted disk)
  - errUnformattedDisk
  - Or any other error such as errCorruptedDisk.

Fixes #4903
2017-09-12 12:17:44 -07:00
Krishna Srinivas
f66239e82f Expose common S3 headers in CORS setting (#4839)
fixes #4838
2017-09-11 08:15:51 -07:00
Harshavardhana
6c2bc0568b Increase default read/write timeouts from 30sec to 15minutes (#4888)
The default timeout of 30secs is not enough for high latency
environments, change these values to use 15 minutes instead.

With 30secs I/O timeouts seem to be quite common, this leads
to pretty much most SDKs and clients reconnect. This in-turn
causes significant performance problems. On a low latency
interconnect this can be quite challenging to transfer large
amounts of data. Setting this value to 15minutes covers
pretty much all known cases.

This PR was tested with `wondershaper <NIC> 20000 20000` by
limiting the network bandwidth to 20Mbit/sec. Default timeout
caused a significant amount of I/O timeouts, leading to
constant retires from the client. This seems to be more common
with tools like rclone, restic which have high concurrency set
by default. Once the value was fixed to 15minutes i/o timeouts
stopped and client could steadily upload data to the server
even while saturating the network.

Fixes #4670
2017-09-07 11:16:45 -07:00
poornas
0d154871d5 Admin: Raise error if config and env credentials mismatch (#4870) 2017-09-07 11:16:13 -07:00
Bala FA
189b6682d6 azure: allow parts > 100MiB size to work properly (#4869)
Previously if any multipart part size > 100MiB is uploaded, azure
gateway returns error.

This patch fixes the issue by creating sub parts sizing each 100MiB of
given multipart part.  On complete multipart, it fetches all uploaded
azure block ids for each parts and performs completion.

Fixes #4868
2017-09-05 16:56:23 -07:00
Harshavardhana
cf479eb401 Move to latest release of minio-go (#4886)
- Region handling can now use region endpoints directly.
- All uploads are streaming no more large buffer needed.
- Major API overhaul for CopyObject(dst, src)
- Fixes bugs present in existing code for copying
  - metadata replace directive CopyObject
  - PutObjectPart doesn't require md5Sum and sha256
2017-09-05 14:45:22 -07:00
Harshavardhana
72490bf8db Implement proper reConnect logic for amqp notification target. (#4867)
Fixes #4597
2017-09-04 17:45:30 -07:00
Justin Clift
5a73aecb5c fix: Trivial typo in error message (#4878) 2017-09-03 13:53:03 -07:00
Harshavardhana
e26a706dff Ignore reservedBucket checks for net/rpc requests (#4884)
All `net/rpc` requests go to `/minio`, so the existing
generic handler for reserved bucket check would essentially
erroneously send errors leading to distributed setups to
wait infinitely.

For `net/rpc` requests alone we should skip this check and
allow resource bucket names to be from `/minio` .
2017-09-01 12:16:54 -07:00
Andrej Pregl
9e9c7b4f22 Lower object name length when running in docker to support aufs. (#4879) 2017-09-01 11:00:47 -07:00
Krishna Srinivas
ff8e2b5b4f Init HTTP client and transport for azure sdk (#4871)
Fixes segfault
2017-08-31 17:19:03 -07:00
Frank Wessels
61e0b1454a Add support for timeouts for locks (#4377) 2017-08-31 14:43:59 -07:00
Frank Wessels
93f126364e Updated version of klauspost/reedsolomon with NEON support for ARM (#4865) 2017-08-30 09:49:00 -07:00
Harshavardhana
6dca044ea8 fs: Convert repeated code in rwpool.Open() into a single function. (#4864)
Refer https://github.com/minio/minio/issues/4658 for more information.
2017-08-30 09:48:19 -07:00
Harshavardhana
9e9faf7b53 Support source compilation on s390 and ppc64le (#4859)
NOTE: This doesn't validate that minio will work fine
on these platforms and is tested. Since we do not validate
on these architectures this is to be treated as just a build fix.

Fixes #4858
2017-08-30 09:45:36 -07:00
Nitish Tiwari
2bca51ab2c Add steps to run GCS gateway on Kubernetes via YAML files (#4819) 2017-08-28 12:58:52 -07:00
Leo Arias
34e780e690 add the mount-observe plug to the snap (#4853) 2017-08-28 11:40:26 -07:00
Harshavardhana
6cab6d802d api: Fix the conditional to check for reserved buckets. (#4856)
Current code was an logical `and` instead we should do `or`.

Fixes https://github.com/minio/mc/issues/2231
2017-08-28 11:39:48 -07:00
Andreas Auernhammer
ea65350308 add vscode config files to gitignore (#4855) 2017-08-27 13:44:39 -07:00
Harshavardhana
1bb9d49eaa fs: ListObjects() was reading ETag at wrong offsets (#4846)
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes #4842
2017-08-23 17:59:14 -07:00
Harshavardhana
db5af1b126 fix: tests error conditions should be used properly. (#4833) 2017-08-23 17:58:52 -07:00
Andreas Auernhammer
b233345f19 remove bcrypt code from code-base (#4844) (#4845)
Bcrypt is not neccessary and not used properly. This change
replace the whole bcrypt hash computation through a constant time
compare and removes bcrypt from the code base.
2017-08-23 15:59:37 -07:00
Leo Arias
07ccf4c0d0 Add the install instructions for the snap (#4843) 2017-08-23 15:59:13 -07:00
Aditya Manthramurthy
77d2870f5b Fix validation in PutBucketNotification handler (#4841)
Fixes #4813

If a TopicConfiguration element or CloudFunction element is found in
configuration submitted to PutBucketNotification API, an BadRequest
error is returned.
2017-08-23 15:58:02 -07:00
Andreas Auernhammer
3a73c675a6 restirct max size of http header and user metadata (#4634) (#4680)
S3 only allows http headers with a size of 8 KB and user-defined metadata
with a size of 2 KB. This change adds a new API error and returns this
error to clients which sends to large http requests.

Fixes #4634
2017-08-22 16:53:35 -07:00
Bala FA
b694c1a4d7 fix: bufconn and listener tests for megacheck (#4827)
Fixes #4824
2017-08-20 12:25:08 -07:00
Harshavardhana
2e6ee68409 fix: [minor] Avoid unnecessary typecasting. (#4828)
We don't need to typecast identifiers from
their base to type to same type again. This
is not a bug and compiler is fine to skip
it but it is better to avoid if not needed.
2017-08-18 11:45:16 -07:00
Bala FA
7505bac037 tests: create temporary dir/files than /usr directory. (#4820)
Fixes #4816
2017-08-18 11:44:54 -07:00
Harshavardhana
9dca0c1889 fix: [minor] functions should take inputs with required functionality. (#4823) 2017-08-17 13:49:57 -07:00
Nitish Tiwari
69555f1224 Update Docker commands to use /data as example directory (#4825)
/data as default makes it easy to understand and shortens
the example Minio command for Docker.
2017-08-17 10:56:25 -07:00
Harshavardhana
879cef37a1 Fail to start server if detected cross-device mounts. (#4807)
Fixes #4764
2017-08-15 15:10:50 -07:00
wd256
3d21119ec8 Provide 200 response with per object error listing on access denied for delete multiple object request (#4817) 2017-08-15 12:49:31 -07:00
Frank Wessels
a2f2044528 Minor corrections in comments for xl utils (#4815) 2017-08-14 18:09:29 -07:00
Andreas Auernhammer
85fcee1919 erasure: simplify XL backend operations (#4649) (#4758)
This change provides new implementations of the XL backend operations:
 - create file
 - read   file
 - heal   file
Further this change adds table based tests for all three operations.

This affects also the bitrot algorithm integration. Algorithms are now
integrated in an idiomatic way (like crypto.Hash).
Fixes #4696
Fixes #4649
Fixes #4359
2017-08-14 18:08:42 -07:00
Leo Arias
617f2394fb Hardcode snap version while a store bug is fixed (#4806)
* Hardcode snap version while a store bug is fixed
This is a workaround for for a failure in the store, which limits the version to 32 characters.
https://forum.snapcraft.io/t/versions-can-be-at-most-32-characters/1642
Capitalize the summary
* add the snapcraft dirs to gitignore bring back settings.json to ignore
2017-08-14 11:12:46 -07:00
Bala FA
1729e82361 tests: use port '0' for auto-detecting free port. (#4803)
Fixes #4774
2017-08-14 11:11:38 -07:00
Harshavardhana
58633a383a Deprecate intel-32, arm64, arm support for minio builds. (#4811)
It was decided that we will be deprecating ARM support
for minio builds. ARM users should simply compile from source.

Additionally 32bit version of Linux, Windows and FreeBSD (64bit)
are deprecated.
2017-08-14 11:08:58 -07:00
Nitish Tiwari
d4b107adf4 Retry name lookup for kubernetes and docker swarm environment (#4800)
Wait for remote hosts to resolve instead of failing on first host
resolution error, when running in Kubernetes or Docker environment.

Note that

- Waiting is based on exponential back-off mechanism
- If run as a binary, server fails if remote host is not resolvable

This is needed because in orchestration platforms like Kubernetes, remote
hosts are started sequentially and all the hosts are not up initially,
though they are expected to come up in a short time frame
It is difficult to identify a cap on the waiting time due to
non-deterministic nature of infrastructure platforms, so the server waits
infinitely for the hosts to come up, while logging the error messages to
the console.

Fixes: https://github.com/minio/minio/issues/4669
2017-08-13 13:34:10 -07:00
Nitish Tiwari
53f84d6084 Report healthy status for initial 120s, then switch to healthcheck (#4799)
Swarm routes traffic only to containers that report healthy status,
while Minio in distributed mode needs to talk to other peers before
it can respond to healthcheck probe. As the Minio containers are not
able to talk to each other, distributed Minio is not getting started
on Docker Swarm.

With this PR, Minio Healthcheck report healthy status for initial
120s enough for distributed Minio to start. After that normal
Healthcheck resumes. Also changed the healthcheck method name
in accordance with Google shell styleguide.

Fixes: https://github.com/minio/minio/issues/4761
2017-08-12 19:26:35 -07:00
Harshavardhana
d864e00e24 posix: Deprecate custom removeAll/mkdirAll implementations. (#4808)
Since go1.8 os.RemoveAll and os.MkdirAll both support long
path names i.e UNC path on windows. The code we are carrying
was directly borrowed from `pkg/os` package and doesn't need
to be in our repo anymore. As a side affect this also
addresses our codecoverage issue.

Refer #4658
2017-08-12 19:25:43 -07:00
Harshavardhana
b69aa9c4d0 fs: Return errVolumeNotEmpty properly if path not empty. (#4794)
Refer #4770
2017-08-12 19:24:20 -07:00
Harshavardhana
d20cdac304 Move all Dockerfiles to alpine-3.6 (#4797)
Refer #4759.

Fix this to avoid issues like below

```
docker run --rm minio/minio:edge-armhf version
minio: <ERROR> .... is not compiled by Go >= 1.8.3. ... recompile...
```
2017-08-12 19:17:37 -07:00
Frank Wessels
fffe4ac7e6 Prevent unnecessary verification of parity blocks while reading (#4683)
* Prevent unnecessary verification of parity blocks while reading erasure
  coded file.
* Update klauspost/reedsolomon and just only reconstruct data blocks while
  reading (prevent unnecessary parity block reconstruction)
* Remove Verification of (all) reconstructed Data and Parity blocks since
  in our case we are protected by bit rot protection. And even if the
  verification would fail (essentially impossible) there is no way to
  definitively say whether the data is still correct or not, so this call
  make no sense for our use case.
2017-08-11 18:25:46 -07:00
Frank Wessels
98b62cbec8 Implement an offline mode for a distributed node (#4646)
Implement an offline mode for remote storage to cache the
offline status of a node in order to prevent network calls
that are bound to fail. After a time interval an attempt
will be made to restore the connection and mark the node
as online if successful.

Fixes #4183
2017-08-11 11:49:35 -07:00
Dee Koder
1978b9d8f9 Prevent minio server starting in standalone erasure mode for wrong inputs. (#4700)
It is possible at times due to a typo when distributed mode was intended
a user might end up starting standalone erasure mode causing confusion.
Add code to check this based on some standard heuristic guess work and
report an error to the user.

Fixes #4686
2017-08-11 11:47:28 -07:00
Harshavardhana
3544e5ad01 fs: Fix Shutdown() behavior and handle tests properly. (#4796)
Fixes #4795
2017-08-10 14:11:57 -07:00
Harshavardhana
e7cdd8f02c fs: Avoid non-idempotent code flow in ListBuckets() (#4798)
Under the call flow

```
Readdir
   +
   |
   |
   | path-entry
   |
   |
   v
StatDir
```

Existing code was written in a manner where say
a bucket/top-level directory was indeed deleted
between Readdir() and before StatDir() we would
ignore certain errors. This is not a plausible
situation and might not happen in almost all
practical cases. We do not have to look for
or interpret these errors returned by StatDir()
instead we can just collect the successful
values and return back to the client. We do not
need to pre-maturely decide on bucket access
we just let filesystem decide subsequently for
real I/O operations.

Refer #4658
2017-08-10 13:36:11 -07:00
Leo Arias
3ff09b9b72 Add the packaging metadata to build the minio snap (#4749)
* Add the packaging metadata to build the minio snap.
2017-08-10 09:39:30 -07:00
Nitish
b3b42c72a9 Update orchestration examples to latest release 2017-08-08 17:15:46 -07:00
Aditya Manthramurthy
32da1aa9d6 XL: Simplify heal-format operations
This is in preparation for updated admin heal API.

* Improve case analysis of healFormatXL() - fixes a case where disks
  could have unhandled errors.

* Simplify healFormatXLFreshDisks() and healFormatXLCorruptedDisks()
  to share more code and handle fewer cases for improved simplicity
  and reduced code repetition.

* Fix test cases.
2017-08-08 17:14:24 -07:00
Andreas Auernhammer
b10fa507b2 set http transport config for gateway (#4765)
This change sets the http config for the minio client used by the
minio server in gateway mode.

Fixes #4765
2017-08-08 16:23:52 -07:00
Andreas Auernhammer
1e8925cea0 add settings.json to .gitignore (#4786)
settings.json is the VS code workspace settings file
2017-08-08 13:06:57 -07:00
Harshavardhana
f346ca44f0 config: Avoid stale credentials in memory. (#4466) 2017-08-08 12:14:32 -07:00
A. Elleuch
6f7ace3d3e Honor overriding response headers for HEAD (#4784)
Though not clearly mentioned in S3 specification, we should override
response headers for presigned HEAD requests as we do for GET.
2017-08-08 11:04:04 -07:00
Harshavardhana
812142f007 Fix typo in webhook docs (#4787) 2017-08-07 21:31:17 -07:00
Andrej Pregl
fa52d491c5 Added support for macOS in TestNewHTTPListener (#4782) 2017-08-07 16:02:34 -07:00
Harshavardhana
283ad7e5e5 browser: update ui-assets for new changes. (#4780) 2017-08-07 12:48:51 -07:00
poornas
748b1d6495 azure: For container access type private treat as no policy set. (#4729) 2017-08-06 22:24:40 -07:00
A. Elleuch
b4dc6df35c go1.8: Changes to support golang 1.8 (#4759)
QuirkConn is added to replace net.Conn as a workaround to a golang bug:
https://github.com/golang/go/issues/21133
2017-08-06 11:27:33 -07:00
Aditya Manthramurthy
218049300c Fix testcase to not overflow int type (#4739)
The int type is only 32-bits wide on 32-bit CPUs.

Set the type in the tests to int32 to avoid setting problematic
maxKeys values.

Fixes #4738
2017-08-05 02:36:47 -07:00
Aaron Walker
5db533c024 bucket-policy: Add IPAddress/NotIPAddress conditions support (#4736) 2017-08-05 01:00:05 -07:00
Brendan Ashworth
aeafe668d8 posix: do not upstream errors in deleteFile (#4771)
This commit changes posix's deleteFile() to not upstream errors from
removing parent directories. This fixes a race condition.

The race condition occurs when multiple deleteFile()s are called on the
same parent directory, but different child files. Because deleteFile()
recursively removes parent directories if they are empty, but
deleteFile() errors if the selected deletePath does not exist, there was
an opportunity for a race condition. The two processes would remove the
child directories successfully, then depend on the parent directory
still existing. In some cases this is an invalid assumption, because
other processes can remove the parent directory beforehand. This commit
changes deleteFile() to not upstream an error if one occurs, because the
only required error should be from the immediate deletePath, not from a
parent path.

In the specific bug report, multiple CompleteMultipartUpload requests
would launch multiple deleteFile() requests. Because they chain up on
parent directories, ultimately at the end, there would be multiple
remove files for the ultimate parent directory,
.minio.sys/multipart/{bucket}. Because only one will succeed and one
will fail, an error would be upstreamed saying that the file does not
exist, and the CompleteMultipartUpload code interpreted this as
NoSuchKey, or that the object/part id doesn't exist. This was faulty
behavior and is now fixed.

The added test fails before this change and passes after this change.

Fixes: https://github.com/minio/minio/issues/4727
2017-08-04 16:51:20 -07:00
Krishnan Parthasarathi
54f3a0946f Avoid superfluous error messages after connect (#4762)
Peek could fail legitimately when clients abruptly close connection. So,
io.EOF and network timeout errors are not logged while all other errors
will be logged.
2017-08-04 14:35:07 -07:00
Krishna Srinivas
be6bd52978 fs: use keymarker and uploadidmarker in the multipart purging loop (#4775)
related to #4564
2017-08-04 14:14:23 -07:00
Krishnan Parthasarathi
75c43bfb6c ListMultipartUploads, ListObjectParts return empty response (#4694)
Also, periodically removes incomplete multipart uploads older than 2 weeks.
2017-08-04 10:45:57 -07:00
ebozduman
0aca2ab970 Stop attempting to close nil Listener (#4753) 2017-08-04 10:44:46 -07:00
Brendan Ashworth
28bc5899fd posix: test isDirEmpty, change error conditional (#4743)
This commit adds a new test for isDirEmpty (for code coverage) and
changes around the error conditional. Previously, there was a `return
nil` statement that would only be triggered under a race condition and
would trip up our test coverage for no real reason. With this new error
conditional, there's no awkward 'else'-esque condition, which means test
coverage will not change between runs for no reason in this specific
test. It's also a cleaner read.
2017-08-04 10:43:51 -07:00
Nitish Tiwari
fcc61fa46a Remove minimum inodes reqd check (#4747) 2017-08-03 20:07:22 -07:00
Nitish Tiwari
f7889e1af7 Add TLS request for healthcheck (#4740) 2017-08-03 20:05:45 -07:00
Brendan Ashworth
bccc386994 fs: drop Stat() call from fsDeleteFile,deleteFile (#4744)
This commit makes fsDeleteFile() simply call deleteFile() after calling
the relevant path length checking functions. This DRYs the code base.

This commit removes the Stat() call from deleteFile(). This improves
performance and removes any possibility of a race condition.

This additionally adds tests and a benchmark for said function. The
results aren't very consistent, although I'd expect this commit to make
it faster.
2017-08-03 20:04:28 -07:00
ebozduman
0f401b67ad Removes max limit requirement on accessKey and secretKey length (#4730) 2017-08-03 20:03:37 -07:00
Laurentiu Nicola
108decfa76 Fix sysctl proposed values (#4741)
`sched_wakeup_granularity_ns` and `sched_wakeup_granularity_ns` are measured in `ns`, so a value of `10` or `15` is way too low.
2017-08-02 14:08:58 -07:00
Nitish Tiwari
6afbd502e8 Add Docker command in erasure code document (#4735) 2017-07-31 11:40:14 -07:00
Brendan Ashworth
ec5293ce29 jwt,browser: allow short-expiry tokens for GETs (#4684)
This commit fixes a potential security issue, whereby a full-access
token to the server would be available in the GET URL of a download
request. This fixes that issue by introducing short-expiry tokens, which
are only valid for one minute, and are regenerated for every download
request.

This commit specifically introduces the short-lived tokens, adds tests
for the tokens, adds an RPC call for generating a token given a
full-access token, updates the browser to use the new tokens for
requests where the token is passed as a GET parameter, and adds some
tests with the new temporary tokens.

Refs: https://github.com/minio/minio/pull/4673
2017-07-24 12:46:37 -07:00
Harshavardhana
4785555d34 api: Upon bucket delete remove in-memory state properly. (#4716)
This PR fixes the issue of cleaning up in-memory state
properly. Without this PR we can lead to security
situations where new bucket would inherit wrong
permissions on bucket and expose objects erroneously.

Fixes #4714
2017-07-23 19:35:18 -07:00
A. Elleuch
b918a6592f gcs: Better parsing of address flag (#4709) 2017-07-20 16:39:11 -07:00
Krishna Srinivas
eb787d8613 gateway-gcs: remove files older than 2 weeks in minio.sys.temp (#4599).
Rename ##minio## to {minio}.
2017-07-20 15:36:48 -07:00
Aaron Kunz
0a1501bc1b Fix typo (#4695) 2017-07-19 15:19:03 -07:00
Harshavardhana
f8bd9cfd83 rpc: Do not use read/write deadlines for rpc connections. (#4647)
Fixes #4626
2017-07-18 09:30:46 -07:00
Brendan Ashworth
c59b995f7b build: ditch verifiers on make (#4679)
This commit ditches running verifiers automatically when just building
the server. It retains the verifiers when running tests.

There is very little point to running the verifiers each time a
developer builds the library but has no intent of running the tests.
They're expensive in time; this commit halves the build time on my
system, from 57 seconds to 29 seconds. This is because verifiers updates
the libraries from GitHub each time, which is slightly wasteful.
Additionally, computing cyclomatic complexity is expensive
computationally and isn't necessary to build the library.

Additionally, this allows the library to be built offline. It no longer
requires internet to run make.
2017-07-15 12:12:03 -07:00
Harshavardhana
bc73a1a1cb gcs: Save partNumber as part of backend format. (#4666)
Fixes #4637
2017-07-13 23:20:16 -07:00
Krishna Srinivas
ce7c9c651d gateway-azure: Return right error when Part size is > 100MB (#4652) 2017-07-12 16:42:14 -07:00
Bala FA
c3dd7c1f6c Refactor HTTP server to address bugs (#4636)
* Refactor HTTP server to address bugs
* Remove unnecessary goroutine to start multiple TCP listeners.
* HTTP server waits for shutdown to maximum of Server.ShutdownTimeout
  than per serverShutdownPoll.
* Handles new connection errors properly.
* Handles read and write timeout properly.
* Handles error on start of HTTP server properly by exiting minio
  process.

Fixes #4494 #4476 & fixed review comments
2017-07-12 16:33:21 -07:00
Harshavardhana
2d23cd4f39 gcs: Fetch port as GlobalString(). (#4657)
Currently we were looking for `address` flag
under local flags. This PR fixes #4656
2017-07-11 18:06:26 -07:00
Harshavardhana
ce7af3aae1 gcs: Fix writer/reader go-routine leaks and code re-use (#4651)
This PR serves to fix following things in GCS gateway.

- fixes leaks in object reader and writer, not getting closed
  under certain situations. This led to go-routine leaks.

- apparent confusing issue in case of complete multipart upload,
  where it is currently possible for an entirely different
  object name to concatenate parts of a different object name
  if you happen to know the upload-id and parts of the object.
  This is a very rare scenario but it is possible.

- succint usage of certain parts of code base and re-use.
2017-07-11 09:25:19 -07:00
Krishna Srinivas
1b92c5136b Append "-1" to etag when it is not MD5 (#4641)
* gateway-azure: append "-1" to ETag so that clients do not interpret it as MD5. fixes #4537. Added unit tests.
2017-07-10 18:21:12 -07:00
Harshavardhana
cc8a8cb877 posix: Check for min disk space and inodes (#4618)
This is needed such that we don't start or
allow writing to a posix disk which doesn't
have minimum total disk space available.

One part fix for #4617
2017-07-10 18:14:48 -07:00
Krishna Srinivas
ce403fdaa0 GCS documentation (#4622)
* GCS documentation and review fixes.
2017-07-10 09:35:20 -07:00
Nitish Tiwari
45fbb0d618 Add NATS Streaming doc to event notification doc (#4645) 2017-07-07 23:37:12 -07:00
Harshavardhana
f5ce685aa1 Remove dead unused errs and constants. (#4627) 2017-07-07 14:31:42 -07:00
Krishna Srinivas
c83055500d fs: Fail CompleteMultipartUpload if partSize < 5M unless it is last part (#4642)
fixes #4625
2017-07-07 08:41:29 -07:00
Andreas Auernhammer
b0fbddc051 fix confusing code for http.Header handling (#4623)
Fixed header-to-metadat extraction. The extractMetadataFromHeader function should return an error if the http.Header contains a non-canonicalized key. The reason is that the keys can be manually set (through a map access) which can lead to ugly bugs.
Also fixed header-to-metadata extraction. Return a InternalError if a non-canonicalized key is found in a http.Header. Also log the error.
2017-07-05 16:56:10 -07:00
Harshavardhana
4e0c08e9c5 ListenBucketNotification should set proper MIME type. (#4621)
This is needed to avoid proxies buffering the connection
this is also a HTTP standard way to handle this situation
where server is sending back events in asynchronously.

For more details read https://goo.gl/RCML9f

Fixes - https://github.com/minio/minio-go/issues/731
2017-07-03 19:59:41 -07:00
Nitish Tiwari
344f9ec608 Fix gateway browser screenshot (#4613) 2017-06-30 09:47:40 -07:00
Dee Koder
28ff62716f Fixed one of the images with missing details. (#4612) 2017-06-29 15:49:09 -07:00
Dee Koder
1f69a75efa Updated docs with latest images. (#4611) 2017-06-29 11:41:21 -07:00
Nitish Tiwari
02a81ee564 Remove deployment scenarios from erasure code guide (#4607) 2017-06-29 11:40:59 -07:00
Nitish Tiwari
e91e9e8a38 GCS ListObjectV2 honours continuationToken (#4608) 2017-06-29 11:19:55 -07:00
Dee Koder
e45f9057d6 Added new optimized images. (#4609) 2017-06-29 09:10:47 -07:00
Bala FA
53e7fdc847 create subcomposed objects if total parts are > 32 (#4593) 2017-06-27 22:27:05 -07:00
Aditya Manthramurthy
1af331c05c Remove unnecessary newline at beginning of server output (#4600) 2017-06-27 19:46:58 -07:00
ebozduman
0e5b9c7fe4 Adds 'gcs not ready for production' msg (#4604) 2017-06-27 19:44:47 -07:00
Harshavardhana
a86dc8a4c5 cleanup makefile and enable CGO_ENABLED=0 (#4598) 2017-06-26 18:07:06 -07:00
A. Elleuch
c88dca984d web: Encode path in presigned GET urls (#4596)
When the browser asks for a GET presigned url, this latter is not
encoded and can be confusing when the user copies-pastes it somewhere,
especially when the path contains a space.
2017-06-25 18:39:14 -07:00
Harshavardhana
1054f9cbf0 fix: Remove adverb from erasure coded startup message. (#4594) 2017-06-25 18:38:55 -07:00
Harshavardhana
aaacce4c43 browser: update ui-assets with new changes. (#4595) 2017-06-25 18:09:24 -07:00
Nitish Tiwari
7bd1f44491 Add support for helm package info in useragent (#4592) 2017-06-24 13:17:28 -07:00
Harshavardhana
ba6d997d18 build: Error out properly when unsupported arch is given. (#4585) 2017-06-24 01:05:35 -07:00
A. Elleuch
eaa41e4086 gcs: Check if the given project id argument exists (#4583)
Using GCS resource manager API, check if the provided
project id is already created and associated to the current
user account.
2017-06-23 22:10:29 -07:00
Krishna Srinivas
6b70f429ed gateway/azure: Parse error responses for anonymous requests (#4543)
fixes #4481
2017-06-23 22:07:46 -07:00
Krishna Srinivas
0a6e9a1834 gateway-gcs: cleanup minio.sys.temp before deleting the bucket (#4582)
fixes #4560
fixes #4569
2017-06-23 17:57:25 -07:00
Nitish Tiwari
15b65a8342 Added AnonListObjectsV2 support to GCS (#4584) 2017-06-23 17:35:45 -07:00
Harshavardhana
8b7df7da37 api: No need to set x-amz-bucket-region if region is empty. (#4586) 2017-06-23 16:05:40 -07:00
Krishnan Parthasarathi
237c8af5ef Improve GCS gateway example instruction (#4587) 2017-06-23 13:13:26 -07:00
Krishna Srinivas
ff036c171f gateway-gcs: double quotes should be striped from ETag (#4590) 2017-06-23 12:19:10 -07:00
Harshavardhana
a3b085300d gcs: Add missing AnonListObjectsV2 2017-06-22 12:09:13 -07:00
Harshavardhana
b90cefdb88 Merge remote-tracking branch 'origin/feature-gcs' 2017-06-22 11:52:12 -07:00
Harshavardhana
f3506b8958 tests: Enable previously disabled UNC tests on CI. (#4575)
Windows CI had a bug which has been fixed, re-enable
those commented tests.
2017-06-22 07:54:22 -07:00
Frank Wessels
46897b1100 Name return values to prevent the need (and unnecessary code bloat) (#4576)
This is done to explicitly instantiate objects for every return statement.
2017-06-21 19:53:09 -07:00
Harshavardhana
cec8b238f3 sign: StreamingSign should use region from client. (#4577)
This is a fix to make streaming signature to behave
the same as regular signature and presigned signature.

Fixes https://github.com/minio/minio-go/issues/718
2017-06-21 11:30:34 -07:00
Krishna Srinivas
13ab8e17e2 gateway-gcs: use minio.sys.temp/multipart/v1 as url base (#4562) 2017-06-21 10:27:44 -07:00
Nitish Tiwari
9d3d64df2d Added HEALTHCHECK in release Dockerfiles (#4550)
A new script `healthcheck.sh` checks for http status 200/404
2017-06-20 19:24:45 -07:00
Krishnan Parthasarathi
146bc3e638 Add MINIO_REGION to server help message (#4558)
* Add e.g for setting MINIO_REGION env variable
* Add MINIO_REGION to region table
2017-06-20 15:02:18 -07:00
Harshavardhana
0543d45fb3 fix: OwnerID in response should be 64 character in length. (#4554)
Rather than sending a custom "minio" string, we can
change this to `sha256('arn:aws:iam::minio:user/admin')`.

Fixes #4553
2017-06-20 15:01:13 -07:00
Krishnan Parthasarathi
fe426944ea Fix GCS help message (#4570) 2017-06-20 14:25:16 -07:00
Aditya Manthramurthy
c1a6ca0c33 Fix spelling of function name to startLockMaintenance (#4561) 2017-06-20 12:10:02 -07:00
Harshavardhana
5a78266821 gateway/gcs: Complete minio browser support for gcs. (#4552)
Fixes #4460
2017-06-19 19:45:13 -07:00
Harshavardhana
f5b4b0765a Update minio-go dependency (#4551)
This updates dependency for

 - AWS S3 backend.
 - pkg/madmin

```
- Relax isValidBucketName to allow reading existing buckets. (#708) (3 minutes ago) <Harshavardhana>
- For GCS the size limit of S3 is not useful. (#711) (3 days ago) <Harshavardhana>
- s3utils: Support AWS S3 US GovCloud endpoint. (#701) (3 days ago) <Harshavardhana>
- api: Always strip 80/443 port from host (#709) (3 days ago) <Anis Elleuch>
- Redact signature strings properly. (#706) (4 days ago) <Harshavardhana>
- api: Single putObject can use temporary file always. (#703) (6 days ago) <Harshavardhana>
- Spelling fix (#704) (7 days ago) <Jacob Taylor>
- api/encrypt: Get() on encrypted object should be a reader. (#699) (2 weeks ago) <Harshavardhana>
- get: Fix reading an object if its size is unknown (#694) (3 weeks ago) <Anis Elleuch>
- fixes #696 by updating the examples for put-encrypted-object and get-encrypted-object (#697) (3 weeks ago) <Tejay Cardon>
- fix InvalidAccessKeyId error according to amazon documentation (#692) (4 weeks ago) <samkevich>
- Add AWS S3 SSE-C example. (#689) (4 weeks ago) <Harshavardhana>
- According to RFC7232 Etag should be in quotes for If-Match. (#688) (5 weeks ago) <Harshavardhana>
- api: getReaderSize() should honor seeked file descriptors. (#681) (5 weeks ago) <Harshavardhana>
- tests: Use bytes.Repeat() when generating big data (#683) (5 weeks ago) <Anis Elleuch>
- api: Failed call retry with region only when http.StatusBadRequest. (#678) (5 weeks ago) <Harshavardhana>
- api: Add NewWithCredentials() (#646) (5 weeks ago) <Harshavardhana>
```
2017-06-19 16:02:35 -07:00
Krishna Srinivas
3928c1e14c gateway/gcs: Change in multipart backend format (#4455) 2017-06-17 16:00:41 -07:00
Krishnan Parthasarathi
94241cd153 Update Minio on DC/OS doc for latest Minio package (#4549) 2017-06-17 12:21:37 -07:00
Harshavardhana
a86c2e2ce1 xl/fs: Return InvalidPart{} error for part ETag mismatch. (#4541)
Fixes #4539
2017-06-17 11:20:39 -07:00
Harshavardhana
e99244be02 xl: prepare storage should Abort properly. (#4542)
Current state-machine didn't honor a situation
which can arise when there is a combination of

 - formatted
 - unformatted
 - corrupted

disks - this combination invariably goes into a
mode where all servers are waiting perpetually
forever thinking we will get quorum in future.

At this point there is a distant possibility of
ever getting a quorum since we don't even have
quorum number of disks offline.

We should exit and print a proper message per disk
to indicate what went wrong and what was detected
by the server.

Refer #4477
2017-06-17 11:20:12 -07:00
Nitish Tiwari
58833711e0 Added ListObjectsV2 and ListObjectsV2 Anon support to Gateway S3 and Azure. (#4547) 2017-06-16 22:17:00 -07:00
Harshavardhana
f99f218999 Add support for reading and saving config on Gateway. (#4463)
This is also a first step towards supporting bucket
notification for gateway.
2017-06-16 16:01:41 -07:00
Krishnan Parthasarathi
4fb5fc72d7 GCS gateway allows apps to supply their own marker (#4495)
Most s3 compatible apps use object keys returned in listing as
marker. This change allows this behaviour with gateway-gcs too.
2017-06-16 15:02:07 -07:00
Remco Verhoef
d86973dcca Allow bucket creation in different regions, closes #4287 and #4241
* I needed to remove the region check from PutBucketHandler
2017-06-16 15:02:07 -07:00
Krishnan Parthasarathi
8085ba4494 Filter out internal object prefix during listing (#4435)
We use ZZZZ-Minio/ prefix internally in our GCS gateway which should be
filtered out in the response to ListObjects.
2017-06-16 15:02:07 -07:00
poornas
9bd0eb1a9e Set default ETag value if vendor returns empty md5 string (#4409)
The ETag is constructed from md5 atttribute of object attributes
returned by the vendor's Composer. The md5 attribute comes back
as nil for large uploads. Instead the CRC32C should be used.

Refer to https://cloud.google.com/storage/docs/hashes-etags

Fixes #4397
2017-06-16 15:02:07 -07:00
Anis Elleuch
e4e0abfc05 fix: Check project id before starting gateway (#4412) 2017-06-16 15:02:07 -07:00
poornas
12b2fc894b Remove profile option for gcs from gateway help message (#4421) 2017-06-16 15:01:34 -07:00
Krishna Srinivas
2aa76e7407 Change md5Sum to etag (#4399) 2017-06-16 14:58:49 -07:00
Remco Verhoef
0dab038858 Cleanup and update the PR with the master branch. 2017-06-16 14:55:32 -07:00
Remco Verhoef
a76556ec1b Map only default region us-east-1 to gcs us region 2017-06-16 14:54:37 -07:00
Harshavardhana
91c7bb65c5 gateway/gcs: send proper error responses for Get/SetBucket policies. (#4338)
Fixes #4323
2017-06-16 14:54:37 -07:00
Anis Elleuch
5d602034ea gateway: Use default params when no args provided (#4315)
For S3 & Azure, use default parameters when no arguments (endpoint) are
provided. This also avoids a crash.
2017-06-16 14:54:37 -07:00
Nitish Tiwari
b829ec4a6b Fixes https://github.com/minio/minio/issues/4320 (#4332)
- Add description for error ErrBucketAlreadyExists
2017-06-16 14:54:37 -07:00
Remco Verhoef
9c50a9f567 Fix ListObjectParts to list properly all parts - closes #4322 2017-06-16 14:54:37 -07:00
Remco Verhoef
52122c0309 Fix uploadIDMarker handling. 2017-06-16 14:54:37 -07:00
Remco Verhoef
3b9d313c87 Fix issue with AbortMultipartUpload, closes #4322 2017-06-16 14:54:37 -07:00
Remco Verhoef
bfff251e2a Fix issue with UNSIGNED payloads.
Additionally also fixes escaping slashes in
temporary multipart names
2017-06-16 14:54:37 -07:00
Remco Verhoef
52b500cce9 Verify multipart etag during complete, closes #4288 2017-06-16 14:54:37 -07:00
Remco Verhoef
c63cdca11f Support iterating through ListObjectParts using NextPartNumberMarker, closes #4284 2017-06-16 14:54:37 -07:00
Remco Verhoef
4430085981 Add region to gcs gateway example 2017-06-16 14:54:37 -07:00
Remco Verhoef
5c78415b31 Verify md5 content hash, closes #4285 2017-06-16 14:54:37 -07:00
Remco Verhoef
bd67117756 Use maxKeys for iterator 2017-06-16 14:54:37 -07:00
Remco Verhoef
f3e5e9fb29 Support marker, closes #4286 2017-06-16 14:54:37 -07:00
Remco Verhoef
2de1921fe8 Use MINIO_REGION environment variable for region configuration, closes #4287 2017-06-16 14:54:37 -07:00
Remco Verhoef
dd7e47f264 Add access and secret key to example, needed to access Minio Gateway 2017-06-16 14:54:37 -07:00
Remco Verhoef
fe9d826bef Implement bucket policies 2017-06-16 14:53:36 -07:00
Remco Verhoef
6dbc5aba09 Return correct error when PutObject fails 2017-06-16 14:53:36 -07:00
Remco Verhoef
de5374f74c Map S3 regions to Google (multi)regions 2017-06-16 14:53:36 -07:00
Remco Verhoef
bf55591c64 Make every backend responsible for parsing its own arguments, fixes #4293 2017-06-16 14:53:36 -07:00
Remco Verhoef
2d814e340f Return BucketAlreadyExists when bucket exists with another user 2017-06-16 14:53:36 -07:00
Remco Verhoef
0a8cf1a6b0 Allow bucket creation in different regions, closes #4287 and #4241
* I needed to remove the region check from PutBucketHandler
2017-06-16 14:53:36 -07:00
Remco Verhoef
07949f68d8 Translate gcs errors to S3 compatible errors, fixes #4278 2017-06-16 14:53:36 -07:00
Remco Verhoef
909a89647b Use default endpoint when not supplied 2017-06-16 14:53:36 -07:00
Remco Verhoef
6508da5fde Add usage for GCS gateway, closes #4280 2017-06-16 14:53:36 -07:00
Remco Verhoef
3379f005a5 Initial implementation of Google Cloud Storage 2017-06-16 14:47:02 -07:00
Remco Verhoef
4be609eb82 Added AllAccessDisabled error 2017-06-16 14:47:02 -07:00
Remco Verhoef
dd5b975001 Add comment, gateway should validate object name 2017-06-16 14:47:02 -07:00
Remco Verhoef
9ac3538141 Move anonymous error to object translation from Azure specific to gateway 2017-06-16 14:47:02 -07:00
Remco
ace4f9fd15 Implement gateway support Google Cloud Storage 2017-06-16 14:47:02 -07:00
Nitish Tiwari
b283a2c21f Bump docs references to latest Minio release RELEASE.2017-06-13T19-01-01Z (#4546) 2017-06-15 14:20:36 -07:00
Anand Babu (AB) Periasamy
ed37bf3703 Reduce macOS instructions 2017-06-14 23:26:33 -07:00
Rushan
d0f18dc1b5 browser: Use custom input number step-up/down function (#4524) 2017-06-14 17:34:11 -07:00
splinter98
8293f546af Add support for MQTT server as a notification target (#4474)
This implementation is similar to AMQP notifications:

* Notifications are published on a single topic as a JSON feed
* Topic is configurable, as is the QoS. Uses the paho.mqtt.golang
  library for the mqtt connection, and supports connections over tcp
  and websockets, with optional secure tls support.
* Additionally the minio server configuration has been bumped up
  so mqtt configuration can be added.
* Configuration migration code is added with tests.

MQTT is an ISO standard M2M/IoT messaging protocol and was
originally designed for applications for limited bandwidth
networks. Today it's use is growing in the IoT space.
2017-06-14 17:27:49 -07:00
Anis Elleuch
af8071c86a xl: Fix rare freeze after many disk/network errors (#4438)
xl.storageDisks is sometimes passed to some low-level XL functions. Some disks in
xl.storageDisks are set to nil when they encounter some errors. This means all
elements in xl.storageDisks will be nil after some time which lead to an unusable XL.
2017-06-14 17:14:27 -07:00
Daniel Lind
dce76d9307 Fix xl.diskWithAllParts to proper checksum algorithm (#4509) 2017-06-14 17:13:02 -07:00
Harshavardhana
11c4223f2c Add docker release files. (#4473)
We used to release by building directly on the docker
hub auto build process, which is sufficient for edge
but it is not a good idea to do it for stable releases.

Do not build docker release binaries again, but instead
use the released binaries themselves which are signed
and validated.
2017-06-14 02:08:35 -07:00
Nitish Tiwari
26903da8c2 Add steps to remove Minio volumes in the swarm (#4536) 2017-06-13 16:10:11 -07:00
Harshavardhana
353f2d3a6e fs: Hold format.json readLock ref to avoid GC. (#4532)
Looks like if we follow pattern such as

```
_ = rlk
```

Go can potentially kick in GC and close the fd when
the reference is lost, only speculation is that
the cause here is `SetFinalizer` which is set on
`os.close()` internally in `os` stdlib.

This is unexpected and unsual endeavour for Go, but
we have to make sure the reference is never lost
and always dies with the server.

Fixes #4530
2017-06-13 08:29:07 -07:00
Nitish Tiwari
c8947af227 Update Kubernetes-yaml deployment example and Helm deployment doc with Minio image update steps (#4515) 2017-06-13 01:37:14 -07:00
Harshavardhana
075b8903d7 fs: Add safe locking semantics for format.json (#4523)
This patch also reverts previous changes which were
merged for migration to the newer disk format. We will
be bringing these changes in subsequent releases. But
we wish to add protection in this release such that
future release migrations are protected.

Revert "fs: Migration should handle bucketConfigs as regular objects. (#4482)"
This reverts commit 976870a391.

Revert "fs: Migrate object metadata to objects directory. (#4195)"
This reverts commit 76f4f20609.
2017-06-12 17:40:28 -07:00
Harshavardhana
b8463a738c Add support for DCOS host detection, improve Docker detection. (#4525)
isDocker was currently reading from `/proc/cgroup` file. But
this file alone is rather not conclusive evidence. Docker
internally has `.dockerenv` as a special file which we should
use instead.

Fixes #4456
2017-06-13 00:33:21 +00:00
Frank Wessels
6f4862659f Investigate issue #4461 (#4521)
* Code to investigate issue #4461 (rare test failure in TestListenAndServeTLS)

* Use UTCNow() instead of time.Now().UTC()
2017-06-13 00:20:29 +00:00
Nitish Tiwari
f59d7a04b4 Removed references to docker-machine in Swarm guide (#4502) 2017-06-10 21:44:20 -07:00
Dee Koder
b28d5fa633 Clarify macOS instructions on brew paths. Deleted homebrew upgrade instructions. (#4501) 2017-06-09 14:24:32 -07:00
Harshavardhana
48dbd49980 Add support for kubernetes host detection (#4514)
Additionally improve what we print for `docker pull`
such that its precisely the relevant release tag.

Fixes #4456
2017-06-09 02:42:12 -07:00
Bala FA
3dfe254a11 gateway: make each backend as subcommands. (#4506)
Fixes #4450
2017-06-08 23:28:45 -07:00
Krishna Srinivas
ec2920e981 Allow "minio server ." to start minio in fs mode (#4513) 2017-06-08 18:58:51 -07:00
Rushan
0f5483f497 browser: Disable usage/free stats for browser-gateway (#4497) 2017-06-08 15:09:50 -07:00
Krishnan Parthasarathi
8a6b0cc0cd TestInitListeners: Use port 0 pick available port (#4508) 2017-06-08 12:08:21 -07:00
Krishna Srinivas
2c56788f8d Validate gateway arguments (#4376)
Fixes #4355
2017-06-08 11:20:56 -07:00
Frank Wessels
145328ac9f tests: Run select statement in separate goroutine (#4499)
Instead of after the wg.Wait() so as to make sure that the 'earliest'
of the two select case that becomes active is selected..
2017-06-08 07:39:50 -07:00
poornas
45a568dd85 Give more specific error message on browser for nested policies (#4488) 2017-06-07 19:31:23 -07:00
Frank Wessels
7dcc1e92b4 Prevent unnecessary (superfluous) initialization of return variable (#4490) 2017-06-08 00:24:46 +00:00
poornas
999ae1cb96 Fix browser download returning zero bytes for s3 (#4483) 2017-06-06 18:19:35 -07:00
poornas
6651c2fc5f disable settings change on browser in gateway mode (#4472) 2017-06-06 14:56:41 -07:00
Harshavardhana
976870a391 fs: Migration should handle bucketConfigs as regular objects. (#4482)
Current code failed to anticipate the existence of files
which could have been created to corrupt the namespace such
as `policy.json` file created at the bucket top level.

In the current release creating such as file conflicts
with the namespace for future bucket policy operations.
We implemented migration of backend format to avoid situations
such as these.

This PR handles this situation, makes sure that the
erroneous files should have been moved properly.

Fixes #4478
2017-06-06 12:15:35 -07:00
Harshavardhana
f99987e47c Generate sha1sum as well for release for backward compatibility. (#4475)
Additionally remove support for arm6vl in release, since
go 1.8 the support for armv6 has been dropped and we do
not see high usage events from this platform.
2017-06-06 11:25:06 -07:00
Harshavardhana
1c3f244fc5 creds: Secretkey should be generated upto 40 characters in length. (#4471)
Current code allowed it wrongly to generate secret key upto 100
we should only use 100 as a value to validate but for generating
it should be 40.

Fixes #4470
2017-06-05 15:18:03 -07:00
Aditya Manthramurthy
986aa8fabf Bypass network in lock requests to local server (#4465)
This makes lock RPCs similar to other RPCs where requests to the local
server bypass the network. Requests to the local lock-subsystem may
bypass the network layer and directly access the locking
data-structures.

This incidentally fixes #4451.
2017-06-05 12:25:04 -07:00
poornas
2559614bfd fix: Set UIversion in reply for policy API (#4469) 2017-06-05 08:11:54 -07:00
Harshavardhana
a4d1ef1b62 browser: update ui-assets with new changes. (#4467)
Fixes #4269
2017-06-02 15:11:47 -07:00
Harshavardhana
432bf7d99e Fail if formatting is wrong in our CI tests. (#4459)
We didn't fail before, we should helps in avoiding
formatting issues to creep into the codebase.
2017-06-02 14:05:51 -07:00
poornas
18c4e5d357 Enable browser support for gateway (#4425) 2017-06-01 09:43:20 -07:00
Aditya Manthramurthy
64f4dbc272 Disable redirect of HTTP request to a HTTPS Minio server (#4454)
Fixes #4452
2017-05-31 20:33:13 -07:00
Frank Wessels
9ba57a8df0 Add errCorruptedFormat to list of ignored errors for metadata operations. (#4447)
Fixes listing of objects where xl.json is empty or corrupted to skip to the next disk/server (issue 4354).
2017-05-31 20:03:32 -07:00
Dee Koder
5621e6a494 Refactor service stop signal message. (#4428) 2017-05-31 11:53:04 -07:00
Frank Wessels
0f0758aece Load IO error count for posix atomically (#4448)
* Load error count atomically in order to check for maximum allowed number of IO errors.

* Remove unused (previously atomic) network IO error count
2017-05-31 09:22:53 -07:00
Aditya Manthramurthy
a0e02f43e1 Fix and cleanup update message and improve related tests (#4361)
Fixes #4232
2017-05-31 09:22:00 -07:00
Harshavardhana
28352f3f5d log: Startup banner should strip standard ports. (#4443)
APIEndpoints list should strip off standard ports
to avoid confusion with clients.
2017-05-31 09:21:28 -07:00
Harshavardhana
975972d57e server: Redirection should use globalMinioPort with host without port. (#4445)
Currently redirection doesn't work in following scenarios

 - server started with port ":80" and TLS is configured
   client requested insecure request on port "80"
   gets redirected to port 443 and fails.
2017-05-31 09:21:02 -07:00
Harshavardhana
458f22f37c log: Fix printing of signature error request headers. (#4444)
The following commit f44f2e341c
fix was incomplete and we still had presigned URLs printing
in query strings in wrong fashion.

This PR fixes this properly. Avoid double encoding
percent encoded strings such as

`s3%!!(MISSING)A(MISSING)`

Print properly as json encoded.

`s3%3AObjectCreated%3A%2A`
2017-05-31 00:11:06 -07:00
Krishna Srinivas
0bba3cc8e3 gateway-azure: Convert S3 metadata to azure metadata (#4384)
fixes #4292
2017-05-30 20:05:41 -07:00
Cesar Alvernaz
bac5303b10 Some minor fixes (#4441)
Word terminations
2017-05-30 11:03:25 -07:00
Harshavardhana
e01b2fc06d Disable network share test, appveyor bug. (#4446) 2017-05-30 11:02:31 -07:00
Harshavardhana
072fcf3ba6 fs: Make sure to validate bucket first in PutObject() (#4427)
Currently even when bucket doesn't exist we wrongly
return success, when an object is a directory prefix with
 '/' as suffix and is of size 0.

This PR fixes this behavior.
2017-05-25 09:22:43 -07:00
Harshavardhana
b78f6fbcc5 Do not send envVars in ServerInfo() (#4422)
Sending envVars along with access and secret
exposes the entire minio server's sensitive
information. This will be an unexpected
situation for all users.

If at all we need to look for things like if
credentials are set through env, we should
only have access to only this information
not the entire set of system envs.
2017-05-24 21:09:23 -07:00
samkevich
99ca8a2928 fix InvalidAccessKeyId error according to amazon documentation (#4404)
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
2017-05-23 20:07:52 -07:00
poornas
9b3dd44607 Add dotnet library to minio startup message (#4410) 2017-05-23 13:57:27 -07:00
Krishnan Parthasarathi
3c5db69ffd Treat 0.0.0.0 as local address in --address flag (#4386) 2017-05-23 12:07:39 -07:00
Krishna Srinivas
28c26a9e59 Generate random ETag if client does not provide MD5 for PutObjectPart (#4385)
fixes #4289
fixes #4290
2017-05-22 15:48:48 -07:00
morph027
9136734c02 added secrets to distributed swarm minio (#4374) 2017-05-22 11:30:30 -07:00
Harshavardhana
348aa6566c Add nsswitch.conf to our docker image (#4379)
This will cause the alpine images to resolve DNS
using /etc/hosts along with the normal DNS resolver.
2017-05-22 07:05:39 -07:00
luomeiqin
9d98bf1c0f Bucket names can contain hyphen (#4324) 2017-05-19 07:30:00 -07:00
Rushan
a767ad321a Browser: Fix Safari Blob download issue (#4357) 2017-05-17 17:03:44 -07:00
Harshavardhana
1886d94e95 server/mux: Use constants provided by Go http (#4360) 2017-05-17 11:57:52 -07:00
Anis Elleuch
542f7ae42c gateway: Reject endpoint pointing to local gateway (#4310)
Show an error when the user enters an endpoint url pointing
to the gateway server itself.
2017-05-16 21:13:29 -07:00
Harshavardhana
59b3e0b79b auth/rpc: Add RWMutex instead of Mutex for granular locking. (#4352)
Refer https://github.com/minio/minio/issues/4345
2017-05-16 14:34:56 -07:00
Aditya Manthramurthy
8975da4e84 Add new ReadFileWithVerify storage-layer API (#4349)
This is an enhancement to the XL/distributed-XL mode. FS mode is
unaffected.

The ReadFileWithVerify storage-layer call is similar to ReadFile with
the additional functionality of performing bit-rot checking. It
accepts additional parameters for a hashing algorithm to use and the
expected hex-encoded hash string.

This patch provides significant performance improvement because:

1. combines the step of reading the file (during
erasure-decoding/reconstruction) with bit-rot verification;

2. limits the number of file-reads; and

3. avoids transferring the file over the network for bit-rot
verification.

ReadFile API is implemented as ReadFileWithVerify with empty hashing
arguments.

Credits to AB and Harsha for the algorithmic improvement.

Fixes #4236.
2017-05-16 14:21:52 -07:00
Frank
cae4683971 Make clearing of stale debug lock info independent of deleting map entry of lock itself. (#4353)
This is believed to address issue #4337 where stale information for debug locks in shown.
2017-05-16 07:19:17 -07:00
Krishna Srinivas
5db1e9f3dd signature: use region from Auth header if server's region not configured (#4329) 2017-05-15 18:17:02 -07:00
Anis Elleuch
465274cd21 server-info: Change Error type to string (#4346)
Golang std error type doesn't marshal/unmarshal with json. So errors
are not actually being sent when a client calls ServerInfo() API.
2017-05-15 07:28:47 -07:00
Harshavardhana
87fb911d38 Rename structs for azure and s3 gateway to be consistent. (#4347) 2017-05-15 00:52:33 -07:00
Harshavardhana
155a90403a fs/erasure: Rename meta 'md5Sum' as 'etag'. (#4319)
This PR also does backend format change to 1.0.1
from 1.0.0.  Backward compatible changes are still
kept to read the 'md5Sum' key. But all new objects
will be stored with the same details under 'etag'.

Fixes #4312
2017-05-14 12:05:51 -07:00
Harshavardhana
c63afabc9b build/release: Generate sha256sums also without the release tag. (#4318)
Ref #4306
2017-05-12 21:40:22 -07:00
poornas
0404e747cf Fix broken link (#4344) 2017-05-12 18:29:50 -07:00
Aditya Manthramurthy
3bc9e6101c Add minimum requirements sections to notifications docs (#4328) 2017-05-11 17:34:27 -07:00
Anis Elleuch
f2ed149714 Add slack channel link to corrupted disk err msg (#4270) 2017-05-11 14:27:32 -07:00
Harshavardhana
5a16dcf4cf Add a graceful msg when CTRL+C is pressed. (#4248) 2017-05-11 14:27:18 -07:00
Harshavardhana
fa3d5d0f46 build/release: Generate sha256sums for built binaries. (#4311)
We used to build sha1sum deprecate it and
use sha256sum instead.

Fixes #4306
2017-05-10 11:22:05 -07:00
Krishna Srinivas
bb292e4e38 web-handler: Allow anonymous download of zip (#4309)
fixes #4230
2017-05-10 09:54:24 -07:00
poornas
d1971b9a4d Prevent duplicate policy rows from being created (#4276) 2017-05-10 09:52:31 -07:00
Harshavardhana
fa3f6d75b6 fs: Verify if parent is an object before i/o. (#4304)
PutObject() needs to verify and fail.

Fixes #4301
2017-05-09 17:46:46 -07:00
Harshavardhana
298b470f69 fs/erasure: Ignore objects with / even for DeleteObject() (#4303)
Additionally GetObject() also returns errFileNotFound similar
to HeadObject().

Fixes #4302
2017-05-09 14:32:24 -07:00
Krishna Srinivas
fc774957fe gateway: reject requests with unknown authorization (#4297) 2017-05-09 07:53:31 -07:00
Nitish Tiwari
c6258f5e97 Multi tenancy doc (#4215)
* Add multi-tenancy doc

* Multi-tenancy documents

* Remove intro

* Update deploy-multiple-minio.md

* Update deploy-multiple-minio.md

* Update deploy-multiple-minio.md

* Update deploy-multiple-minio.md

* Update multi-tenant details

* Remove file

* Rename deploy-multiple-minio.md to README.md

* update ports

* Add multi-tenancy diagrams

* Link diagrams and update disk name in the commands

* Fix tenant config directory
2017-05-08 19:22:34 -07:00
Anis Elleuch
85bc6003e9 gateway-s3: Avoid x2 double quotes in ListParts (#4295)
ListParts response returns doubled double quotes in ETag field.
This commit cleans ETag when receiving it from minio client to
fix the issue.
2017-05-08 14:42:05 -07:00
Nitish Tiwari
0d9de50e21 Bump Docker compose file to latest release (#4271)
* Bump Docker compose file to latest release
* Bump Docker Swarm compose file to latest release
2017-05-07 18:58:24 -07:00
Rushan
d13aa1c42d browser: make input number types readonly in share objects modal (#4273) 2017-05-07 18:02:19 -07:00
Harshavardhana
610dbe3479 config: Do not migrate config file if not needed. (#4264)
Also improve the error message returned by `pkg/quick`.

Fixes #4233
2017-05-06 10:16:59 -07:00
Anis Elleuch
2df1e2e9a9 doc: Fix pgsql cmd example (#4265) 2017-05-05 15:20:29 -07:00
Harshavardhana
e372b5ed67 build: Fix release build names. (#4263)
Currently due to the occurrence of 6 arguments from
`gen-ldflags.go` leads to a bug where the binaries
genenerated have wrong names.

As shown below.

```
If you want to build for all, Just press Enter: linux/amd64
-->    linux/amd64:github.com/minio/minio
$ ls release/linux-amd64/
[2017-05-04 23:08:51 PDT]  17MiB minio
[2017-05-04 23:08:51 PDT]  17MiB minio.2017-05-05T06:08:22Z
[2017-05-04 23:08:51 PDT]    76B minio.shasum
```

This PR fixes this issue by retaining the previous release
binary names.

```
If you want to build for all, Just press Enter: linux/amd64
-->    linux/amd64:github.com/minio/minio
$ ls release/linux-amd64/
[2017-05-04 23:08:51 PDT]  17MiB minio
[2017-05-04 23:08:51 PDT]  17MiB minio.RELEASE.2017-05-05T06-08-22Z
[2017-05-04 23:08:51 PDT]    76B minio.shasum
```
2017-05-05 13:16:58 -07:00
Harshavardhana
76f4f20609 fs: Migrate object metadata to objects directory. (#4195)
Fixes #3352
2017-05-05 08:49:09 -07:00
Harshavardhana
99ddd35343 docs: use IEC format such as iB everywhere. (#4247) 2017-05-05 08:28:08 -07:00
Remco Verhoef
01e9adc4b3 Implement anonymous uploads, fixes #4250 (#4259) 2017-05-04 20:03:56 -07:00
Nitish Tiwari
4c63fd06c6 Add Kubernetes yaml file deployment example (#4262)
* Add Kubernetes yaml file deployment example

* Change the Docker image tag to latest

* Update latest release tag
2017-05-04 19:30:46 -07:00
Nitish Tiwari
bb4efbf258 Add minimum requirements for MySQL notification (#4260) 2017-05-04 17:30:56 -07:00
Harshavardhana
a89c7299d1 browser: Update ui-assets with new fixes. (#4246)
Brings two fixes.

 - browser: Listing should append instead of replacing previous listing (#4188)
 - browser: Make login form browser auto-fill compatible (#4091) fixes #4235
 - browser: Selecting a new bucket appends objects list to previous bucket's list (#4252)
2017-05-04 14:57:41 -07:00
Harshavardhana
df027a8f51 Webhook endpoints can fail, we must start the server. (#4255)
This PR fixes a regression introduced in #4060
2017-05-04 13:43:54 -07:00
Aditya Manthramurthy
a02575ebf9 Bump up minio-go to (fixes #4243) (#4256) 2017-05-04 13:43:23 -07:00
Krishna Srinivas
972a527b66 browser: Selecting a new bucket appends objects list to previous bucket's list (#4252) 2017-05-04 11:12:46 -07:00
Krishnan Parthasarathi
02910725c5 Make gateway help for s3/azure similar (#4249) 2017-05-04 10:38:48 -07:00
Harshavardhana
0ea8bfaf78 Add waiting on hosts in docker entrypoint for distributed setups. (#4244)
Thanks to Remco Verhoef <remco@dutchcoders.io> for the script.

Fixes #4225
2017-05-04 00:48:13 -07:00
Remco Verhoef
069cf9e8aa Use s3.amazonaws.com as default endpoint, fixes #4240 (#4242) 2017-05-03 22:41:03 -07:00
Aditya Manthramurthy
2121b78ea7 Fix bug in JSON representation of object properties (#4238)
Introduced in #4003
2017-05-03 20:10:00 -07:00
Remco Verhoef
5016649f47 Add s3 backend to help, fixes #4219 (#4221)
* Add s3 backend to help, fixes #4219

* Add samples for Gateway usage with S3
2017-05-03 17:55:30 -07:00
Bala FA
2b78444056 fix: ignore TLS handshake error. (#4227)
Fixes #4200
2017-05-03 03:23:15 -07:00
Karthic Rao
9b58a669e5 tests: Fix rare test crash (#4175)
Fix rare test crash by improving the randomness logic.
2017-05-02 23:54:22 -07:00
Krishna Srinivas
e5b2e25caf gateway-s3: vendor-update minio-go (#4220) 2017-05-02 18:46:39 -07:00
Krishna Srinivas
4aa65910e5 gateway: Restore bucket policy functionality for Azure (#4209) 2017-05-02 12:27:25 -07:00
Harshavardhana
8b272a3163 config: Improve config migrate messaging. (#4216)
Previous message

```
Migration from version ‘17’ to ‘18’ completed successfully.
```

For example didn't provide any meaningful insights.

This PR attempts to improve this message as below

```
Configuration file '/home/harsha/.minio/config.json' migrated from version '17' to '18' successfully.
```

Fixes #4199
2017-05-02 11:43:27 -07:00
Harshavardhana
f0b5c0ec7c windows: Support all REPARSE_POINT attrib files properly. (#4203)
This change adopts the upstream fix in this regard at
https://go-review.googlesource.com/#/c/41834/ for Minio's
purposes.

Go's current os.Stat() lacks support for lot of strange
windows files such as

 - share symlinks on SMB2
 - symlinks on docker nanoserver
 - de-duplicated files on NTFS de-duplicated volume.

This PR attempts to incorporate the change mentioned here

   https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/

The article suggests to use Windows I/O manager to
dereference the symbolic link.

Fixes #4122
2017-05-02 02:35:27 -07:00
Remco Verhoef
44d53c9c67 cleanup and fix comments (#4212) 2017-05-01 14:44:31 -07:00
Krishna Srinivas
6cf6828a4c gateway: Rename gateway files to have "gateway-" prefix (#4207) 2017-05-01 10:32:18 -07:00
Krishna Srinivas
01f04c717e gateway: reject bad path segments in URL (#4202) 2017-04-28 17:17:18 -07:00
Krishna Srinivas
0d32b22359 gateway: Fix help message for gateway (#4201) 2017-04-28 16:42:16 -07:00
Harshavardhana
cab298d68f pkg: Update the rpm spec with latest release. (#4187) 2017-04-28 12:35:02 -07:00
Krishna Srinivas
1ea53b4d9f browser: Listing should append instead of replacing previous listing (#4188)
Fixes #4144
2017-04-28 09:30:26 -07:00
Anis Elleuch
d36dd80a8a cors: Set Access-Control-Allow-Credentials to true (#4185)
This allow browsers to send credentials with preflighted requests.
2017-04-27 12:40:22 -07:00
Remco Verhoef
3a539ce660 Implement gateway S3 support (#3940) 2017-04-27 11:26:00 -07:00
Harshavardhana
57c5c75611 web: Simplify and converge common functions in web/obj API. (#4179)
RemoveObject() in webAPI currently re-implements some part
of the code to remove objects combine them for simplicity
and code convergence.
2017-04-26 23:27:48 -07:00
Bala FA
cf1fc45142 Improve duration humanization. (#4071) 2017-04-26 03:38:35 -07:00
Dee Koder
64c1c0f37d docs: Update with home brew special note for macOS upgrades. (#4180) 2017-04-25 20:18:48 -07:00
Dee Koder
82857cd6df docs: Document homebrew install path changes for minio. (#4178)
* docs: Document homebrew install path changes for minio.

* updates: Updated with the feedback provided.

* docs: Fix breaking change message as per feedback.

* docs: fix typos in readme.

* typo: grammar refactor of update instructions.
2017-04-25 19:43:22 -07:00
Krishnan Parthasarathi
3cdc0c57c8 Provide command to help fill issue template (#4174) 2017-04-25 00:58:11 -07:00
Harshavardhana
dc365bca44 build: -s -w should be added by gen-ldflags.go (#4172) 2017-04-24 23:01:38 -07:00
Harshavardhana
3b1626216d docs: Point docker compose to new release. (#4171) 2017-04-24 21:37:13 -07:00
Harshavardhana
48aa2ac392 server: Validate path for bad components in a handler. (#4170) 2017-04-24 18:13:46 -07:00
Frank
0d1e2ab509 Remove hardcoded min and max limit for erasure coding (#4157) 2017-04-24 10:00:33 -07:00
Nitish Tiwari
ebf4c447bb docs: Add Minikube deployment to k8s docs (#4133) 2017-04-24 09:20:51 -07:00
Peter Tribble
2b96d9f706 Enable build on solaris (#4115) 2017-04-23 11:10:18 -07:00
Anis Elleuch
83abad0b37 admin: ServerInfo() returns info for each node (#4150)
ServerInfo() will gather information from all nodes before returning
it back to the client.
2017-04-21 07:15:53 -07:00
Harshavardhana
df346753e1 api: Fix registering of s3 endpoint peers properly (#4159)
We need to have local peer initialized properly
for listen bucket to work, current code did initialize
properly but the resulting code was initializing
peer on a wrong target v/s what listen bucket expected
it to be.

This regression came in de204a0a52

Fixes #4158
2017-04-20 15:28:29 -07:00
Harshavardhana
f1d7780167 lock: Vendorize all the new changes made in minio/dsync (#4154)
Fixes #4139
2017-04-19 14:22:35 -07:00
Harshavardhana
5a3c5aec31 server/mux: Fix serverMux to set deadlines based on UTC time. (#4146)
Avoid using `time.Now()` instead rely on UTC time
for the final deadline, this is to be consistent with
all our internal functions.

Reduce the default read timeout to 15 seconds
in lieu with a newly discovered issue
   - https://github.com/minio/minio/issues/4139

Additionally also change the Read() conn wrapper
to set deadline only upon successful Reads().
2017-04-19 13:16:06 -07:00
Aditya Manthramurthy
a4305742e8 Add key for Kafka messages (fixes #4143) (#4151) 2017-04-19 11:26:35 -07:00
Harshavardhana
640ebb2f79 lock: Fix missing formatting directives while printing. (#4147)
Current log prints in this form

```
ERRO[8150] Lock maintenance failed to remove entry for write
lock (should never happen)%!!(MISSING)(EXTRA ....
```

Fix this by using proper formatting directive.
2017-04-19 10:37:56 -07:00
Harshavardhana
402a5e3bea docs: Fix and reword FreeBSD documentation. (#4145) 2017-04-19 00:25:20 -07:00
Harshavardhana
f4dac979a2 server: Fix message when corrupted or unsupported format is found. (#4142)
Refer https://github.com/minio/minio/issues/4140

This is a fix to provide a little more elaborate message.
2017-04-18 10:35:17 -07:00
Krishnan Parthasarathi
3032f0f505 Remove duration field from lock instrumentation (#4111)
Duration for which a lock was held can be computed from the `Since`
field of `OpsLockState`. It is the difference between current time and
time at which the namespace lock was held. This change avoids
superfluous instrumentation.
2017-04-15 11:40:01 -07:00
Harshavardhana
7765081db7 cache: Increasing caching GC percent from 20 to 50. (#4041)
Previous value was set to avoid large cache value build
up but we can clearly see this can cause lots of GC
pauses which can lead to significant drop in performance.

Change this value to 50% and decrease the value to 25%
once the 75% cache size is used. To have a larger
window for GC pauses.

Another change is to only allow caching if a server has
more than 24GB of RAM instead of 8GB.
2017-04-15 02:16:49 -07:00
Anabel V
18bfe5cba6 docs: Created new illustration for docs. (#4012) 2017-04-14 17:21:57 -07:00
Anis Elleuch
14f0047295 fs: Remove fs meta lock when PutObject() fails (#4114)
Removing the fs meta lock file when PutObject() encounters any error
during its execution, such as upload getting permatuerly cancelled
by the client.
2017-04-14 12:06:24 -07:00
Krishna Srinivas
e6b2253da9 gateway: Fix help message for custom Azure Blob Storage endpoint. (#4113) 2017-04-14 11:02:43 -07:00
Krishnan Parthasarathi
ca64b86112 Return possible states a heal operation (#4045) 2017-04-14 10:28:35 -07:00
Karthic Rao
5f065e2a96 server: Fix CI build complaints (#4119)
- Ineffassign fixes.
- Spell check correction.
2017-04-14 08:00:04 -07:00
Harshavardhana
a7afa469e2 xl: Add stat calls to keep track of ignored errors. (#4117)
Such that in a situation where all errors were
ignored we need to reduce the errors using
readQuorum to get a consistent error value.

Without this change errors generated will
never be consistent with for an expected scenario.

For example in a 6 disk setup 1 disk is missing
and 5 do not have the volume (testbucket)

Without this change Stat() would result in different
errors depending on which disk died. Can cause
confusion to S3 client application.

This change addresses need to track type of
errors we ignored and bring readQuorum to
choose the maximally occuring as the value
of truth.
2017-04-14 01:46:16 -07:00
Bala FA
d103d5fb7c server: Error out if loopback addr is used for Distributed Erasure (#4105) 2017-04-12 20:27:24 -07:00
Harshavardhana
6683247080 tests: Fix the sopradic test failure in TestListObjectPartsDiskNotFound (#4107)
getBucketInfo() should keep track errors ignored,
such that in a situation where all errors were
ignored we need to reduce the errors using readQuorum
to get a consistent error value.

This is the problem we see with DiskNotFound test
disks are randomly removed.

Fixes #4095
2017-04-12 15:38:35 -07:00
Anis Elleuch
e4bd882f11 handlers: Ignore malformatted datetime type header (#4097)
Ignore headers, such as If-Modified-Since, If-Unmodified-Since, etc.. when they
are received with a format other than HTTP date.
2017-04-12 12:34:57 -07:00
Nitish Tiwari
4448285a83 Cleanup service docs (#4103)
- Remove windows service doc and add to Minio-service
- Remove Linux service doc as it is duplicated
2017-04-12 12:18:34 -07:00
Harshavardhana
952c618441 server: Fix a regression in printing startup banner. (#4100)
Octect based sorting was lost in the previous commit

de204a0a52

This PR fixes a regression - fixes #4099
2017-04-12 09:22:35 -07:00
Krishna Srinivas
c5249c35d3 gateway: Support for custom endpoint. (#4086) 2017-04-11 17:44:26 -07:00
Bala FA
de204a0a52 Add extensive endpoints validation (#4019) 2017-04-11 15:44:27 -07:00
Harshavardhana
1b1b9e4801 lock/rpc: change rpcPath to be called serviceEndpoint. (#4088)
This is a cleanup to ensure proper naming.
2017-04-11 10:25:21 -07:00
Rushan
1b0b2c1c76 Browser: Make login form browser auto-fill compatible (#4091) 2017-04-11 10:11:42 -07:00
koolhead17
7f5e037846 docs: Update docker quick-start guide and adds relevant project URLS (#4075) 2017-04-11 00:53:33 -07:00
Karthic Rao
929a13f33f Fix for writes from Apache Spark. (#4074)
- Due to usage of amazon SDK, spark expects md5sum of empty string to be
  returned when it does PUT on a directory.
- The fix returns md5sum of a empty string for the above mentioned case.
- This fixes the issue of Apache Spark not being able to write into Minio.
2017-04-10 19:51:23 -07:00
Krishna Srinivas
a4209c10ea signature-v4: Use sha256("") for calculating canonical request (#4064) 2017-04-10 09:58:08 -07:00
Harshavardhana
b927523223 server: Introduce a new env MINIO_REGION. (#4078)
This is implemented to be able to override region
through command line just like how access and
secret keys are provided.
2017-04-09 10:44:10 -07:00
Aditya Manthramurthy
604417baf4 Allow cluster to start when only n/2 servers are up (#4066)
Fixes #3234.

Relaxes the quorum requirement to start the object layer, and skips
quick-healing at start-up (as no write quorum is present).
2017-04-09 00:28:27 -07:00
Harshavardhana
6e9ac8db59 docker: Support docker swarm secrets. (#3977)
Fixes #3896
2017-04-08 01:43:40 -07:00
Harshavardhana
0497d5c342 api: SourceInfo should be populated in GET/HEAD notification. (#4073)
Refer https://github.com/minio/mc/issues/2073
2017-04-08 01:39:20 -07:00
Harshavardhana
6b4f368dfe notify: Webhook endpoints can fail, but we must start the server. (#4060)
Ignore any network errors when registering a webhook
notifier during Minio startup sequence. This way server
can be started even if the webhook endpoint is not available
and unreachable.
2017-04-08 01:13:55 -07:00
Nitish Tiwari
6507c30bbc Add steps to run Minio distributed on Windows (#4068) 2017-04-08 01:12:00 -07:00
Harshavardhana
f44f2e341c log: Dump signature request properly. (#4063)
Currently percent encoded strings are not properly encoded.

`s3%!!(MISSING)A(MISSING)`

Print properly as json encoded.

`s3%3AObjectCreated%3A%2A`
2017-04-07 14:37:32 -07:00
Krishnan Parthasarathi
60ea2b17ba Fix xml block syntax in admin-api Readme (#4062) 2017-04-07 03:38:01 -07:00
Harshavardhana
5ed1a8ad23 docs: macOS brew now refers to Minio fork (#4059) 2017-04-07 01:42:59 -07:00
Harshavardhana
27749c2124 admin/info: Add HTTPStats value as part of serverInfo() struct. (#4049)
Remove our counter implementation instead use atomic external
package which supports more types and methods.
2017-04-06 23:08:33 -07:00
Krishna Srinivas
1d99a560e3 refactor: extractSignedHeaders() handles headers removed by Go http server (#4054)
* refactor: extractSignedHeaders() handles headers removed by Go http server.
* Cleanup extractSignedHeaders() TestExtractSignedHeaders()
2017-04-05 17:00:24 -07:00
Krishna Srinivas
af82d27018 signature-v4: Support for transfer-encoding request header (#4053) 2017-04-05 15:08:33 -07:00
Anis Elleuch
f205689ff5 build: Fix compilation in 32 bits platforms (#4052)
go fails to build Minio under at least, armv6 and 386 due to some
inconsistencies in the type of one syscall variable in different
architectures. This PR casts that variable to uint64 to achieve
the desired consistency.
2017-04-05 11:17:59 -07:00
Harshavardhana
393c01d078 browser: Generate new UI assets. 2017-04-04 23:03:36 -07:00
Rajeesh C V
e9037b9d36 fix: add white space in storage usage section (#4038)
Fix for #3962 - add whitespace before storage usage details section in
 Browse component
2017-04-04 09:14:45 -07:00
Harshavardhana
4747adfcb4 fs: Enable returning ETag along with ListObjects() (#4042)
This is to comply with S3 behavior, we previously removed
reading `fs.json` for optimization reasons but we have a
reason to believe that providing ETag and using gjson
provides needed benefit of not having to deal with
unmarshalling overhead of golang stdlib.

Fixes #4028
2017-04-04 09:14:03 -07:00
Anis Elleuch
52d8f564bf sigv2: Unespace canonicalized resources values (#4034)
Values of canonicalized query resources should be unescaped before calculating
the signature. This bug is not noticed before because partNumber and uploadID
values in Minio doesn't have characters that need to be escaped.
2017-04-03 17:55:14 -07:00
Harshavardhana
3fe33e7b15 handler: simplify parsing valid location constraint. (#4040)
Separate out validating v/s parsing logic in
isValidLocationConstraint() into parseLocationConstraint()
and isValidLocation()

Additionally also set `X-Amz-Bucket-Region` as part of the
common headers for the clients to fallback on in-case of any
region related errors.
2017-04-03 14:50:09 -07:00
Krishnan Parthasarathi
4041e5f20d Provide mc-admin-heal command on start-up (#4031)
Healing of buckets, objects and incomplete uploads are implemented and
available via admin REST APIs. Additionally, it is available via mc admin 
sub-command. The warning is no longer relevant.

Fixes #4030
2017-04-03 14:24:25 -07:00
Krishnan Parthasarathi
96c46c15e7 madmin: Rename HealObjectResult to HealResult (#4035)
madmin.HealObjectResult is used in HealObject and HealUpload. It only
makes sense to rename it to HealResult.
2017-04-03 08:25:32 -07:00
Harshavardhana
3bf67668b6 sys/stats: return cgroup mem limit, fall back to sysinfo() (#4002)
This is necessary where in certain environments where
cgroup is used to limit memory usage of a container or
a particular process.

GetStats() is used by caching module to figure out the
optimal cacheable size in memory with cgroup limits
what sysinfo reports might not be the right value set
for a given process.

Fixes #4001
2017-04-02 10:46:16 -07:00
Anis Elleuch
e31e2c3bc2 doc: Explain how to create certificate chain file (#4032)
public.crt needs sometimes to have a chain certificate, this PR
explains how to construct public.crt when certificate are issued
by a certificate authority.
2017-04-02 04:47:56 -07:00
Harshavardhana
214279aa57 build: Reduce binary size by using -s -w (#4027)
Refer #3939
2017-04-01 01:06:16 -07:00
Harshavardhana
4de6b15fca vet: Fix all the go vet complaints (#4029)
```
go tool vet -atomic -bool -copylocks -nilfunc \
   -printf -shadow -rangeloops -unreachable \
   -unsafeptr -unusedresult cmd/
```
2017-04-01 01:06:06 -07:00
Krishnan Parthasarathi
2bd694dbc8 Add disksUnavailable healStatus const (#3990)
`disksUnavailable` healStatus constant indicates that a given object
needs healing but one or more of disks requiring heal are offline. This
can be used by admin heal API consumers to distinguish between a
successful heal and a no-op since the outdated disks were offline.
2017-03-31 17:55:15 -07:00
Aditya Manthramurthy
a2a8d54bb6 Add access format support for Elasticsearch notification target (#4006)
This change adds `access` format support for notifications to a
Elasticsearch server, and it refactors `namespace` format support.

In the case of `access` format, for each event in Minio, a JSON
document is inserted into Elasticsearch with its timestamp set to the
event's timestamp, and with the ID generated automatically by
elasticsearch. No events are modified or deleted in this mode.

In the case of `namespace` format, for each event in Minio, a JSON
document is keyed together by the bucket and object name is updated in
Elasticsearch. In the case of an object being created or over-written
in Minio, a new document or an existing document is inserted into the
Elasticsearch index. If an object is deleted in Minio, the
corresponding document is deleted from the Elasticsearch index.

Additionally, this change upgrades Elasticsearch support to the 5.x
series. This is a breaking change, and users of previous elasticsearch
versions should upgrade.

Also updates documentation on Elasticsearch notification target usage
and has a link to an elasticsearch upgrade guide.

This is the last patch that finally resolves #3928.
2017-03-31 14:11:27 -07:00
Harshavardhana
2040d32ef8 server/tls: Do not rely on a specific cipher suite (#4021)
Do not rely on a specific cipher suite instead let the
go choose the type of cipher needed, if the connection
is coming from clients which do not support forward
secrecy let the go tls handle this automatically based
on tls1.2 specifications.

Fixes #4017
2017-03-31 13:28:45 -07:00
Harshavardhana
f1015a5096 notifiers: Stop using url.Parse in validating address format. (#4011)
url.Parse() wrongly parses an address of format "address:port"
which is fixed in go1.8.  This inculcates a breaking change
on our end. We should fix this wrong usage everywhere so that
migrating to go1.8 eventually becomes smoother.
2017-03-31 04:47:40 -07:00
Aditya Manthramurthy
096427f973 Add deliveryMode parameter for AMQP notfication target (#4008)
Configuration migration was done.

Also adds documentation about AMQP configuration parameters.

Fixes #3982
2017-03-31 03:34:26 -07:00
Rushan
5cec6bd80d Browser: Use object name with prefix to delete sub-path objects (#4013) 2017-03-30 23:28:28 -07:00
Bala FA
6e9c91f43a fix: use its own lock in serverConfigV17 (#4014)
Previously serverConfigV17 used a global lock that made any instance of
serverConfigV17 depended on single global serverConfigMu.

This patch fixes by having individual lock per instances.
2017-03-30 22:26:24 -07:00
Bala FA
2df8160f6a server: handle command line and env variables at one place. (#3975) 2017-03-30 11:21:19 -07:00
Romain Bouyé
447fdd4097 docs: Fix typo in docs/config/README.md (#4009) 2017-03-30 11:09:15 -07:00
Harshavardhana
28c5a887de event: Set contentType as well under NotificationEvent. (#4003)
This is an enhancement change to to cater support all
the data fields present on the object. Currently
we only send a subset of data which object info
provides us.

It also helps us keep a full namespace mirror on
notification targets for efficient query.
2017-03-30 08:58:14 -07:00
Anis Elleuch
fbe8b3259d webhook: Add support of custom CAs (#4000) 2017-03-29 13:42:55 -07:00
Anis Elleuch
e2aba9196f obj-handlers: Rewrite src & dst path cmp in Copy() (#3998)
CopyObjectHandler() was incorrectly performing comparison
between destination and source object paths, which sometimes
leads to a lock race. This PR simplifies comparaison and add
one test case.
2017-03-29 09:21:38 -07:00
Aditya Manthramurthy
61b08137b0 Add access format support for Redis notification target (#3989)
This change adds `access` format support for notifications to a Redis
server, and it refactors `namespace` format support.

In the case of `access` format, a list is used to store Minio
operations in Redis. Each entry in the list is a JSON encoded list of
two items - the first is the Minio server timestamp of the event, and
the second is an object describing the operation that created/replaced
the object in the server.

In the case of `namespace` format, a hash is used. Entries in the hash
may be updated or removed if objects in Minio are updated or deleted
respectively. The field values in the Redis hash are JSON encoded.

Also updates documentation on Redis notification target usage.

Towards resolving #3928
2017-03-29 08:55:53 -07:00
Harshavardhana
1caad902cb config/path: Figure out absolute paths properly on windows. (#3996)
The following form of arguments such as

```
minio.exe -C some_dir server dir
```

has stopped working because of lack of handling of
absolute paths for config directory. Always calculate
absolute path for any relative paths on any operating
system.

The following fix converts all config directory relative
paths into absolute paths.

Fixes #3991
2017-03-29 08:55:33 -07:00
Nitish Tiwari
d99efa2c93 Update filename (#3995) 2017-03-28 23:14:28 -07:00
Krishna Srinivas
9ee83b89bb config: Appropriate error message when newer config file is found (#3972) 2017-03-28 18:41:16 -07:00
Nitish Tiwari
a8cb43926a Docker guide fix (#3992)
* Update Docker quick start guide

- Add Compose to orchestration section.

- Refer Swarm from Docker quick start guide.

- Add common Docker commands to quick start guide.

* Paragraph cleanup
2017-03-28 13:54:19 -07:00
Harshavardhana
b62cd8ed84 sign/streaming: Content-Encoding is not set in newer aws-java-sdks (#3986)
We can't use Content-Encoding to verify if `aws-chunked` is set
or not. Just use 'streaming' signature header instead.

While this is considered mandatory, on the contrary aws-sdk-java
doesn't set this value

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

```
Set the value to aws-chunked.
```

We will relax it and behave appropriately. Also this PR supports
saving custom encoding after trimming off the `aws-chunked`
parameter.

Fixes #3983
2017-03-27 17:02:04 -07:00
Anis Elleuch
1b3a517683 lock, Windows: O_CREAT infers only GENERIC_WRITE (#3981)
Avoid using GENERIC_ALL flag when creating a file since it could
return permission denied in some Windows versions.
2017-03-27 12:47:39 -07:00
Nitish Tiwari
a9c0f1e0a4 Added kernel tuning docs (#3921) 2017-03-27 11:29:04 -07:00
Aditya Manthramurthy
a099319e66 Support access format for database notification targets (#3953)
* Add configuration parameter "format" for db targets and perform
  configuration migration.
* Add PostgreSQL `access` format: This causes Minio to append all events
  to the configured table. Prefix, suffix and event filters continue
  to be supported for this mode too.
* Update documentation for PostgreSQL notification target.
* Add MySQL `access` format: It is very similar to the same format for
  PostgreSQL.
* Update MySQL notification documentation.
2017-03-27 11:27:25 -07:00
Bala FA
6e63904048 browser-flag: wrapped bool type denotes browser on/off flag. (#3963)
Statically typed BrowserFlag prevents any arbitrary string value
usage. The wrapped bool marshals/unmarshals JSON according to the
typed value ie string value "on" represents boolean true and "off" as
boolean false.
2017-03-26 12:00:27 -07:00
Harshavardhana
565ac4c861 tests: use url.QueryEscape() when dealing with url query params. (#3974)
This is to keep the portability and also avoid errors that
might occur using the functions written for URL resource name
Since query param values have different escaping requirements.
2017-03-26 11:56:17 -07:00
Harshavardhana
a4ecd8bca2 docs: Add config directory documentation/guide. (#3889) 2017-03-25 02:34:04 -07:00
Harshavardhana
28eff0f6c1 build: Improve build messaging, say where we built Minio. (#3973) 2017-03-25 00:33:57 -07:00
Krishnan Parthasarathi
c27ece409b heal: Check if all parts are available and valid (#3967)
In the algorithm to check if an object requires healing, in addition to
checking if all disks have xl.json present we should check if all parts
of the object are present and have valid blake2b checksums.

Also fixed a minor compilation error in heal-objects-list.go.
2017-03-24 08:40:44 -07:00
Dee Koder
8c0ce2fee9 docs: Removed space from code blocks. (#3965) 2017-03-23 19:06:31 -07:00
Bala FA
d3cb79a57c Refactor logger (#3924)
This patch fixes below

* Previously fatalIf() never writes log other than first logging target.
* quiet flag is not honored to show progress messages other than startup messages.
* Removes console package usage for progress messages.
2017-03-23 16:36:00 -07:00
Anis Elleuch
11e15f9b4c config: Remove level in console/file loggers (#3938)
Also rename fileName field in file logger to filename
2017-03-23 08:27:22 -07:00
Krishnan Parthasarathi
4e92b2ecb8 Fix listDirHealFactory merging of entries across disks (#3959)
For listing of objects needing heal, we list all objects present on all
the disks and return the set union. We were incorrectly dropping objects
that weren't already seen in disks so far.

Sample directory layout of disks in a 4-disk setup:
`/tmp/1`, `/tmp/2`, `/tmp/3`, `/tmp/4` are directories used as disks here.
`test` is the bucket, `obj1` and obj2` are the objects.
```
/tmp/1/test
└── obj2
    ├── part.1
    ├── part.2
    └── xl.json
/tmp/2/test
└── obj1
    ├── part.1
    ├── part.2
    └── xl.json
/tmp/3/test
├── obj1
│   ├── part.1
│   ├── part.2
│   └── xl.json
└── obj2
    ├── part.1
    ├── part.2
    └── xl.json
/tmp/4/test
[This is empty]

```
2017-03-23 08:24:59 -07:00
koolhead17
80b83a51a3 Docs: Fix for Self signed certificate. (#3957) 2017-03-23 08:20:39 -07:00
Krishna Srinivas
777d12d928 browser: update ui-assets.go (#3956) 2017-03-22 23:17:14 -07:00
Krishnan Parthasarathi
607c8a9611 Add sourceInfo to NotificationEvent (#3937)
This change adds information like host, port and user-agent of the
client whose request triggered an event notification.

E.g, if someone uploads an object to a bucket using mc. If notifications
were configured on that bucket, the host, port and user-agent of mc
would be sent as part of event notification data.

Sample output:
```
"source": {
          "host": "127.0.0.1",
          "port": "55808",
          "userAgent": "Minio (linux; amd64) minio-go/2.0.4 mc ..."
}
```
2017-03-22 18:44:35 -07:00
Anis Elleuch
7f5a5b5e9d config: Do not validate creds when set via env (#3955)
It is useless to validate access/secret keys stored in
config file when the user sets them in the environment.
2017-03-22 16:11:58 -07:00
Bala FA
d4ca2ee1a3 pkg/quick: add Save() function and other enhancements. (#3951)
* Add a new function Save() which saves given configuration into given file.
* Simplify Load() function.
* Remove unused CheckVersion().
* CheckData() is a private function now.
* quick_test.go is part of quick package now.
* minio server uses top level quick.Load() and quick.Save() functions.
2017-03-22 10:23:25 -07:00
Krishnan Parthasarathi
417ec0df56 HealObject should succeed when only N/2 disks have data (#3952) 2017-03-22 10:15:16 -07:00
Rushan
fbfb4fc5a0 Browser: Use polyfill to support Object.assign in IE11 (#3942) 2017-03-21 13:48:07 -07:00
koolhead17
97dc34fe06 docs: Fix for README.md markdown table. (#3948) 2017-03-21 12:02:39 -07:00
Krishnan Parthasarathi
9f9ba1e984 XL: Return the right error (#3944) 2017-03-21 10:33:25 -07:00
Krishnan Parthasarathi
13c4ce3617 Add notification for object access via GET/HEAD (#3941)
The following notification event types are available for all targets,
      s3:ObjectAccessed:Get
      s3:ObjectAccessed:Head
      s3:ObjectAccessed:*
2017-03-21 10:32:17 -07:00
Krishnan Parthasarathi
181e002c56 pkg/madmin: Set UploadID in ListUploadsHeal (#3945)
Without this fix, `mc admin heal -I` wouldn't be able to heal ongoing
uploads. `mc` depends on `ListUploadsHeal` API to identify ongoing
uploads to heal given a bucket and an object.
2017-03-21 10:32:02 -07:00
Anis Elleuch
9d6e226692 heal: Set truncate when no more walk entries (#3932) 2017-03-20 15:31:25 -07:00
Krishnan Parthasarathi
eb02261642 XL: Don't return ignored errors in listDirFactory (#3935)
Previously, erasure backend's `listDirFactory` may return errors which
were explicitly ignored. With this change, it returns nil. Superfluous
checks at higher-layers for ignored errors are removed as well.
2017-03-20 11:09:05 -07:00
Pawan Rawal
1396e91dd1 Update link for downloading minio server. (#3934)
This and the link for downloading Minio server at other places in the docs seems to be broken. I suppose this happened while updating the name of the page (which updated the url) in Doctor docs. 
Might be nice for Doctor to update internal links if the name of a page is changed in a background job.
2017-03-19 18:26:23 -07:00
Bala FA
7ebf11b202 words: new package Damerau Levenshtein distance function. (#3929) 2017-03-19 14:23:05 -07:00
Bala FA
1c97dcb10a Add UTCNow() function. (#3931)
This patch adds UTCNow() function which returns current UTC time.

This is equivalent of UTCNow() == time.Now().UTC()
2017-03-18 11:28:41 -07:00
Anis Elleuch
3a6111eff5 admin: Export HealStatus data type (#3930)
`healStatus` can be returned to the API caller. This commit will help
developers to declare a variable with HealStatus type.
2017-03-18 11:27:27 -07:00
Aditya Manthramurthy
2463ae243a Add support for MySQL notifications (fixes #3818) (#3907)
As a new configuration parameter is added, configuration version is
bumped up from 14 to 15.

The MySQL target's behaviour is identical to the PostgreSQL: rows are
deleted from the MySQL table on delete-object events, and are
created/updated on create/over-write events.
2017-03-17 09:29:17 -07:00
Krishnan Parthasarathi
c192e5c9b2 Implement heal-upload admin API (#3914)
This API is meant for administrative tools like mc-admin to heal an
ongoing multipart upload on a Minio server.  N B This set of admin
APIs apply only for Minio servers.

`github.com/minio/minio/pkg/madmin` provides a go SDK for this (and
other admin) operations.  Specifically,

  func HealUpload(bucket, object, uploadID string, dryRun bool) error

Sample admin API request:
POST
/?heal&bucket=mybucket&object=myobject&upload-id=myuploadID&dry-run
- Header(s): ["x-minio-operation"] = "upload"

Notes:
- bucket, object and upload-id are mandatory query parameters
- if dry-run is set, API returns success if all parameters passed are
  valid.
2017-03-17 09:25:49 -07:00
Nitish Tiwari
d4eea224d4 Remove white spaces (#3922) 2017-03-17 09:23:22 -07:00
Nitish Tiwari
e55421ebdd Fixed Docker compose link (#3920) 2017-03-17 01:06:11 -07:00
Dee Koder
0a5d57a91e docs: Update gateway doc with roadmap section. (#3918) 2017-03-16 18:25:01 -07:00
Krishna Srinivas
cea4cfa3a8 Implement S3 Gateway to third party cloud storage providers. (#3756)
Currently supported backend is Azure Blob Storage.

```
export MINIO_ACCESS_KEY=azureaccountname
export MINIO_SECRET_KEY=azureaccountkey
minio gateway azure
```
2017-03-16 12:21:58 -07:00
Anis Elleuch
8426cf9aec config: Accept more address format + unit test (#3915)
checkURL() is a generic function to check if a passed address
is valid. This commit adds support for addresses like `m1`
and `172.16.3.1` which is needed in MySQL and NATS. This commit
also adds tests.
2017-03-16 11:44:01 -07:00
Harshavardhana
f3334159a4 config: Relax browser and region to be empty. (#3912)
- If browser field is missing or empty
  then default to "on".

- If region field is empty or missing then
  default to "us-east-1" (S3 spec behavior)
2017-03-16 11:06:17 -07:00
Harshavardhana
3edff1501e Vendor upstream redis library instead of our fork. (#3913)
We forked the upstream to address a build issue on
go 1.6 - that is long done and we don't need
to manage our forks anymore.
2017-03-16 08:22:47 -07:00
Bala FA
21d73a3eef Simplify credential usage. (#3893) 2017-03-16 00:16:06 -07:00
Krishnan Parthasarathi
051f9bb5c6 Implement list uploads heal admin API (#3885) 2017-03-16 00:15:06 -07:00
Harshavardhana
6509589adb Use canonicalETag helper wherever needed. (#3910) 2017-03-15 20:48:49 -07:00
Anis Elleuch
ae4361cc45 config: Check for duplicated entries in all scopes (#3872)
Validate Minio config by checking if there is double json key
in any scope level. The returned error contains the json path
to the duplicated key.
2017-03-15 16:30:34 -07:00
Krishna Srinivas
cad0d0eb7a browser: Update ui-assets.go (#3902)
fixes #3898
2017-03-15 11:32:51 -07:00
Nitish Tiwari
ba0c11757e Added that no special config changes reqd (#3906)
Added a line saying no special config changes are required for Shared mode. Also the previous `Why Shared Backend` and current `Use cases` are already merged. This fixes the comment: https://github.com/minio/minio/pull/3888#discussion_r105573494
2017-03-15 08:22:03 -07:00
Krishna Srinivas
96050c1e21 browser: Do not show "Loading..." if there are no buckets. (#3904) 2017-03-14 19:07:23 -07:00
Rushan
a27d1b3d86 Browser: Use babel-polyfill to support new ES6 built-ins in older browsers (#3900) 2017-03-14 15:02:20 -07:00
Anis Elleuch
a5e60706a2 xl,fs: Return 404 if object ends with a separator (#3897)
HEAD Object for FS and XL was returning invalid object name when
an object name has a trailing slash separator, this PR changes the
behavior and will always return 404 object not found, this guarantees
a better compatibility with S3 spec.
2017-03-13 22:20:46 -07:00
Harshavardhana
5f7565762e api: postPolicy cleanup. Simplify the code and re-use. (#3890)
This change is cleanup of the postPolicyHandler code
primarily to address the flow and also converting
certain critical parts into self contained functions.
2017-03-13 14:41:13 -07:00
Nitish Tiwari
3314501f19 Simplify shared mode document (#3888) 2017-03-12 16:17:03 -07:00
Harshavardhana
9d53a646a1 Simplify the title for orchestration and some words. (#3887) 2017-03-11 02:17:03 -08:00
Nitish Tiwari
5e0032e165 Update links (#3886) 2017-03-11 00:11:07 -08:00
Harshavardhana
305952d734 browser: Update ui-assets with new changes. 2017-03-10 15:13:26 -08:00
Rushan
e77885d671 Browser: Add Object.assign polyfill to support older browsers (#3884) 2017-03-10 14:30:23 -08:00
Nitish Tiwari
2410eb281e Update kafkacat command with consumer flag (#3882) 2017-03-10 09:01:59 -08:00
Harshavardhana
e54025805f browser: update ui-assets with new changes. 2017-03-09 15:26:25 -08:00
Anis Elleuch
d602495600 madmin: Do not require SSL to set credentials (#3879)
We need to relax this requirement and let the client decides
if it can allow to set credentials API over plain connection.
2017-03-09 14:08:33 -08:00
Nitish Tiwari
03937e7554 Added documentation for orchestration platforms (#3684) 2017-03-09 14:06:51 -08:00
Harshavardhana
e3b627a192 docker: Add ARM64 image build support (#3876) 2017-03-09 13:57:27 -08:00
Rushan
ccc3349f0c Browser: Show complete bucket name by removing the ellipsis cut off (#3877) 2017-03-09 10:42:38 -08:00
Harshavardhana
85cbd875fc cleanup: All conditionals simplified under pkg. (#3875)
Address all the changes reported/recommended by
`gosimple` tool.
2017-03-09 10:13:30 -08:00
Harshavardhana
43317530d5 Fix odd shadowing bug in XL init. (#3874)
Fixes #3873
2017-03-08 20:42:45 -08:00
Bala FA
8a9852220d Make unit testable cert parsing functions. (#3863) 2017-03-08 19:20:01 -08:00
Harshavardhana
47ac410ab0 Code cleanup - simplify server side code. (#3870)
Fix all the issues reported by `gosimple` tool.
2017-03-08 10:00:47 -08:00
Harshavardhana
433225ab0d docker: ca-certificates should not be removed. (#3868) 2017-03-07 16:30:19 -08:00
Harshavardhana
3e655a2c85 docker: Add ARM docker container dockerfile. (#3574) 2017-03-07 15:15:05 -08:00
Anis Elleuch
a2eae54d11 xl: Respect min. space by checking PrepareFile err (#3867)
It was possible to upload a big file which overcomes the minimal
disk space limit in XL, PrepareFile was actually checking for disk
space but we weren't checking its returned error. This patch fixes
this behavior.
2017-03-07 14:48:56 -08:00
Anis Elleuch
79e0b9e69a Relax minio server start when disk threshold is reached and adds space check in FS (#3865)
* fs: Rename tempObjPath variable in fsCreateFile()
* fs/posix: Factor checkDiskFree() function
* fs: Add disk free check in fsCreateFile()
* posix: Move free disk check to createFile()
* xl: Relax free disk check in POSIX initialization
* fs: checkDiskFree checks for space to store data
2017-03-07 12:25:40 -08:00
Krishna Srinivas
29ff9674a0 browser: Humanize expiry time for Share-Object. (#3861) 2017-03-06 20:13:52 -08:00
Bala FA
bff4d29415 Remove commands and commandsTree global variables. (#3855) 2017-03-06 19:35:26 -08:00
Krishna Srinivas
436db49bf3 Share object expiry value modification modal was not working (#3860) 2017-03-06 18:50:25 -08:00
Rushan
966818955e Browser: Implement multiple object delete (#3859) 2017-03-06 15:43:43 -08:00
Harshavardhana
e49efcb9d9 xl: quickHeal heal bucket only when needed. (#3854)
This improves the startup time significantly
for clusters which have lot of buckets.

Also fixes a bug where `.minio.sys` is created
on disks which do not have `format.json`
2017-03-06 02:00:15 -08:00
Harshavardhana
6f931d29c4 rpm: Add RPM spec for minio build. (#3853)
Currently the package is built and hosted at

https://copr.fedorainfracloud.org/coprs/minio/minio/

To enable minio repo one has to download.

Fedora - 25
https://copr.fedorainfracloud.org/coprs/minio/minio/repo/fedora-25/minio-minio-fedora-25.repo

Fedora - 26
https://copr.fedorainfracloud.org/coprs/minio/minio/repo/fedora-26/minio-minio-fedora-26.repo

Enables for both i386 and x86_64.

Fixes #3576
2017-03-05 13:09:31 -08:00
Krishnan Parthasarathi
e3fd4c0dd6 XL: Make listOnlineDisks and outDatedDisks consistent w/ each other. (#3808) 2017-03-04 14:53:28 -08:00
Harshavardhana
b05c1c11d4 browser: Update UI assets with new changes. 2017-03-03 18:05:02 -08:00
Krishna Srinivas
0bae3330e8 browser: Send correct arguments for RemoveObjects web handler. (#3848)
Fixes #3839
2017-03-03 17:07:17 -08:00
Harshavardhana
05e53f1b34 api: CopyObjectPart was copying wrong offsets due to shadowing. (#3838)
startOffset was re-assigned to '0' so it would end up
copying wrong content ignoring the requested startOffset.

This also fixes the corruption issue we observed while
using docker registry.

Fixes https://github.com/docker/distribution/issues/2205

Also fixes #3842 - incorrect routing.
2017-03-03 16:32:04 -08:00
Anis Elleuch
0c8c463a63 tests: Fix web handlers testing with faulty disks (#3845) 2017-03-03 15:46:28 -08:00
Aditya Manthramurthy
6df7bc42b8 Fix check for bucket name: (#3832)
* Do not allow bucket names with adjacent hypen and periods.
* Improve performance by eliminating the usage of regular expressions.
2017-03-03 10:23:41 -08:00
Anis Elleuch
6c00a57a7c quick: Add yaml format support (#3833)
quick Save() and Load() infers config file's format from
file name extension.
2017-03-03 10:22:09 -08:00
Harshavardhana
bc52d911ef api: Increase the maximum object size limit from 5GiB to 16GiB. (#3834)
The globalMaxObjectSize limit is instilled in S3 spec perhaps
due to certain limitations on S3 infrastructure. For minio we
don't have such limitations and we can stream a larger file
instead.

So we are going to bump this limit to 16GiB.

Fixes #3825
2017-03-03 10:14:17 -08:00
Anis Elleuch
28c53a3555 obj: Make checkBucketExist() returns all errors (#3843)
This function was returning BucketNotFound for all errors
which at least hides the fact that disks could be corrupted.
This commit fixes the behavior by returning all errors that,
are, by the way, Object API errors.
2017-03-03 10:12:43 -08:00
Harshavardhana
e5d4e7aa9d web: Validate if bucket names are reserved (#3841)
Both '.minio.sys' and 'minio' should be never allowed
to be created from web-ui and then fail to list it
by filtering them out.

Fixes #3840
2017-03-03 03:01:42 -08:00
Anis Elleuch
cddc684559 admin: Set Config returns errSet and errMsg (#3822)
There is no way to see if a node encountered an error
when trying to set a new config set, this commit adds
a bool errSet field.
2017-03-03 02:53:48 -08:00
Zejun Li
32d0d3d4ac Enhanced newObjectLayerFn (#3837) 2017-03-03 01:07:45 -08:00
Bala FA
98d17d2a97 Remove globalQuiet and globalConfigDir global variables (#3830) 2017-03-02 14:21:30 -08:00
Bala FA
208dd15245 Remove globalMaxCacheSize and globalCacheExpiry variables (#3826)
This patch fixes below

* Remove global variables globalMaxCacheSize and globalCacheExpiry.
* Make global variables into constant in objcache package.
2017-03-02 10:34:37 -08:00
Anis Elleuch
a179fc9658 quick: Simplify Load() and CheckVersion() (#3831) 2017-03-02 10:29:06 -08:00
Zejun Li
d1afd16955 Using RWMutex to guard closing and listeners (#3829) 2017-03-02 10:00:22 -08:00
Bala FA
2348ae7a19 Make default values as constants (#3828) 2017-03-02 04:58:39 -08:00
Bala FA
480ea826dc Move rlimit functions into sys package. (#3824)
This patch addresses below

* go build works for bsd family
* probe total RAM size for bsd family
* make unit testable functions
2017-03-01 21:51:57 -08:00
Aditya Manthramurthy
09e9fd745c Close client connection after checking for release update (#3820) 2017-03-01 09:18:55 -08:00
Anis Elleuch
77c1998a38 config: Fix creating new config with wrong version (#3821)
Simplify a little config code to avoid making mistake
next time.
2017-03-01 09:17:04 -08:00
Krishna Srinivas
91cf54f895 web-handlers: Support removal of multiple objects at once. (#3810) 2017-02-28 19:07:28 -08:00
Karthic Rao
2b0ed21f08 tests: Fix test server init - cleanup (#3806) 2017-02-28 18:05:52 -08:00
Harshavardhana
472fa4a6ca api: Multi object delete should be protected. (#3814)
Add missing protection from deleting multiple objects
in parallel. Currently we are deleting objects without
proper locking through this API.

This can cause significant amount of races.
2017-02-28 18:00:24 -08:00
Bala FA
097cec676a fix: Set globalMaxCacheSize to allowable value. (#3816)
If memory resource limit and total RAM are more than 8GiB, either 50%
of memory resource limit or total RAM is set to globalMaxCacheSize.
2017-02-28 16:51:52 -08:00
Rushan
fcad4a44fd Browser: Remove duplicate object entries while sorting (#3813) 2017-02-28 13:11:34 -08:00
Anis Elleuch
9b3c014bab config: Add browser parameter (#3807)
browser new parameter receives "on" or "off" parameter which is similar
to MINIO_BROWSER
2017-02-27 14:59:53 -08:00
Krishnan Parthasarathi
c9619673fb Implement SetConfig admin API handler. (#3792) 2017-02-27 11:40:27 -08:00
Anis Elleuch
dce0345f8f Set disk to nil after write which needs quorum (#3795)
Ignore a disk which wasn't able to successfully perform an action to
avoid eventual perturbations when the disk comes back in the middle
of write change.
2017-02-26 11:58:32 -08:00
Anis Elleuch
461b2bbd37 admin: Move SetCredentials from Service to Generic (#3805)
Setting credentials doesn't belong to service management API
anymore.
2017-02-25 11:06:08 -08:00
Bala FA
69777b654e event: use common initialization logic (#3798)
Previously creating and adding targets for each notification type was
repeated.  This patch fixes it.
2017-02-24 18:27:52 -08:00
Harshavardhana
70d2cb5f4d rpc: Remove time check for each RPC calls. (#3804)
This removal comes to avoid some redundant requirements
which are adding more problems on a production setup.

Here are the list of checks for time as they happen

 - Fresh connect (during server startup) - CORRECT
 - A reconnect after network disconnect - CORRECT
 - For each RPC call - INCORRECT.

Verifying time for each RPC aggravates a situation
where a RPC call is rejected in a sequence of events
due to enough load on a production setup. 3 second
might not be enough time window for the call to be
initiated and received by the server.
2017-02-24 18:26:56 -08:00
Harshavardhana
cff45db1b9 cli: Use ADDRESS:PORT to clarify --address behavior (#3803)
Currently we document as IP:PORT which doesn't provide
if someone can use HOSTNAME:PORT. This is a change
to clarify this by calling it as ADDRESS:PORT which
encompasses both a HOSTNAME and an IP.

Fixes #3799
2017-02-24 14:19:20 -08:00
Harshavardhana
bcc5b6e1ef xl: Rename getOrderedDisks as shuffleDisks appropriately. (#3796)
This PR is for readability cleanup

- getOrderedDisks as shuffleDisks
- getOrderedPartsMetadata as shufflePartsMetadata

Distribution is now a second argument instead being the
primary input argument for brevity.

Also change the usage of type casted int64(0), instead
rely on direct type reference as `var variable int64` everywhere.
2017-02-24 09:20:40 -08:00
Harshavardhana
25b5a0534f browser: Update ui-assets and fix the copyright header. (#3790) 2017-02-22 17:27:26 -08:00
Rushan
52d6678bf0 Browser: Implement multi select user interface for object listings (#3730) 2017-02-22 14:51:38 -08:00
Nitish Tiwari
d8950ba7c5 Added server times note and fix Notes rendering for Doctor. (#3787) 2017-02-22 02:12:03 -08:00
Harshavardhana
cc28765025 xl/multipart: Make sure to delete temp renamed object. (#3785)
Existing objects before overwrites are renamed to
temp location in completeMultipart. We make sure
that we delete it even if subsequenty calls fail.

Additionally move verifying of parent dir is a
file earlier to fail the entire operation.

Ref #3784
2017-02-21 19:43:44 -08:00
Harshavardhana
fe86319c56 ci: For windows builds stick to go1.7.5 (#3786) 2017-02-21 17:24:11 -08:00
Harshavardhana
99a12613a3 update: For source builds look for absolute path. (#3780)
os.Args[0] doesn't point to absolute path we need
use exec.LookPath to find the absolute path before
sending os.Stat().
2017-02-21 01:32:05 -08:00
Nitish Tiwari
097dd7418a Remove unused erasure diagram (#3783) 2017-02-20 21:01:08 -08:00
Nitish Tiwari
a7d3ea8c15 Update erasure code image (#3782) 2017-02-20 20:12:21 -08:00
Krishnan Parthasarathi
2745bf2f1f Implement ServerConfig admin REST API (#3741)
Returns a valid config.json of the setup. In case of distributed
setup, it checks if quorum or more number of nodes have the same
config.json.
2017-02-20 12:58:50 -08:00
Anis Elleuch
70d825c608 doc: Small rewrite of bucket events notif intro (#3775) 2017-02-20 12:07:27 -08:00
Harshavardhana
6b68c0170f For streaming signature do not save content-encoding in PutObject() (#3776)
Content-Encoding is set to "aws-chunked" which is an S3 specific
API value which is no meaning for an object. This is how S3
behaves as well for a streaming signature uploaded object.
2017-02-20 12:07:03 -08:00
Aditya Manthramurthy
0a905e1a8a Fix rabbitmq reconnect problem (#3778) 2017-02-20 12:05:21 -08:00
Harshavardhana
9eb8e375c5 cli: Make sure to add --help flag for subcommands. (#3773)
--help is now back and prints properly with command
help template.
2017-02-19 20:46:06 -08:00
Harshavardhana
7ea1de8245 copyObject: Be case sensitive for windows only server. (#3766)
For case sensitive platforms we should honor case.

Fixes #3765

```
1) python s3cmd -c s3cfg_localminio put logo.png s3://testbucket/xyz/etc2/logo.PNG

2) python s3cmd -c s3cfg_localminio ls s3://testbucket/xyz/etc2/
2017-02-18 10:58     22059   s3://testbucket/xyz/etc2/logo.PNG

3) python s3cmd -c s3cfg_localminio cp s3://testbucket/xyz/etc2/logo.PNG s3://testbucket/xyz/etc2/logo.png
remote copy: 's3://testbucket/xyz/etc2/logo.PNG' -> 's3://testbucket/xyz/etc2/logo.png'

4) python s3cmd -c s3cfg_localminio ls s3://testbucket/xyz/etc2/
2017-02-18 10:58     22059   s3://testbucket/xyz/etc2/logo.PNG
2017-02-18 11:10     22059   s3://testbucket/xyz/etc2/logo.png
```
2017-02-18 13:41:59 -08:00
Anis Elleuch
54a18592e9 flags: Fix --version output (#3772) 2017-02-18 13:41:33 -08:00
Anis Elleuch
7e84c7427d server-mux: Rewrite graceful shutdown mechanism (#3771)
Old code uses waitgroup Add() and Wait() in different threads,
which eventually can lead to a race.
2017-02-18 13:28:54 -08:00
Bala FA
d12f3e06b1 config-old: Use interface to avoid code repetition. (#3769) 2017-02-18 10:45:37 -08:00
Harshavardhana
0137ff498a auth/rpc: Token can be concurrently edited protect it. (#3764)
Make sure we protect when we access `authToken` in authClient.

Fixes #3761
2017-02-18 03:15:42 -08:00
Harshavardhana
34d9a6b46a Make sure client initializes to proper lock RPC path. (#3763)
Fixes a regression introduced in previous commit.
2017-02-18 02:52:11 -08:00
Harshavardhana
50b4e54a75 fs: Do not return reservedBucket names in ListBuckets() (#3754)
Make sure to skip reserved bucket names in `ListBuckets()`
current code didn't skip this properly and also generalize
this behavior for both XL and FS.
2017-02-16 14:52:14 -08:00
Harshavardhana
8816b08aae Fix the systemd config path to the new URL 2017-02-15 21:28:06 -08:00
Harshavardhana
271e3ecde5 Fix tests from cli changes 2017-02-15 18:05:55 -08:00
Harshavardhana
611bd68739 cli: Add new features from CLI.
use `.HelpName` for template to pass down
proper command names. Also treat flags
to be optional in all of them.
2017-02-15 17:45:11 -08:00
Harshavardhana
25eeb88a8f cli: Bring upstream changes to minio server. 2017-02-15 17:30:31 -08:00
Krishna Srinivas
3e770defae browser: Update ui assets with new changes (#3751) 2017-02-15 14:44:34 -08:00
Krishna Srinivas
ae4656e699 browser: Remove currentPath prefix in object names from list result. (#3750) 2017-02-15 13:07:47 -08:00
Harshavardhana
f6a9e690bf browser: Update ui assets with new changes. 2017-02-15 11:29:23 -08:00
Anis Elleuch
7f86a21317 admin: Add ServerInfo API() (#3743) 2017-02-15 10:45:45 -08:00
Harshavardhana
fb39c7c26b sRPC/client: Properly trim storageRPCPath for actual disk path. (#3749)
Never print internal RPC endpoint paths.
2017-02-15 03:47:47 -08:00
Harshavardhana
13c3b9cbcb main/cli: Don't print hidden flags/commands in help template. (#3748)
Always use .VisibleFlags and .VisibleCommands to not print
Hidden flags as expected from help template.
2017-02-15 02:25:38 -08:00
Bala FA
602dac8773 mainUpdate: refactor to handle quiet flag properly (#3744) 2017-02-15 00:31:00 -08:00
Dee Koder
c6e76160ad Update README.md 2017-02-14 14:20:29 -08:00
Bala FA
a53b909fcd mainVersion: Remove minio init and quiet handling (#3739) 2017-02-13 00:24:34 -08:00
Harshavardhana
22909c849e objcache: Return io.ReaderAt to avoid Seeking and Reading. (#3735) 2017-02-11 17:17:58 -08:00
Bala FA
440866d26c Move go version check into main() (#3734)
Previously the check was done in init().  This patch moves into main
for unit testable friendly function.
2017-02-11 01:27:27 -08:00
Krishna Srinivas
25b936c369 browser: Implement infinite scrolling for object listing. (#3720)
fixes #2831
2017-02-10 22:54:42 -08:00
Nitish Tiwari
8f66cfa316 Fix https://github.com/minio/minio/issues/3453 (#3733) 2017-02-10 22:53:17 -08:00
Bala FA
8912b6bf3b trie: new package (#3729)
This implements a simple trie tree for minio server/tools.
This package borrows idea from
https://godoc.org/golang.org/x/text/internal/triegen.
2017-02-10 11:51:41 -08:00
Karthic Rao
a799351926 Fix macOS builds (#3728) 2017-02-10 00:37:22 -08:00
Harshavardhana
9df01035da Remove XL references in public docs to Erasure. (#3725)
Ref #3722
2017-02-09 23:26:44 -08:00
Krishnan Parthasarathi
1ad96ee09f Handle uptime for single-node instances differnt from distributed setup (#3726) 2017-02-09 20:38:14 -08:00
Harshavardhana
1b4bb94ac4 config: setter/getter for Notifier and Logger into its own struct. (#3721)
This is an attempt cleanup code and keep the top level config
functions simpler and easy to understand where as move the
notifier related code and logger setter/getter methods as part
of their own struct.

Locks are now held properly not globally by configMutex, but
instead as private variables.

Final fix for #3700
2017-02-09 15:20:54 -08:00
Andreas Auernhammer
f38222c0cc update the blake2b implementation (#3724)
Fixes a performance bug caused by SSE-AVX register savings on amd64.
2017-02-09 15:01:00 -08:00
Anis Elleuch
c9b1468c3b Presigend Post: Error out when File is not found (#3723)
Follow S3 behavior when no File is sent in the presigned
post request form.
2017-02-09 12:37:32 -08:00
Krishna Srinivas
6800902b43 web-handlers: Implement API to download files as a zip file. (#3715) 2017-02-08 23:39:08 -08:00
Krishnan Parthasarathi
e5773e11c6 Make minio server compile on OpenBSD, NetBSD, Solaris (#3719) 2017-02-08 22:27:35 -08:00
Nitish Tiwari
0c7694894b Added spaces for formatting (#3717) 2017-02-08 00:51:08 -08:00
Krishnan Parthasarathi
ce9aa2f2b2 Add uptime to ServiceStatus (#3690) 2017-02-08 00:13:02 -08:00
Nitish Tiwari
7547f3c8a3 Added Webhook notification details (#3706) 2017-02-07 23:19:32 -08:00
Harshavardhana
31dff87903 Honor envs properly for access and secret key. (#3703)
Also changes the behavior of `secretKeyHash` which is
not necessary to be sent over the network, each node
has its own secretKeyHash to validate.

Fixes #3696
Partial(fix) #3700 (More changes needed with some code cleanup)
2017-02-07 12:51:43 -08:00
Anis Elleuch
fd72c21e0e tests: Reduce two functions work to gain test time (#3712)
TestListObjectsHeal and TestFSShutdown takes around 3 min,
this PR reduces the number of created test objects
2017-02-07 12:51:23 -08:00
Harshavardhana
310bf5bd36 auth/rpc: Make auth rpc client retry configurable. (#3695)
Currently the auth rpc client defaults to to a maximum
cap of 30seconds timeout. Make this to be configurable
by the caller of authRPCClient during initialization, if no
such config is provided then default to 30 seconds.
2017-02-07 02:16:29 -08:00
Harshavardhana
a170e44689 Let hijacker and flusher interfaces to be reflected. (#3709)
Ideally here if the interface is not found it would
fail the server, as it should be because without these
we can't even have a working server in the first place.

Just like how it fails in master invariably inside Go
net/http code path.

Fixes #3708
2017-02-06 23:52:47 -08:00
Anis Elleuch
70e70446bb signv4: Read always returns EOF when stream ends (#3692)
When EOF is reached, further calls of Read() doesn't return io.EOF
but continue to work as it expects to have more data, this PR fixes
the behavior
2017-02-06 14:19:27 -08:00
Krishna Srinivas
45d9cfa0c5 signature-v4: stringToSign and signingKey should use Scope's date. (#3688)
fixes #3676
2017-02-06 13:09:09 -08:00
Anis Elleuch
93fd269329 stats: Add network and http statisics (#3686)
Network: total bytes of incoming and outgoing server's data
by taking advantage of our ConnMux Read/Write wrapping

HTTP: total number of different http verbs passed in http
requests and different status codes passed in http responses.
This is counted in a new http handler.
2017-02-06 09:29:53 -08:00
Harshavardhana
6717a0b68c Add consistency guarantees 2017-02-05 18:31:48 -08:00
Harshavardhana
533338bdeb all/windows: Be case in-sensitive about pattern matching. (#3682)
Resource strings and paths are case insensitive on windows
deployments but if user happens to use upper case instead of
lower case for certain configuration params like bucket
policies and bucket notification config. We might not honor
them which leads to a wrong behavior on windows.

This is windows only behavior, for all other platforms case
is still kept sensitive.
2017-02-03 23:27:50 -08:00
Anis Elleuch
b6ebf2aba8 server-mux: Simplify graceful shutdown behavior (#3681)
`*http.Server` is no more used, doing some cleanup.
2017-02-03 22:53:30 -08:00
Anis Elleuch
ed4fcb63f7 Require content-length in POST & Upload requests (#3671)
Avoid passing size = -1 to PutObject API by requiring content-length
header in POST request (as AWS S3 does) and in Upload web handler.
Post handler is modified to completely store multipart file to know
its size before sending it to PutObject().
2017-02-02 10:45:00 -08:00
Harshavardhana
4b4cb07fb6 Update browser assets with new changes. 2017-02-02 03:01:49 -08:00
Harshavardhana
fc446e0b81 docs: Add browser doc for running dev in custom port (#3675)
Fixes #3674
2017-02-01 13:12:13 -08:00
Anis Elleuch
f612a7dd85 madmin: Fix a typo in Locks duration query name (#3673) 2017-02-01 11:46:49 -08:00
Krishnan Parthasarathi
0472e5c1e1 Change query param name to duration in list/clear locks API (#3664)
Following is a sample list lock API request schematic,

  /?lock&bucket=mybucket&prefix=myprefix&duration=holdDuration
  x-minio-operation: list

The response would contain the list of locks held on mybucket matching
myprefix for a duration longer than holdDuration.
2017-02-01 11:17:30 -08:00
Harshavardhana
6a6c930f5b xl: Abort multipart upload should honor quorum properly. (#3670)
Current implementation didn't honor quorum properly and didn't
handle the errors generated properly. This patch addresses that
and also moves common code `cleanupMultipartUploads` into xl
specific private function.

Fixes #3665
2017-02-01 11:16:17 -08:00
Karthic Rao
35ca3e5d9b Fix unresponsiveness of doneCh due to Sleep call. (#3667) 2017-02-01 09:17:32 -08:00
Harshavardhana
a9ab01731f docs: Remove reference word from notification title 2017-01-31 18:17:31 -08:00
Harshavardhana
cd80e6df29 docs: Move the notifications into docs/bucket
Cleanup some formatting issues.
2017-01-31 18:07:39 -08:00
koolhead17
4dea4f3b89 docs: added event-notification back end doc. (#3510) 2017-01-31 17:04:36 -08:00
Harshavardhana
1b30a3be2b xl/utils: getPartSizeFromIdx should return error. (#3669) 2017-01-31 15:34:49 -08:00
Krishna Srinivas
f7f103725b browser: Make logo and "Minio Browser" text non-clickable. (#3668)
fixes #3601
2017-01-31 13:44:07 -08:00
Harshavardhana
77a192a7b5 Implement CopyObjectPart API (#3663)
This API is implemented to allow copying data from an
existing source object to an ongoing multipart operation

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html

Fixes #3662
2017-01-31 09:38:34 -08:00
Harshavardhana
cb48517a78 Update browser assets with new changes. 2017-01-30 16:38:49 -08:00
Anis Elleuch
e9394dc22d xl PutObject: Split object into parts (#3651)
For faster time-to-first-byte when we try to download a big object
2017-01-30 15:44:42 -08:00
Krishna Srinivas
46743c7918 browser: Component attr names should not clash with redux state keys. (#3656)
fixes #270
2017-01-30 14:25:55 -08:00
Krishna Srinivas
b288eaddb3 xl: bit-rot algo was not set in get-object. (#3652)
fixes #3650
2017-01-30 14:25:28 -08:00
Krishnan Parthasarathi
2665aba555 Fail PutBucketPolicy if conditions are incompatible with actions. (#3659) 2017-01-30 09:20:16 -08:00
Krishnan Parthasarathi
9b6bcb30d9 Fix handling of StringNotEquals condition operator (#3660) 2017-01-30 09:18:10 -08:00
Harshavardhana
ac9ba13c19 build: Increase the travis build test timeout to 15mins 2017-01-30 01:39:53 -08:00
Krishnan Parthasarathi
864b8795aa heal: Should delete stale object parts before healing (#3649) 2017-01-30 00:45:56 -08:00
Bala FA
cc1575f944 fix: do port availability check only on macOS. (#3654)
On macOS, if a process already listens on 127.0.0.1:PORT, net.Listen() falls back
to IPv6 address ie minio will start listening on IPv6 address whereas another
(non-)minio process is listening on IPv4 of given port.
To avoid this error sutiation we check for port availability only for macOS.

Note: checkPortAvailability() tries to listen on given port and closes it.
It is possible to have a disconnected client in this tiny window of time.
2017-01-30 00:44:36 -08:00
Krishnan Parthasarathi
b408d0e87d Add aws:Referer condition key support. (#3641)
This change implements bucket policy enhancements required to restrict access based on HTTP referer.
See https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-4 for more information.

Fixes #3540
2017-01-29 19:45:11 -08:00
Nitish Tiwari
69b81af93e Fix Minio quick start guide URL (#3655) 2017-01-29 07:21:30 -08:00
Harshavardhana
cf558ff6d1 docs: Fix TLS doc to be docs.minio.io compatible 2017-01-28 22:46:29 -08:00
Harshavardhana
5af7bd6a01 docs: Fix a typo in TLS doc description 2017-01-28 22:41:52 -08:00
Harshavardhana
73f4f29110 docs: Restructure docs, move the files to their relevant location. (#3648)
Also combines windows TLS docs with single doc with Linux.
2017-01-28 19:45:30 -08:00
Harshavardhana
ccd949d8ca Add -v for race tests to avoid travis and appveyor timeouts. (#3647)
This is added so that CI's don't wrong fully timeout on
little longer running tests.
2017-01-27 12:31:02 -08:00
Anis Elleuch
e1bc99e4fe xl: Fix GET of an empty multiparted object (#3646)
GetObject returns unsatisfied range error when we try to download an object
uploaded using multipart mechanism.
2017-01-27 10:51:02 -08:00
Alex Ellis
02194ee3c6 Fix issue with webhook events not closing Response body, leaving connections open with remote server (#3645) 2017-01-27 09:50:08 -08:00
Harshavardhana
85f2b74cfd jwt: Cache the bcrypt password hash. (#3526)
Creds don't require secretKeyHash to be calculated
everytime, cache it instead and re-use.

This is an optimization for bcrypt.

Relevant results from the benchmark done locally, negative
value means improvement in this scenario.

```
benchmark                       old ns/op     new ns/op     delta
BenchmarkAuthenticateNode-4     160590992     80125647      -50.11%
BenchmarkAuthenticateWeb-4      160556692     80432144      -49.90%

benchmark                       old allocs     new allocs     delta
BenchmarkAuthenticateNode-4     87             75             -13.79%
BenchmarkAuthenticateWeb-4      87             75             -13.79%

benchmark                       old bytes     new bytes     delta
BenchmarkAuthenticateNode-4     15222         9785          -35.72%
BenchmarkAuthenticateWeb-4      15222         9785          -35.72%
```
2017-01-26 16:51:51 -08:00
Krishna Srinivas
152cdf1c05 fs: Move traceError() to lower functions where possible. (#3633) 2017-01-26 15:40:10 -08:00
Krishna Srinivas
17dd1c19df cleanup: refactor common code between FS and XL listDirFactory. (#3639) 2017-01-26 15:39:22 -08:00
Anis Elleuch
8e49a3d047 Simplify running cmd.Main() for external tests (#3636)
An external test that runs cmd.Main() has a difficulty to set cmd arguments
and MINIO_{ACCESS,SECRET}_KEY values, this commit changes a little the current
behavior in a way that helps external tests.
2017-01-26 15:22:41 -08:00
Krishna Srinivas
cccf77229d cleanup: Move code in *-multipart-common.go to *-multipart.go (#3638)
The code in *-multipart-common.go is not common anymore.
2017-01-26 12:51:12 -08:00
Harshavardhana
dafdc74605 fs: if fs.json is empty ignore it while reading metadata. (#3634)
This is needed so that we don't send wrong errors
on previously failed PutObject() which would have
left a stale `fs.json` entry.
2017-01-26 10:19:07 -08:00
Anis Elleuch
c753b5dfeb madmin: All APIs return ErrorResponse error type (#3632)
Avoid returning errors.New("Got HTTP Status") in all APIs
2017-01-25 16:01:29 -08:00
Krishna Srinivas
82373e3d50 fs: cleanup - do not cache size of metafiles (#3630)
* Remove Size() method and size field from lock.LockedFile
* WriteTo method of fsMeta and uploadsV1 now takes concrete type *lock.LockedFile
2017-01-25 12:29:06 -08:00
Anis Elleuch
76b8abcd8e madmin: Fix args order in listObjectsHeal() (#3629)
The order of marker and delimiter and in listObjectsHeal() internal function
are switched. That will give wrong result in case of a non recursive objects
heal list.
2017-01-25 09:58:49 -08:00
Harshavardhana
dbb5408906 Add ARM64 build binary for README.md
Fixes #3626
2017-01-24 20:28:54 -08:00
Harshavardhana
18c9d49a32 Build browser with new changes and update yarn.lock 2017-01-24 19:11:37 -08:00
Harshavardhana
d41dcb784b Move to blake2b-simd due to perf problems in golang.org/x/crypto
Ref https://github.com/golang/go/issues/18563
2017-01-24 18:07:34 -08:00
Krishna Srinivas
659d5aabd1 browser: Access to private paths redirects to login. (#3622) 2017-01-24 12:08:00 -08:00
Krishna Srinivas
b4343a28b7 browser: add yarn.lock and .gitignore files. (#3621) 2017-01-24 11:56:30 -08:00
Krishnan Parthasarathi
0e693e0284 Add dry-run query param for HealFormat API (#3618) 2017-01-24 08:11:05 -08:00
Anis Elleuch
fc880f9b23 admin: Enhance set credentials test (#3619)
Add more test cases and ignore access and secret keys set from the env
2017-01-24 08:08:36 -08:00
Krishna Srinivas
cead24b0f7 miniobrowser: Bring Minio browser source into minio repo. (#3617) 2017-01-23 18:07:22 -08:00
Krishna Srinivas
8489f22fe2 signature-v2: Use request.RequestURI for signature calculation. (#3616)
* signature-v2: Use request.RequestURI for signature calculation.

* Use splitStr instead of strings.Split
2017-01-23 17:01:44 -08:00
Anis Elleuch
fc6f804865 server-mux: Keep listening after Accept() err (#3613)
Accept() can return errors like: `too many open files`, no need to totally quit listening in this case.
2017-01-23 09:55:34 -08:00
Anis Elleuch
d1d89116f1 admin: Add version to service Status API response (#3605)
Add server's version field to service status API:

"version":{
	"version":"DEVELOPMENT.GOGET",
	"commitID":"DEVELOPMENT.GOGET"
}
2017-01-23 08:56:06 -08:00
Anis Elleuch
e1f64141a2 presign-v2: Compute signature on encoded URL path (#3612)
Encode the path of the passed presigned url before calculating the signature. This fixes
presigning objects whose names contain characters that are found encoded in urls.
2017-01-23 08:54:32 -08:00
Krishnan Parthasarathi
586058f079 Implement mgmt REST APIs to heal storage format. (#3604)
* Implement heal format REST API handler
* Implement admin peer rpc handler to re-initialize storage
* Implement HealFormat API in pkg/madmin
* Update pkg/madmin API.md to incl. HealFormat
* Added unit tests for ReInitDisks rpc handler and HealFormatHandler
2017-01-23 00:32:55 -08:00
Krishna Srinivas
4e926b292f vendor-update: Minio Browser (#3609) 2017-01-22 21:02:09 -08:00
Anis Elleuch
47358e3104 server-mux: Add tcp idle read timeout (#3607)
Avoid many idle client connections i.e client didn't send any data until a given stipulated amount of time. 

Default chosen here is 30 seconds.
2017-01-22 14:48:27 -08:00
Harshavardhana
3640c63289 server/mux: PeekProtocol() should return error and connection be closed. (#3608)
For TLS peekProtocol do not assume the incoming request to be a TLS
connection perform a handshake() instead and validate.

Also add some security related defaults to `tls.Config`.
2017-01-22 12:14:00 -08:00
Harshavardhana
51fa4f7fe3 Make PutObject a nop for an object which ends with "/" and size is '0' (#3603)
This helps majority of S3 compatible applications while not returning
an error upon directory create request.

Fixes #2965
2017-01-20 16:33:01 -08:00
Andrei Kopats
c3f7d1026f fs: start even if there are not enough free space (#3606) 2017-01-20 09:30:20 -08:00
Harshavardhana
80f1387877 Initialize peers properly for localhost. (#3600)
This introduced a regression.

Fixes #3594
2017-01-19 11:32:13 -08:00
Andreas Kohn
0674fa43ff Handle the region for GetBucketLocation and PutBucket properly (#3596)
This adjusts the code for these two handlers to match the logic in ListBucketHandler.

Fixes #3595
2017-01-19 11:31:51 -08:00
Harshavardhana
a17f1e875c server/mux: Close the connection even if buffer.Flush() returns error. (#3599)
It is possible that buf.Flush() might return an error, leading to a
potential leak in active sockets.
2017-01-19 11:19:57 -08:00
Jeffery Utter
9e1f1b50e0 Don't Check Available Inodes on NFS (#3598)
In some cases (such as with VirutualBox, this value gets hardcoded
to 1000, which is less than the required minimum of 10000.

Fixes #3592
2017-01-19 10:39:44 -08:00
Anis Elleuch
0715032598 heal: Add ListBucketsHeal object API (#3563)
ListBucketsHeal will list which buckets that need to be healed:
  * ListBucketsHeal() (buckets []BucketInfo, err error)
2017-01-19 09:34:18 -08:00
Harshavardhana
dfc2ef3004 storage/rpc: Remove network error restriction. (#3591)
This restriction has lots of side affects, since
we do not have a mechanism to clear states like
this it is better not to keep them.

Network errors are common and can occur with
simple cable removal etc. Since we already have
a retry mechanism this error count and stateful
nature can bring problems on a long running
cluster.
2017-01-18 12:55:57 -08:00
Harshavardhana
62f8343879 Add constants for commonly used values. (#3588)
This is a consolidation effort, avoiding usage
of naked strings in codebase. Whenever possible
use constants which can be repurposed elsewhere.

This also fixes `goconst ./...` reported issues.
2017-01-18 12:24:34 -08:00
Anis Elleuch
f803bb4b3d admin: Add service Set Credentials API (#3580) 2017-01-17 14:25:59 -08:00
Andy Brown
20a65981bd add delete and options methods to CORS whitelist (#3589) 2017-01-17 14:20:05 -08:00
Harshavardhana
09b450d610 Fix fs tests to avoid deleting /usr to certain systems. 2017-01-17 14:05:07 -08:00
Krishnan Parthasarathi
c194b9f5f1 Implement mgmt REST APIs for heal subcommands (#3533)
The heal APIs supported in this change are,
- listing of objects to be healed.
- healing a bucket.
- healing an object.
2017-01-17 10:02:58 -08:00
Harshavardhana
98a6a2bcab obj: Return objectInfo for CompleteMultipartUpload(). (#3587)
This patch avoids doing GetObjectInfo() in similar way
how we did for PutOject().
2017-01-16 19:23:43 -08:00
Harshavardhana
1c699d8d3f fs: Re-implement object layer to remember the fd (#3509)
This patch re-writes FS backend to support shared backend sharing locks for safe concurrent access across multiple servers.
2017-01-16 17:05:00 -08:00
Harshavardhana
a054c73e22 Add slack replace gitter (#3584) 2017-01-16 15:26:26 -08:00
Harshavardhana
b580ad24e2 server/http: Add missing keep alive for incoming tcp connections. (#3585)
This was seen reproducing a bug with @gowithplanb. Windows Cloud
Berry clients do not close their respective connections.
2017-01-16 03:38:06 -08:00
Harshavardhana
f8a3b1e164 Fix a bug in previous patch. 2017-01-16 01:48:34 -08:00
Harshavardhana
bf2b8879b7 config: Allow non-standard config dir to be configured with SSL. (#3583) 2017-01-15 16:53:01 -08:00
Harshavardhana
caecd75a2a Deprecate and remove service stop API. (#3578)
Fixes #3570
2017-01-14 14:48:52 -08:00
Anis Elleuch
2959c104b3 peer rpc: Fix typo in cluster credentials update (#3579)
Update credentials in cluster wasn't working due to a typo
2017-01-14 14:06:23 -08:00
Harshavardhana
50796e481d build: Add ARM binary builds for ARMv6 and Aarch64. (#3577) 2017-01-13 18:51:17 -08:00
Anis Elleuch
f64f8b03cb admin: Enhance locks list json response (#3573) 2017-01-13 14:25:34 -08:00
Harshavardhana
7b85756c64 notify/webhook: Handle webendpoints without port (#3568)
Fixes and issue initializing webhook notification

```
FATA[0000] Initializing object layer failed              cause=Unable to initialize event \
    notification. dial tcp: missing port in address requestb.in source=[server-main.go:448:serverMain()]
```
2017-01-12 20:08:00 -08:00
Alex
d6a327fbc5 Add notifications by webhook.
Add a new config entry moving to version 13.
```
		"webhook": {
			"1": {
				"enable": true,
				"address": "http://requestb.in/1i9al7m1"
			}
		}
```
2017-01-12 10:19:59 -08:00
Anis Elleuch
f24753812a nats: Add support of NATS.io Streaming server (#3494) 2017-01-11 16:41:05 -08:00
Harshavardhana
08b6cfb082 ssl: Set a global boolean to enable SSL across Minio (#3558)
We have been using `isSSL()` everywhere we can set
a global value once and re-use it again.
2017-01-11 13:59:51 -08:00
Krishna Srinivas
12a7a15daa browser: Allow anonymous browsing of readable buckets. (#3515) 2017-01-11 13:26:42 -08:00
Harshavardhana
7c6d77734a Fix vendor.json to remove minio/blake2b-simd ref 2017-01-11 00:26:00 -08:00
Harshavardhana
b0cfceb211 event: Enhance event message struct to provide origin server. (#3557)
`principalId` i.e user identity is kept as AccessKey in
accordance with S3 spec.

Additionally responseElements{} are added starting with

`x-amz-request-id` is a hexadecimal of the event time itself in nanosecs.
`x-minio-origin-server` - points to the server generating the event.

Fixes #3556
2017-01-10 16:43:48 -08:00
Harshavardhana
0563a9235a handlers: Handle crash if r.URL.Path is empty. (#3554)
URL paths can be empty and not have preceding separator,
we do not yet know the conditions this can happen inside
Go http server.

This patch is to ensure that we do not crash ourselves
under conditions where r.URL.Path may be empty.

Fixes #3553
2017-01-10 11:01:23 -08:00
Anis Elleuch
eb6d53d2f5 heal: Fix new entries computation in listDirHeal (#3551)
A crash was happening due to an incorrect interpreation of the return value of sort.SearchString()
2017-01-09 15:48:13 -08:00
Anis Elleuch
95d9e47353 Presign V2: Unescape non-std queries in urls (#3549)
A client sends escaped characters in values of some query parameters in a presign url.
This commit properly unescapes queires to fix signature calculation.
2017-01-09 14:22:20 -08:00
Harshavardhana
e1142e99f2 rpc/lock: Make sure to capitalize for proper marshalling. (#3544)
Distributed setup stopped working for certain
types of operations `6d10f4c19af6861e4de1b22ac20a3e5136f69d67`

This is a regression.

Fixes #3543
2017-01-08 20:37:53 -08:00
Harshavardhana
a091fe3ed6 docs: Fix caching docs to comply with current features. 2017-01-08 11:17:02 -08:00
koolhead17
f37f56ac15 docs: Added guide for distributed minio on Windows (#3507) 2017-01-07 18:35:12 -08:00
Karthic Rao
2f4a7483ea Test Function to reset globals. (#3538)
- Adding reset functions for important global variables.
- Using them in tests.
2017-01-07 11:27:01 -08:00
Harshavardhana
464f9d34d6 Remove all references to GO15VENDOREXPERIMENT 2017-01-06 18:42:32 -08:00
Harshavardhana
926c75d0b5 api: Set appropriate content-type for success/error responses. (#3537)
Golang HTTP client automatically detects content-type but
for S3 clients this content-type might be incorrect or
might misbehave.

For example:

```
Content-Type: text/xml; charset=utf-8
```

Should be
```
Content-Type: application/xml
```

Allow this to be set properly.
2017-01-06 00:37:00 -08:00
Krishnan Parthasarathi
c8f57133a4 Implement list, clear locks REST API w/ pkg/madmin support (#3491)
* Filter lock info based on bucket, prefix and time since lock was held
* Implement list and clear locks REST API
* madmin: Add list and clear locks API
* locks: Clear locks matching bucket, prefix, relTime.
* Gather lock information across nodes for both list and clear locks admin REST API.
* docs: Add lock API to management APIs
2017-01-03 23:39:22 -08:00
Harshavardhana
cae62ce543 browser: Handle proper login errors. (#3518)
Also additionally log the remote address.

Fixes #3514
2017-01-03 01:33:00 -08:00
Harshavardhana
7bbb532b4b Add a isErr function to check for errs.
DisksInfo() should handle collection of some
base errors as offlineDisks.
2017-01-02 10:52:43 -08:00
Bala.FA
6d10f4c19a Adopt dsync interface changes and major cleanup on RPC server/client.
* Rename GenericArgs to AuthRPCArgs
* Rename GenericReply to AuthRPCReply
* Remove authConfig.loginMethod and add authConfig.ServiceName
* Rename loginServer to AuthRPCServer
* Rename RPCLoginArgs to LoginRPCArgs
* Rename RPCLoginReply to LoginRPCReply
* Version and RequestTime are added to LoginRPCArgs and verified by
  server side, not client side.
* Fix data race in lockMaintainence loop.
2017-01-02 20:57:42 +05:30
Matthew Hall
cde6496172 checkdeps.sh: support unusual git version strings (for OS X et. al.) (#3512) 2016-12-31 19:57:33 -08:00
Harshavardhana
8562b22823 Fix delays and iterim fix for the partial fix in #3502 (#3511)
This patch uses a technique where in a retryable storage
before object layer initialization has a higher delay
and waits for longer period upto 4 times with time unit
of seconds.

And uses another set of configuration after the disks
have been formatted, i.e use a lower retry backoff rate
and retrying only once per 5 millisecond.

Network IO error count is reduced to a lower value i.e 256
before we reject the disk completely. This is done so that
combination of retry logic and total error count roughly
come to around 2.5secs which is when we basically take the
disk offline completely.

NOTE: This patch doesn't fix the issue of what if the disk
is completely dead and comes back again after the initialization.
Such a mutating state requires a change in our startup sequence
which will be done subsequently. This is an interim fix to alleviate
users from these issues.
2016-12-30 17:08:02 -08:00
Harshavardhana
dd68cdd802 Auto-reconnect for regular authRPC client. (#3506)
Implement a storage rpc specific rpc client,
which does not reconnect unnecessarily.

Instead reconnect is handled at a different
layer for storage alone.

Rest of the calls using AuthRPC automatically
reconnect, i.e upon an error equal to `rpc.ErrShutdown`
they dial again and call the requested method again.
2016-12-29 19:42:02 -08:00
Harshavardhana
41cf580bb1 Improve reconnection logic, allow jitters. (#3502)
Attempt a reconnect also if disk not found.

This is needed since any network operation error
is converted to disk not found but we also need
to make sure if disk is really not available. 

Additionally we also need to retry more than
once because the server might be in startup
sequence which would render other servers to
wrongly think that the server is offline.
2016-12-29 03:13:51 -08:00
Dee Koder
00baec1746 syntax highlighting : Remove sh from code block which incorrectly highlights shell commands. (#3504) 2016-12-28 16:07:05 -08:00
Harshavardhana
f9025652df Support building specific architectures. 2016-12-27 17:52:24 -08:00
Bala FA
ee0172dfe4 Have simpler JWT authentication. (#3501) 2016-12-27 08:28:10 -08:00
Harshavardhana
69559aa101 objAPI: Implement CopyObject API. (#3487)
This is written so that to simplify our handler code
and provide a way to only update metadata instead of
the data when source and destination in CopyObject
request are same.

Fixes #3316
2016-12-26 16:29:26 -08:00
Krishnan Parthasarathi
36fd317eb2 Clean up lock-instrumentation and improve comments (#3499)
- Add a lockStat type to group counters
- Remove unnecessary helper functions
- Fix stats computation on force unlock
- Removed unnecessary checks and cleaned up comments
2016-12-26 10:29:55 -08:00
Bala FA
e8ce3b64ed Generate and use access/secret keys properly (#3498) 2016-12-26 10:21:23 -08:00
Andreas Auernhammer
6ee27daac1 fix blake2b tests on non-amd64 machines (#3496)
Fix the TestHashes Test for non-amd64 machines
2016-12-25 17:03:08 -08:00
koolhead17
cdc6c2d578 docs: Removed unmaintained docs. (#3493)
* docs: Removed unmaintained docs.

* docs: removed associated screenshots with the older docs.
2016-12-23 22:06:01 -08:00
Harshavardhana
855ef4f1aa Fix typo in erasure-utils.go 2016-12-22 10:47:10 -08:00
Harshavardhana
5878fcc086 bit-rot: Default to sha256 on ARM64. (#3488)
This is to utilize an optimized version of
sha256 checksum which @fwessels implemented.

blake2b lacks such optimizations on ARM platform,
this can provide us significant boost in performance.

blake2b on ARM64 as expected would be slower.
```
BenchmarkSize1K-4           	   30000	     44015 ns/op	  23.26 MB/s
BenchmarkSize8K-4           	    5000	    335448 ns/op	  24.42 MB/s
BenchmarkSize32K-4          	    1000	   1333960 ns/op	  24.56 MB/s
BenchmarkSize128K-4         	     300	   5328286 ns/op	  24.60 MB/s
```

sha256 on ARM64 is faster by orders of magnitude giving close to
AVX performance of blake2b.
```
BenchmarkHash8Bytes-4	 1000000	      1446 ns/op	   5.53 MB/s
BenchmarkHash1K-4    	  500000	      3229 ns/op	 317.12 MB/s
BenchmarkHash8K-4    	  100000	     14430 ns/op	 567.69 MB/s
BenchmarkHash1M-4    	    1000	   1640126 ns/op	 639.33 MB/s
```
2016-12-22 08:25:03 -08:00
Andreas Auernhammer
1ac36a95aa replace blake2b implementation (#3481)
* replace blake2b implementation
replace the blake2b-simd with the golang/x/crypto implementation

```
name        old time/op    new time/op     delta
Size64-8       715ns ±13%      614ns ± 3%    ~     (p=0.084 n=6+6)
Size128-8      612ns ± 5%      634ns ± 8%    ~     (p=0.084 n=6+6)
Size1K-8      2.18µs ± 5%     2.09µs ± 7%    ~     (p=0.084 n=6+6)
Size8K-8      13.1µs ± 2%     13.4µs ± 3%    ~     (p=0.084 n=6+6)
Size32K-8     48.5µs ± 1%     49.5µs ± 3%    ~     (p=0.775 n=6+6)
Size128K-8     199µs ± 0%      198µs ± 3%    ~     (p=0.468 n=6+6)

name        old speed      new speed       delta
Size64-8    92.6MB/s ±11%  104.2MB/s ± 3%    ~     (p=0.139 n=6+6)
Size128-8    208MB/s ± 6%    202MB/s ± 8%    ~     (p=0.102 n=6+6)
Size1K-8     466MB/s ± 7%    492MB/s ± 7%    ~     (p=0.139 n=6+6)
Size8K-8     621MB/s ± 2%    610MB/s ± 3%    ~     (p=0.102 n=6+6)
Size32K-8    672MB/s ± 2%    669MB/s ± 1%    ~     (p=0.818 n=6+6)
Size128K-8   657MB/s ± 1%    672MB/s ± 0%  +2.28%  (p=0.002 n=6+6)

name        old time/op   new time/op   delta
Size64-4      334ns ± 1%    243ns ± 0%  -27.14%  (p=0.029 n=4+4)
Size128-4     296ns ± 1%    242ns ± 0%  -18.21%  (p=0.029 n=4+4)
Size1K-4     1.44µs ± 0%   1.28µs ± 0%  -10.83%  (p=0.029 n=4+4)
Size8K-4     10.0µs ± 0%    9.4µs ± 0%   -6.23%  (p=0.029 n=4+4)
Size32K-4    39.8µs ± 1%   37.3µs ± 0%   -6.31%  (p=0.029 n=4+4)
Size128K-4    162µs ± 3%    149µs ± 0%   -7.72%  (p=0.029 n=4+4)

name        old speed     new speed     delta
Size64-4    192MB/s ± 1%  263MB/s ± 0%  +37.24%  (p=0.029 n=4+4)
Size128-4   431MB/s ± 0%  526MB/s ± 0%  +22.04%  (p=0.029 n=4+4)
Size1K-4    713MB/s ± 0%  800MB/s ± 0%  +12.17%  (p=0.029 n=4+4)
Size8K-4    815MB/s ± 0%  869MB/s ± 0%   +6.64%  (p=0.029 n=4+4)
Size32K-4   823MB/s ± 1%  878MB/s ± 0%   +6.72%  (p=0.029 n=4+4)
Size128K-4  810MB/s ± 3%  877MB/s ± 0%   +8.23%  (p=0.029 n=4+4)
```
See: https://go-review.googlesource.com/#/c/34319/
2016-12-21 14:20:01 -08:00
Harshavardhana
15b4c49621 fs/xl: Simplify bucket metadata reading. (#3486)
ObjectLayer GetObject() now returns the entire object
if starting offset is 0 and length is negative. This
also allows to simplify handler layer code where
we always had to use GetObjectInfo() before proceeding
to read bucket metadata files examples `policy.json`.

This also reduces one additional call overhead.
2016-12-21 11:29:32 -08:00
Harshavardhana
f57f773189 admin: Add missing madmin examples and API docs. (#3483) 2016-12-20 18:49:48 -08:00
Harshavardhana
e7b4e4e105 admin: ServiceStatus() shouldn't have to write double http headers. (#3484)
Fixes #3482
2016-12-20 18:05:25 -08:00
Harshavardhana
1d134c1a94 admin: Fix wrong vendorized location for s3signer and s3utils. 2016-12-20 15:18:37 -08:00
Anis Elleuch
329a910b86 Admin Lib: Implement Service API (#3426)
Three APIs were added to control a minio server

* NewAdminClient()
* ServiceStop()
* ServiceRestart()
* ServiceStatus()
2016-12-20 14:45:17 -08:00
Harshavardhana
4309727354 docs: Add docs for minio limitations. (#3477) 2016-12-20 14:42:13 -08:00
Anis Elleuch
ef3319a49d Fix: Typo in non canonicalized header extraction (#3480)
Extracting metadata from headers was doing wrong when Headers are not well canonicalized, fixing typo.
2016-12-20 11:59:08 -08:00
Anis Elleuch
d8e4d3c9c8 POSTForm: Return http 303 if redirect is specified (#3479)
success_action_redirect in the sent Form means that the server needs to return 303 in addition to a well specific redirection url, this commit adds this feature
2016-12-20 09:32:17 -08:00
Harshavardhana
faa6b1e925 vendorize deps for snappy, blake2b and sha256 (#3476)
Bring in new optimization and portability changes.

Fixes https://github.com/minio/minio-go/issues/578
2016-12-19 19:32:55 -08:00
Krishnan Parthasarathi
85c6bb9809 server: Sort disk arguments for consistent ordering (#3469)
This is important in a distributed setup, where the server hosting the
first disk formats a fresh setup. Sorting ensures that all servers
arrive at the same 'first' server.

Note: This change doesn't protect against different disk arguments
with some disks being same across servers.
2016-12-19 17:04:31 -08:00
Anis Elleuch
5404dddcea PostForm: Save supported headers in obj metadata (#3474)
Supported Headers like Content-Type, Cache-Control, Content-Encoding, X-Amz-* , etc.. are now saved in object metadata
2016-12-19 16:14:04 -08:00
Anis Elleuch
4692fdbb8f PostForm: Follow success_action_status requirement (#3467)
S3 spec requires that Post Object response depends on the passed success_action_status. This commit implements that requirement.
2016-12-18 13:39:56 -08:00
Bala FA
1875a47495 rpcclient: fix leaky connection (#3471)
Previously, more than one goroutine calls RPCClient.dial(), each
goroutine gets a new rpc.Client but only one such client is stored
into RPCClient object.  This leads to leaky connection at the server
side.  This is fixed by taking lock at top of dial() and release on
return.
2016-12-17 18:17:40 -08:00
Harshavardhana
9c9f390350 fs: validate filesystem path argument properly. (#3470)
FS should fail for invalid paths like

 - file:///
 - ftp://
 - http://
2016-12-17 13:43:26 -08:00
Bala FA
1b2b16998f Remove regexp usage (#3456)
This patch removes regexp usage in cacheControlHandler.ServeHTTP() and
server-main.checkEndpointsSyntax()
2016-12-17 11:00:16 -08:00
koolhead17
7a17b2a585 Browser: Added character limit for Bucket Name. (#3454) 2016-12-16 09:59:37 -08:00
Aditya Manthramurthy
f7766b49aa Amend ZFS line in README - to make it about Minio on FreeNAS (#3461) 2016-12-16 09:59:02 -08:00
Krishnan Parthasarathi
b2f920a868 Add service API handler stubs for status, stop and restart (#3417) 2016-12-15 22:26:15 -08:00
Anis Elleuch
8ceb969445 tests: Use testTmpDir var to specify tmp directory (#3459)
To be able to specify the directory where tests will be done.  This way,
it will be easier to run Minio tests on a mounted directory like NFS, ..
2016-12-15 22:25:05 -08:00
Harshavardhana
0db484c8f6 signv2: Do not use path encoding for query values. (#3458)
Use query unescape before comparing signature.
2016-12-15 14:56:18 -08:00
dcharbonnier
5c481fbf6e Fix Minio Quickstart Guide link (#3457) 2016-12-15 11:59:16 -08:00
Aditya Manthramurthy
8e6e9301ce Add support for Kafka as a notifications target (#2869) (#3439) 2016-12-15 08:23:48 -08:00
Harshavardhana
664ff063a1 server: checkEndpoints syntax properly. (#3451) 2016-12-14 20:42:19 -08:00
koolhead17
d9fd6f9a96 docs: Removed $/# from code block to make code copy easier. (#3448) 2016-12-14 17:45:47 -08:00
Harshavardhana
b28ff50126 lock/server: Check if the lock server itself is skewed back. (#3447) 2016-12-14 08:02:32 -08:00
Karthic Rao
3fe2d77b70 Adding functions for resetting globals. (#3421) 2016-12-13 11:51:48 -08:00
Krishnan Parthasarathi
ab49498fc3 server: Exit gracefully if no endpoint is local to it. (#3442) 2016-12-13 11:18:31 -08:00
Minio Trusted
46fdd70114 Fix docker file and avoid creating README.md 2016-12-13 09:31:19 -08:00
Krishnan Parthasarathi
29d72b84c0 loginServer should be a member of lockServer (#3441)
Add a unit test to catch a missing LoginHandler method in lockServer.
2016-12-13 08:01:47 -08:00
Anand Babu (AB) Periasamy
f53fcdf10e keep make messages consistent (#3437) 2016-12-13 00:26:45 -08:00
Krishna Srinivas
8e665105b1 Presign-v4: Allow requests that were signed slightly ahead of the current time. (#3435) 2016-12-12 16:32:00 -08:00
Harshavardhana
2062add05f fs/posix: On windows use helpers and init format.json properly. (#3434)
Fixes #3433
2016-12-12 15:43:41 -08:00
Krishnan Parthasarathi
6b4e6bcebf Move LoginHandler into LoginServer which others embed (#3431)
* Move LoginHandler into LoginServer which others embed

* Add unit tests for loginServer
2016-12-12 08:11:23 -08:00
Harshavardhana
2d6f8153fa format: Check properly for disks in valid formats. (#3427)
There was an error in how we validated disk formats,
if one of the disk was formatted and was formatted with
FS would cause confusion and object layer would never
initialize essentially go into an infinite loop.

Validate pre-emptively and also check for FS format
properly.
2016-12-11 15:18:55 -08:00
Anis Elleuch
5c10f4adf0 presign v2: include resp headers in signature calc (#3428)
Include response headers when presigning an url using signature v2 algorithm
2016-12-11 14:32:25 -08:00
Harshavardhana
4daa0d2cee lock: Moving locking to handler layer. (#3381)
This is implemented so that the issues like in the
following flow don't affect the behavior of operation.

```
GetObjectInfo()
.... --> Time window for mutation (no lock held)
.... --> Time window for mutation (no lock held)
GetObject()
```

This happens when two simultaneous uploads are made
to the same object the object has returned wrong
info to the client.

Another classic example is "CopyObject" API itself
which reads from a source object and copies to
destination object.

Fixes #3370
Fixes #2912
2016-12-10 16:15:12 -08:00
Harshavardhana
cd0f350c02 env: Bring back MINIO_BROWSER env. (#3423)
Set MINIO_BROWSER=off to disable web browser completely.

Fixes #3422
2016-12-10 00:42:22 -08:00
Krishna Srinivas
ac554bf663 FS/Multipart: Fix race between PutObjectPart and Complete/Abort multi… (#3419)
FS/Multipart: Fix race between PutObjectPart and Complete/Abort multipart. close(timeoutCh) on complete/abort so that a racing PutObjectPart does not leave a dangling go-routine.

Fixes #3351
2016-12-09 16:10:18 -08:00
Harshavardhana
b363709c11 caching: Optimize memory allocations. (#3405)
This change brings in changes at multiple places

 - Reuse buffers at almost all locations ranging
   from rpc, fs, xl, checksum etc.
 - Change caching behavior to disable itself
   under low memory conditions i.e < 8GB of RAM.
 - Only objects cached are of size 1/10th the size
   of the cache for example if 4GB is the cache size
   the maximum object size which will be cached
   is going to be 400MB. This change is an
   optimization to cache more objects rather
   than few larger objects.
 - If object cache is enabled default GC
   percent has been reduced to 20% in lieu
   with newly found behavior of GC. If the cache
   utilization reaches 75% of the maximum value
   GC percent is reduced to 10% to make GC
   more aggressive.
 - Do not use *bytes.Buffer* due to its growth
   requirements. For every allocation *bytes.Buffer*
   allocates an additional buffer for its internal
   purposes. This is undesirable for us, so
   implemented a new cappedWriter which is capped to a
   desired size, beyond this all writes rejected.

Possible fix for #3403.
2016-12-08 20:35:07 -08:00
Anis Elleuch
410b579e87 startup: Show elapsed time in disks format process (#3413) 2016-12-07 10:22:00 -08:00
Karthic Rao
7b7c0bba58 Use a non member mutex lock for serverConfig access. (#3411)
- This is to ensure that the any new config references made to the
  serverConfig is also backed by a mutex lock.
- Otherwise any new config assigment will also replace the member mutex
  which is currently used for safe access.
2016-12-07 03:41:54 -08:00
Anis Elleuch
0cef971832 Fix max cache size calculation when system RAM is inferior to the default cache size (#3410) 2016-12-06 16:09:26 -08:00
Anis Elleuch
5c9a95df32 srv-mux: do not print peek protocol EOF err msg (#3402)
EOF err message in Peek Protocol is shown when a client closes the
connection in the middle of peek protocol, this commit hides it since it
doesn't make sense to show it
2016-12-05 14:49:32 -08:00
Anis Elleuch
3b455d6137 tests: Add tests for xl-v1-list-objects-heal (#3399) 2016-12-05 09:40:33 -08:00
Anis Elleuch
b2a0e5754b bucket-handlers: More tests for post form handler (#3392) 2016-12-04 12:23:19 -08:00
Anis Elleuch
63d9bb626a postform: fix check when ${filename} is provided (#3391)
Checking key condition when ${filename} is provided wasn't working well,
this patch fixes the wrong behavior
2016-12-04 10:30:52 -08:00
Anis Elleuch
372da5eaf5 tests: Enhance checkPostPolicy() coverage (#3389) 2016-12-03 12:41:07 -08:00
Harshavardhana
cf17fc7774 fs: PutObject create 0byte objects properly. (#3387)
Current code always appends to a file only if 1byte or
more was sent on the wire was affecting both PutObject
and PutObjectPart uploads.

This patch fixes such a situation and resolves #3385
2016-12-03 11:53:12 -08:00
Krishnan Parthasarathi
67509453d3 FS: sync abortMultipart cleanup and bg append (#3388)
backgroundAppend type's abort method should wait for appendParts to finish
writing ongoing appending of parts in the background before cleaning up
the part files.
2016-12-02 23:33:06 -08:00
Harshavardhana
d31f256020 Fail on lint errors during CI build. 2016-12-02 18:08:12 -08:00
Harshavardhana
d67f47927c api: Fix the formatting issues in last patch. 2016-12-02 17:39:21 -08:00
Anis Elleuch
85bb5870a9 Post Policy Form: exhaustive post policy check (#3386)
Add support of all conditions check described in
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html
and simplify a little bit the existing code
2016-12-02 17:00:33 -08:00
Harshavardhana
4a9b205a15 docs: Add missing comments for exported functions. 2016-12-02 14:39:23 -08:00
koolhead17
f3b346cbb3 docs: removed the Edge tag reference. (#3366) 2016-12-02 10:47:18 -08:00
Harshavardhana
ff4ce0ee14 fs/xl: Combine input checks into re-usable functions. (#3383)
Repeated code around both object layers are moved
and combined into simple re-usable functions.
2016-12-01 23:15:17 -08:00
Anis Elleuch
918924796f fs: Enable shutdown test with faulty disks (#3380) 2016-12-01 13:59:06 -08:00
Krishnan Parthasarathi
feb6685359 posix: Use preparePath only for paths used with syscall or os functions (#3377) 2016-11-30 20:56:15 -08:00
Bala FA
0d59ea1e94 postpolicy: handle Amazon S3 compatible content-length-range condition (#3376)
Previously minio server expects content-length-range values as integer
in JSON.  However Amazon S3 handles content-length-range values as
integer and strings.

This patch adds support for string values.
2016-11-30 18:30:59 -08:00
Krishna Srinivas
38edd94282 ListBuckets: Allow listBuckets request to be signed with region configured in config.json (#3374)
Fixes #3373
2016-11-30 13:55:56 -08:00
Krishna Srinivas
8021061bd8 Implement BucketUpdater interface to call BucketMetaState methods. (#3375) 2016-11-30 13:37:38 -08:00
Anand Babu (AB) Periasamy
bc9509bc8a edits on Limits 2016-11-30 01:58:31 -08:00
Krishna Srinivas
e3b4910b66 FS/CompleteMultipart: lock the namespace before renaming the appended tmp file. (#3371) 2016-11-29 23:26:36 -08:00
Harshavardhana
d056f19d07 api: Allow reconnection of policy/notification rpc clients. (#3368)
Since we moved out reconnection logic from net-rpc-client.go
we should do it from the top-layer properly and bring back
the code to reconnect properly in-case the connection is lost.
2016-11-29 22:39:32 -08:00
Harshavardhana
834007728c fs: Do not print redundant md5Sum response header. (#3369)
For both GET and HEAD requests.
2016-11-29 16:47:01 -08:00
Krishnan Parthasarathi
a609a4126c Vendorize recent fixes in dsync (#3365)
* Update 'i' only if lock grant read from buffered channel
2016-11-28 23:03:46 -08:00
Krishna Srinivas
bcd1a2308b FS/Shutdown: cleanup and delete .minio.sys during Shutdown() (#3360) 2016-11-28 22:54:48 -08:00
koolhead17
694bad434c docs: Modified the Docker doc to reflect distributed release image & (#3362)
modification.
2016-11-28 12:45:35 -08:00
Anis Elleuch
01f625824a args: Honor config-dir & quiet wherever they are (#3356)
setGlobalsFromContext() is added to set global variables after parsing
command line arguments. Thus, global flags will be honored wherever
they are placed in minio command.
2016-11-28 12:15:36 -08:00
Bala FA
9ccfb70104 Minor cleanup. (#3361) 2016-11-28 12:14:24 -08:00
Harshavardhana
201a20ac02 handlers: Handle re-direction properly for S3 requests. (#3355)
Make sure all S3 signature requests are not re-directed
to `/minio`. This should be only done for JWT and some
Anonymous requests.

This also fixes a bug found from https://github.com/bji/libs3

```
$ s3 -u list

ERROR: XmlParseFailure
```

Now after this fix shows proper output
```
$ s3 -u list
                         Bucket                                 Created
--------------------------------------------------------  --------------------
andoria                                                   2016-11-27T08:19:06Z
```
2016-11-27 16:30:46 -08:00
Krishna Srinivas
f3322e94c8 FS: Skip creating fs.json for objects created by minio (ex. policy.json) (#3353) 2016-11-27 11:33:08 -08:00
1574 changed files with 157271 additions and 134965 deletions

View File

@@ -24,10 +24,14 @@
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used:
* Version used (`minio version`):
* Environment name and version (e.g. nginx 1.9.1):
* Server type and version:
* Operating System and version:
* Operating System and version (`uname -a`):
* Link to your project:

46
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View File

@@ -0,0 +1,46 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage
assignees: ''
---
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1.
2.
3.
4.
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio version`):
* Environment name and version (e.g. nginx 1.9.1):
* Server type and version:
* Operating System and version (`uname -a`):
* Link to your project:

View File

@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: community, triage
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

View File

@@ -7,6 +7,10 @@
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
## Regression
<!-- Is this PR fixing a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
@@ -24,5 +28,6 @@
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have added tests to cover my changes.
- [ ] I have added unit tests to cover my changes.
- [ ] I have added/updated functional tests in [mint](https://github.com/minio/mint). (If yes, add `mint` PR # here: )
- [ ] All new and existing tests passed.

10
.gitignore vendored
View File

@@ -9,10 +9,18 @@ site/
/.idea/
/Minio.iml
**/access.log
build
build/
vendor/**/*.js
vendor/**/*.json
release
.DS_Store
*.syso
coverage.txt
.vscode/
*.tar.bz2
parts/
prime/
stage/
.sia_temp/
config.json
healthcheck

4
.jshintrc Normal file
View File

@@ -0,0 +1,4 @@
{
"asi": true,
"esnext": true
}

View File

@@ -6,13 +6,13 @@
#
# For explanation on this file format: man git-shortlog
Anand Babu (AB) Periasamy <ab@minio.io> Anand Babu (AB) Periasamy <abperiasamy@users.noreply.github.com>
Anand Babu (AB) Periasamy <ab@minio.io> <ab@unlocksmith.org>
Anand Babu (AB) Periasamy <ab@min.io> Anand Babu (AB) Periasamy <abperiasamy@users.noreply.github.com>
Anand Babu (AB) Periasamy <ab@min.io> <ab@unlocksmith.org>
Anis Elleuch <vadmeste@gmail.com>
Frederick F. Kautz IV <fkautz@minio.io> <fkautz@alumni.cmu.edu>
Harshavardhana <harsha@minio.io> <harsha@harshavardhana.net>
Harshavardhana <harsha@minio.io> <badger@gitter.im>
Harshavardhana <harsha@minio.io>
Krishna Srinivas <krishna@minio.io> <krishna.srinivas@gmail.com>
Frederick F. Kautz IV <fkautz@min.io> <fkautz@alumni.cmu.edu>
Harshavardhana <harsha@min.io> <harsha@harshavardhana.net>
Harshavardhana <harsha@min.io> <badger@gitter.im>
Harshavardhana <harsha@min.io>
Krishna Srinivas <krishna@min.io> <krishna.srinivas@gmail.com>
Matthew Farrellee <matt@cs.wisc.edu>
Nate Rosenblum <flander@gmail.com>

View File

@@ -1,23 +1,59 @@
go_import_path: github.com/minio/minio
sudo: required
dist: trusty
language: go
os:
- linux
# this ensures PRs based on a local branch are not built twice
# the downside is that a PR targeting a different branch is not built
# but as a workaround you can add the branch to this list
branches:
only:
- master
env:
- ARCH=x86_64
- ARCH=i686
matrix:
include:
- os: linux
dist: trusty
sudo: required
env:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
# Enable build cache
# https://restic.net/blog/2018-09-02/travis-build-cache
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- $HOME/go/pkg/mod
script:
- make test GOFLAGS="-race"
- make coverage
go: 1.12.1
script:
- make
- diff -au <(gofmt -s -d cmd) <(printf "")
- diff -au <(gofmt -s -d pkg) <(printf "")
- for d in $(go list ./... | grep -v browser); do CGO_ENABLED=1 go test -v -race --timeout 15m "$d"; done
- make verifiers
- make crosscompile
- make verify
- make coverage
- cd browser && yarn && yarn test && cd ..
- os: windows
env:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
go: 1.12.1
script:
- go build --ldflags="$(go run buildscripts/gen-ldflags.go)" -o %GOPATH%\bin\minio.exe
- bash buildscripts/go-coverage.sh
before_script:
# Add an IPv6 config - see the corresponding Travis issue
# https://github.com/travis-ci/travis-ci/issues/8361
- if [[ "${TRAVIS_OS_NAME}" == "linux" ]]; then sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'; fi
before_install:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then nvm install 11.10.1 ; fi
after_success:
- bash <(curl -s https://codecov.io/bash)
go:
- 1.7.3

View File

@@ -1,71 +1,68 @@
### Install Golang
# MinIO Contribution Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
If you do not have a working Golang environment setup please follow [Golang Installation Guide](https://docs.minio.io/docs/how-to-install-golang).
``MinIO`` community welcomes your contribution. To make the process as seamless as possible, we recommend you read this contribution guide.
## Development Workflow
Start by forking the MinIO GitHub repository, make changes in a branch and then send a pull request. We encourage pull requests to discuss code changes. Here are the steps in details:
### Setup your MinIO GitHub Repository
Fork [MinIO upstream](https://github.com/minio/minio/fork) source repository to your own personal repository. Copy the URL of your MinIO fork (you will need it for the `git clone` command below).
### Setup your Minio Github Repository
Fork [Minio upstream](https://github.com/minio/minio/fork) source repository to your own personal repository. Copy the URL for minio from your personal github repo (you will need it for the `git clone` command below).
```sh
$ mkdir -p $GOPATH/src/github.com/minio
$ cd $GOPATH/src/github.com/minio
$ git clone <paste saved URL for personal forked minio repo>
$ git clone https://github.com/minio/minio
$ go install -v
$ ls /go/bin/minio
```
### Set up git remote as ``upstream``
```sh
$ cd minio
```
### Compiling Minio from source
Minio uses ``Makefile`` to wrap around some of redundant checks done through command line.
```sh
$ make
Checking if proper environment variables are set.. Done
...
Checking dependencies for Minio.. Done
Installed govet
Building Libraries
...
...
```
### Setting up git remote as ``upstream``
```sh
$ cd $GOPATH/src/github.com/minio/minio
$ git remote add upstream https://github.com/minio/minio
$ git fetch upstream
$ git merge upstream/master
...
...
$ make
Checking if proper environment variables are set.. Done
...
Checking dependencies for Minio.. Done
Installed govet
Building Libraries
...
```
### Developer Guidelines
``Minio`` community welcomes your contribution. To make the process as seamless as possible, we ask for the following:
* Go ahead and fork the project and make your changes. We encourage pull requests to discuss code changes.
- Fork it
- Create your feature branch (git checkout -b my-new-feature)
- Commit your changes (git commit -am 'Add some feature')
- Push to the branch (git push origin my-new-feature)
- Create new Pull Request
### Create your feature branch
Before making code changes, make sure you create a separate branch for these changes
* If you have additional dependencies for ``Minio``, ``Minio`` manages its dependencies using [govendor](https://github.com/kardianos/govendor)
- Run `go get foo/bar`
- Edit your code to import foo/bar
- Run `make pkg-add PKG=foo/bar` from top-level directory
```
$ git checkout -b my-new-feature
```
* If you have dependencies for ``Minio`` which needs to be removed
- Edit your code to not import foo/bar
- Run `make pkg-remove PKG=foo/bar` from top-level directory
### Test MinIO server changes
After your code changes, make sure
* When you're ready to create a pull request, be sure to:
- Have test cases for the new code. If you have questions about how to do it, please ask in your pull request.
- Run `make verifiers`
- Squash your commits into a single commit. `git rebase -i`. It's okay to force update your pull request.
- Make sure `go test -race ./...` and `go build` completes.
- To add test cases for the new code. If you have questions about how to do it, please ask on our [Slack](https://slack.min.io) channel.
- To run `make verifiers`
- To squash your commits into a single commit. `git rebase -i`. It's okay to force update your pull request.
- To run `go test -race ./...` and `go build` completes.
* Read [Effective Go](https://github.com/golang/go/wiki/CodeReviewComments) article from Golang project
- `Minio` project is fully conformant with Golang style
- if you happen to observe offending code, please feel free to send a pull request
### Commit changes
After verification, commit your changes. This is a [great post](https://chris.beams.io/posts/git-commit/) on how to write useful commit messages
```
$ git commit -am 'Add some feature'
```
### Push to the branch
Push your locally committed changes to the remote origin (your fork)
```
$ git push origin my-new-feature
```
### Create a Pull Request
Pull requests can be created via GitHub. Refer to [this document](https://help.github.com/articles/creating-a-pull-request/) for detailed steps on how to create a pull request. After a Pull Request gets peer reviewed and approved, it will be merged.
## FAQs
### How does ``MinIO`` manages dependencies?
``MinIO`` uses `go mod` to manage its dependencies.
- Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
To remove a dependency
- Edit your code and remove the import reference.
- Run `go mod tidy` in the source folder to remove dependency from `go.mod` file.
### What are the coding guidelines for MinIO?
``MinIO`` is fully conformant with Golang style. Refer: [Effective Go](https://github.com/golang/go/wiki/CodeReviewComments) article from Golang project. If you observe offending code, please feel free to send a pull request or ping us on [Slack](https://slack.min.io).

View File

@@ -1,18 +1,37 @@
FROM golang:1.7-alpine
FROM golang:1.12-alpine
WORKDIR /go/src/app
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY . /go/src/app
ENV GOPATH /go
ENV CGO_ENABLED 0
ENV GO111MODULE on
RUN \
apk add --no-cache git && \
go-wrapper download && \
go-wrapper install -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
mkdir -p /export/docker && \
cp /go/src/app/docs/docker/README.md /export/docker/ && \
rm -rf /go/pkg /go/src && \
apk del git
RUN \
apk add --no-cache git && \
git clone https://github.com/minio/minio && cd minio && \
go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
cd dockerscripts; go build -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go
FROM alpine:3.9
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
EXPOSE 9000
ENTRYPOINT ["minio"]
VOLUME ["/export"]
COPY --from=0 /go/bin/minio /usr/bin/minio
COPY --from=0 /usr/bin/healthcheck /usr/bin/healthcheck
COPY dockerscripts/docker-entrypoint.sh /usr/bin/
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/data"]
HEALTHCHECK --interval=1m CMD healthcheck
CMD ["minio"]

27
Dockerfile.dev Normal file
View File

@@ -0,0 +1,27 @@
FROM alpine:3.9
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY dockerscripts/docker-entrypoint.sh dockerscripts/healthcheck /usr/bin/
COPY minio /usr/bin/
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck
EXPOSE 9000
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/data"]
HEALTHCHECK --interval=1m CMD healthcheck
CMD ["minio"]

39
Dockerfile.release Normal file
View File

@@ -0,0 +1,39 @@
FROM golang:1.12-alpine
ENV GOPATH /go
ENV CGO_ENABLED 0
ENV GO111MODULE on
RUN \
apk add --no-cache git && \
git clone https://github.com/minio/minio && cd minio/dockerscripts && \
go build -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go
FROM alpine:3.9
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY --from=0 /usr/bin/healthcheck /usr/bin/healthcheck
COPY dockerscripts/docker-entrypoint.sh /usr/bin/
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
curl https://dl.min.io/server/minio/release/linux-amd64/minio > /usr/bin/minio && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck
EXPOSE 9000
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/data"]
HEALTHCHECK --interval=1m CMD healthcheck
CMD ["minio"]

72
Dockerfile.simpleci Normal file
View File

@@ -0,0 +1,72 @@
#-------------------------------------------------------------
# Stage 1: Build and Unit tests
#-------------------------------------------------------------
FROM golang:1.12
COPY . /go/src/github.com/minio/minio
WORKDIR /go/src/github.com/minio/minio
RUN apt-get update && apt-get install -y jq
ENV GO111MODULE=on
RUN git config --global http.cookiefile /gitcookie/.gitcookie
RUN apt-get update && \
apt-get -y install sudo
RUN touch /etc/sudoers
RUN echo "root ALL=(ALL) ALL" >> /etc/sudoers
RUN echo "ci ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN echo "Defaults env_reset" >> /etc/sudoers
RUN echo 'Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go:/usr/local/go/bin"' >> /etc/sudoers
RUN mkdir -p /home/ci/.cache
RUN groupadd -g 999 ci && \
useradd -r -u 999 -g ci ci && \
chown -R ci:ci /go /home/ci && \
chmod -R a+rw /go
USER ci
# -- tests --
RUN make
RUN bash -c 'diff -au <(gofmt -s -d cmd) <(printf "")'
RUN bash -c 'diff -au <(gofmt -s -d pkg) <(printf "")'
RUN for d in $(go list ./... | grep -v browser); do go test -v -race --timeout 15m "$d"; done
RUN make verifiers
RUN make crosscompile
RUN make verify
#-------------------------------------------------------------
# Stage 2: Test Frontend
#-------------------------------------------------------------
FROM node:10.15-stretch-slim
COPY browser /minio/browser
WORKDIR /minio/browser
RUN yarn
RUN yarn test
#-------------------------------------------------------------
# Stage 3: Run Gateway Tests
#-------------------------------------------------------------
FROM ubuntu:16.04
COPY --from=0 /go/src/github.com/minio/minio/minio /usr/bin/minio
COPY buildscripts/gateway-tests.sh /usr/bin/gateway-tests.sh
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
ENV GOROOT /usr/local/go
ENV GOPATH /usr/local
ENV PATH $GOPATH/bin:$GOROOT/bin:$PATH
RUN apt-get --yes update && apt-get --yes upgrade && apt-get --yes --quiet install wget jq curl git dnsmasq && \
git clone https://github.com/minio/mint.git /mint && \
cd /mint && /mint/release.sh
WORKDIR /mint
RUN /usr/bin/gateway-tests.sh

View File

@@ -1,6 +1,6 @@
# For maintainers only
### Setup your minio Github Repository
### Setup your minio GitHub Repository
Fork [minio upstream](https://github.com/minio/minio/fork) source repository to your own personal repository.
```bash

177
Makefile
View File

@@ -1,145 +1,86 @@
LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
PWD := $(shell pwd)
GOPATH := $(shell go env GOPATH)
LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
TAG ?= $(USER)
BUILD_LDFLAGS := '$(LDFLAGS)'
TAG := latest
HOST ?= $(shell uname)
CPU ?= $(shell uname -m)
# if no host is identifed (no uname tool)
# we assume a Linux-64bit build
ifeq ($(HOST),)
HOST = Linux
endif
# identify CPU
ifeq ($(CPU), x86_64)
HOST := $(HOST)64
else
ifeq ($(CPU), amd64)
HOST := $(HOST)64
else
ifeq ($(CPU), i686)
HOST := $(HOST)32
endif
endif
endif
#############################################
# now we find out the target OS for
# which we are going to compile in case
# the caller didn't yet define OS himself
ifndef (OS)
ifeq ($(HOST), Linux64)
arch = gcc
else
ifeq ($(HOST), Linux32)
arch = 32
else
ifeq ($(HOST), Darwin64)
arch = clang
else
ifeq ($(HOST), Darwin32)
arch = clang
else
ifeq ($(HOST), FreeBSD64)
arch = gcc
endif
endif
endif
endif
endif
endif
all: install
all: build
checks:
@echo "Checking deps:"
@echo "Checking dependencies"
@(env bash $(PWD)/buildscripts/checkdeps.sh)
@(env bash $(PWD)/buildscripts/checkgopath.sh)
getdeps: checks
@go get -u github.com/golang/lint/golint && echo "Installed golint:"
@go get -u github.com/fzipp/gocyclo && echo "Installed gocyclo:"
@go get -u github.com/remyoudompheng/go-misc/deadcode && echo "Installed deadcode:"
@go get -u github.com/client9/misspell/cmd/misspell && echo "Installed misspell:"
@go get -u github.com/gordonklaus/ineffassign && echo "Installed ineffassign:"
getdeps:
@mkdir -p ${GOPATH}/bin
@which golint 1>/dev/null || (echo "Installing golint" && go get -u golang.org/x/lint/golint)
@which staticcheck 1>/dev/null || (echo "Installing staticcheck" && wget --quiet -O ${GOPATH}/bin/staticcheck https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_linux_amd64 && chmod +x ${GOPATH}/bin/staticcheck)
@which misspell 1>/dev/null || (echo "Installing misspell" && wget --quiet https://github.com/client9/misspell/releases/download/v0.3.4/misspell_0.3.4_linux_64bit.tar.gz && tar xf misspell_0.3.4_linux_64bit.tar.gz && mv misspell ${GOPATH}/bin/misspell && chmod +x ${GOPATH}/bin/misspell && rm -f misspell_0.3.4_linux_64bit.tar.gz)
verifiers: vet fmt lint cyclo spelling
crosscompile:
@(env bash $(PWD)/buildscripts/cross-compile.sh)
verifiers: getdeps vet fmt lint staticcheck spelling
vet:
@echo "Running $@:"
@GO15VENDOREXPERIMENT=1 go tool vet -all ./cmd
@GO15VENDOREXPERIMENT=1 go tool vet -all ./pkg
@GO15VENDOREXPERIMENT=1 go tool vet -shadow=true ./cmd
@GO15VENDOREXPERIMENT=1 go tool vet -shadow=true ./pkg
@echo "Running $@"
@GO111MODULE=on go vet github.com/minio/minio/...
fmt:
@echo "Running $@:"
@GO15VENDOREXPERIMENT=1 gofmt -s -l cmd
@GO15VENDOREXPERIMENT=1 gofmt -s -l pkg
@echo "Running $@"
@GO111MODULE=on gofmt -d cmd/
@GO111MODULE=on gofmt -d pkg/
lint:
@echo "Running $@:"
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/golint github.com/minio/minio/cmd...
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/golint github.com/minio/minio/pkg...
@echo "Running $@"
@GO111MODULE=on ${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/cmd/...
@GO111MODULE=on ${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/pkg/...
ineffassign:
@echo "Running $@:"
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/ineffassign .
cyclo:
@echo "Running $@:"
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/gocyclo -over 100 cmd
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/gocyclo -over 100 pkg
build: getdeps verifiers $(UI_ASSETS)
deadcode:
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/deadcode
staticcheck:
@echo "Running $@"
@GO111MODULE=on ${GOPATH}/bin/staticcheck github.com/minio/minio/cmd/...
@GO111MODULE=on ${GOPATH}/bin/staticcheck github.com/minio/minio/pkg/...
spelling:
@-GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/misspell -error `find cmd/`
@-GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/misspell -error `find pkg/`
@-GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/misspell -error `find docs/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find cmd/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find pkg/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find docs/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find buildscripts/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find dockerscripts/`
test: build
@echo "Running all minio testing:"
@GO15VENDOREXPERIMENT=1 go test $(GOFLAGS) github.com/minio/minio/cmd...
@GO15VENDOREXPERIMENT=1 go test $(GOFLAGS) github.com/minio/minio/pkg...
# Builds minio, runs the verifiers then runs the tests.
check: test
test: verifiers build
@echo "Running unit tests"
@GO111MODULE=on CGO_ENABLED=0 go test -tags kqueue ./... 1>/dev/null
verify: build
@echo "Verifying build"
@(env bash $(PWD)/buildscripts/verify-build.sh)
coverage: build
@echo "Running all coverage for minio:"
@GO15VENDOREXPERIMENT=1 ./buildscripts/go-coverage.sh
@echo "Running all coverage for minio"
@(env bash $(PWD)/buildscripts/go-coverage.sh)
gomake-all: build
@echo "Installing minio:"
@GO15VENDOREXPERIMENT=1 go build --ldflags $(BUILD_LDFLAGS) -o $(GOPATH)/bin/minio
# Builds minio locally.
build: checks
@echo "Building minio binary to './minio'"
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio 1>/dev/null
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go 1>/dev/null
pkg-add:
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/govendor add $(PKG)
docker: build
@docker build -t $(TAG) . -f Dockerfile.dev
pkg-update:
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/govendor update $(PKG)
pkg-remove:
@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/govendor remove $(PKG)
pkg-list:
@GO15VENDOREXPERIMENT=1 $(GOPATH)/bin/govendor list
install: gomake-all
release: verifiers
@MINIO_RELEASE=RELEASE ./buildscripts/build.sh
experimental: verifiers
@MINIO_RELEASE=EXPERIMENTAL ./buildscripts/build.sh
# Builds minio and installs it to $GOPATH/bin.
install: build
@echo "Installing minio binary to '$(GOPATH)/bin/minio'"
@mkdir -p $(GOPATH)/bin && cp -f $(PWD)/minio $(GOPATH)/bin/minio
@echo "Installation successful. To learn more, try \"minio --help\"."
clean:
@echo "Cleaning up all the generated files:"
@echo "Cleaning up all the generated files"
@find . -name '*.test' | xargs rm -fv
@rm -rf build
@rm -rf release
@find . -name '*~' | xargs rm -fv
@rm -rvf minio
@rm -rvf build
@rm -rvf release

8
NOTICE
View File

@@ -1,9 +1,9 @@
Minio Cloud Storage, (C) 2014,2015 Minio, Inc.
MinIO Cloud Storage, (C) 2014,2015 MinIO, Inc.
This product includes software developed at Minio, Inc.
(https://minio.io/).
This product includes software developed at MinIO, Inc.
(https://min.io/).
The Minio project contains unmodified/modified subcomponents too with
The MinIO project contains unmodified/modified subcomponents too with
separate copyright notices and license terms. Your use of the source
code for these subcomponents is subject to the terms and conditions
of the following licenses.

185
README.md
View File

@@ -1,99 +1,172 @@
# Minio Quickstart Guide [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio)
# MinIO Quickstart Guide
[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
Minio is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
Minio server is light enough to be bundled with the application stack, similar to NodeJS, Redis and MySQL.
MinIO server is light enough to be bundled with the application stack, similar to NodeJS, Redis and MySQL.
## Docker Container
### Stable
```sh
$ docker pull minio/minio
$ docker run -p 9000:9000 minio/minio server /export
```
docker pull minio/minio
docker run -p 9000:9000 minio/minio server /data
```
### Edge
```sh
$ docker pull minio/minio:edge
$ docker run -p 9000:9000 minio/minio:edge server /export
```
Please visit Minio Docker quickstart guide for more [here](https://docs.minio.io/docs/minio-docker-quickstart-guide)
docker pull minio/minio:edge
docker run -p 9000:9000 minio/minio:edge server /data
```
Note: Docker will not display the autogenerated keys unless you start the container with the `-it`(interactive TTY) argument. Generally, it is not recommended to use autogenerated keys with containers. Please visit MinIO Docker quickstart guide for more information [here](https://docs.min.io/docs/minio-docker-quickstart-guide)
## OS X
## macOS
### Homebrew
Install minio packages using [Homebrew](http://brew.sh/)
```sh
$ brew install minio
$ minio server ~/Photos
brew install minio/stable/minio
minio server /data
```
> NOTE: If you previously installed minio using `brew install minio` then it is recommended that you reinstall minio from `minio/stable/minio` official repo instead.
```sh
brew uninstall minio
brew install minio/stable/minio
```
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|Apple OS X|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio|
| Platform | Architecture | URL |
| ---------- | -------- | ------ |
| Apple macOS | 64-bit Intel | https://dl.min.io/server/minio/release/darwin-amd64/minio |
```sh
$ chmod 755 minio
$ ./minio server ~/Photos
chmod 755 minio
./minio server /data
```
## GNU/Linux
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio|
||32-bit Intel|https://dl.minio.io/server/minio/release/linux-386/minio|
||32-bit ARM|https://dl.minio.io/server/minio/release/linux-arm/minio|
| Platform | Architecture | URL |
| ---------- | -------- | ------ |
| GNU/Linux | 64-bit Intel | https://dl.min.io/server/minio/release/linux-amd64/minio |
```sh
$ chmod +x minio
$ ./minio server ~/Photos
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data
```
| Platform | Architecture | URL |
| ---------- | -------- | ------ |
| GNU/Linux | ppc64le | https://dl.min.io/server/minio/release/linux-ppc64le/minio |
```sh
wget https://dl.min.io/server/minio/release/linux-ppc64le/minio
chmod +x minio
./minio server /data
```
## Microsoft Windows
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|Microsoft Windows|64-bit|https://dl.minio.io/server/minio/release/windows-amd64/minio.exe|
||32-bit|https://dl.minio.io/server/minio/release/windows-386/minio.exe|
| Platform | Architecture | URL |
| ---------- | -------- | ------ |
| Microsoft Windows | 64-bit | https://dl.min.io/server/minio/release/windows-amd64/minio.exe |
```sh
C:\Users\Username\Downloads> minio.exe server D:\Photos
minio.exe server D:\Photos
```
## FreeBSD
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|FreeBSD|64-bit|https://dl.minio.io/server/minio/release/freebsd-amd64/minio|
### Port
Install minio packages using [pkg](https://github.com/freebsd/pkg)
```sh
$ chmod 755 minio
$ ./minio server ~/Photos
pkg install minio
sysrc minio_enable=yes
sysrc minio_disks=/home/user/Photos
service minio start
```
Please visit official zfs FreeBSD guide for more details [here](https://www.freebsd.org/doc/handbook/zfs-quickstart.html)
## Install from Source
Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow [How to install Golang](https://docs.minio.io/docs/how-to-install-golang).
Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow [How to install Golang](https://golang.org/doc/install). Minimum version required is [go1.12](https://golang.org/dl/#stable)
```sh
$ go get -u github.com/minio/minio
GO111MODULE=on go get github.com/minio/minio
```
## Test using Minio Browser
Minio Server comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 ensure your server has started successfully.
## Allow port access for Firewalls
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.jpg?raw=true)
By default MinIO uses the port 9000 to listen for incoming connections. If your platform blocks the port by default, you may need to enable access to the port.
## Test using Minio Client `mc`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the Minio Client [Quickstart Guide](https://docs.minio.io/docs/minio-client-quickstart-guide) for further instructions.
### iptables
For hosts with iptables enabled (RHEL, CentOS, etc), you can use `iptables` command to enable all traffic coming to specific ports. Use below command to allow
access to port 9000
```sh
iptables -A INPUT -p tcp --dport 9000 -j ACCEPT
service iptables restart
```
Below command enables all incoming traffic to ports ranging from 9000 to 9010.
```sh
iptables -A INPUT -p tcp --dport 9000:9010 -j ACCEPT
service iptables restart
```
### ufw
For hosts with ufw enabled (Debian based distros), you can use `ufw` command to allow traffic to specific ports. Use below command to allow access to port 9000
```sh
ufw allow 9000
```
Below command enables all incoming traffic to ports ranging from 9000 to 9010.
```sh
ufw allow 9000:9010/tcp
```
### firewall-cmd
For hosts with firewall-cmd enabled (CentOS), you can use `firewall-cmd` command to allow traffic to specific ports. Use below commands to allow access to port 9000
```sh
firewall-cmd --get-active-zones
```
This command gets the active zone(s). Now, apply port rules to the relevant zones returned above. For example if the zone is `public`, use
```sh
firewall-cmd --zone=public --add-port=9000/tcp --permanent
```
Note that `permanent` makes sure the rules are persistent across firewall start, restart or reload. Finally reload the firewall for changes to take effect.
```sh
firewall-cmd --reload
```
## Test using MinIO Browser
MinIO Server comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 ensure your server has started successfully.
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.png?raw=true)
## Test using MinIO Client `mc`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the MinIO Client [Quickstart Guide](https://docs.min.io/docs/minio-client-quickstart-guide) for further instructions.
## Pre-existing data
When deployed on a single drive, MinIO server lets clients access any pre-existing data in the data directory. For example, if MinIO is started with the command `minio server /mnt/data`, any pre-existing data in the `/mnt/data` directory would be accessible to the clients.
The above statement is also valid for all gateway backends.
## Explore Further
- [Minio Erasure Code QuickStart Guide](https://docs.minio.io/docs/minio-erasure-code-quickstart-guide)
- [Use `mc` with Minio Server](https://docs.minio.io/docs/minio-client-quickstart-guide)
- [Use `aws-cli` with Minio Server](https://docs.minio.io/docs/aws-cli-with-minio)
- [Use `s3cmd` with Minio Server](https://docs.minio.io/docs/s3cmd-with-minio)
- [Use `minio-go` SDK with Minio Server](https://docs.minio.io/docs/golang-client-quickstart-guide)
- [The Minio documentation website](https://docs.minio.io)
- [MinIO Erasure Code QuickStart Guide](https://docs.min.io/docs/minio-erasure-code-quickstart-guide)
- [Use `mc` with MinIO Server](https://docs.min.io/docs/minio-client-quickstart-guide)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/docs/aws-cli-with-minio)
- [Use `s3cmd` with MinIO Server](https://docs.min.io/docs/s3cmd-with-minio)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/docs/golang-client-quickstart-guide)
- [The MinIO documentation website](https://docs.min.io)
## Contribute to Minio Project
Please follow Minio [Contributor's Guide](https://github.com/minio/minio/blob/master/CONTRIBUTING.md)
## Contribute to MinIO Project
Please follow MinIO [Contributor's Guide](https://github.com/minio/minio/blob/master/CONTRIBUTING.md)
## License
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fminio%2Fminio.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fminio%2Fminio?ref=badge_large)

View File

@@ -1,200 +0,0 @@
# Minio 快速入门 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio)
Minio是一个对象存储服务基于Apache License v2.0协议. 它完全兼容亚马逊的S3云储存服务非常适合于存储很多非结构化的数据例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等而一个对象文件可以是任意大小从几kb到最大5T不等。
## 1. 下载
Minio是一个非常轻量的服务,可以很简单的和其他应用的结合,类似 NodeJS, Redis 或者 MySQL.
| Platform| Architecture | URL|
| ----------| -------- | ------|
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio|
||32-bit Intel|https://dl.minio.io/server/minio/release/linux-386/minio|
||32-bit ARM|https://dl.minio.io/server/minio/release/linux-arm/minio|
|Apple OS X|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio|
|Microsoft Windows|64-bit|https://dl.minio.io/server/minio/release/windows-amd64/minio.exe|
||32-bit|https://dl.minio.io/server/minio/release/windows-386/minio.exe|
|FreeBSD|64-bit|https://dl.minio.io/server/minio/release/freebsd-amd64/minio|
### Homebrew 安装
使用[Homebrew](http://brew.sh/) 来安装minio
```sh
$ brew install minio
$ minio --help
```
### 源码安装
源码安装只针对开发者和一些高级用户如果你还没有golang的环境请安装golang官网安装[How to install Golang](https://docs.minio.io/docs/zh-CN/how-to-install-golang).
```sh
$ go get -u github.com/minio/minio
```
## 2. 运行Minio服务
### GNU/Linux
```sh
$ chmod +x minio
$ ./minio --help
$ ./minio server ~/Photos
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
### OS X
```sh
$ chmod 755 minio
$ ./minio --help
$ ./minio server ~/Photos
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
### Microsoft Windows
```sh
C:\Users\Username\Downloads> minio.exe --help
C:\Users\Username\Downloads> minio.exe server D:\Photos
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc.exe config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
### Docker
```sh
$ docker pull minio/minio
$ docker run -p 9000:9000 minio/minio
```
访问minio的docker入门指南获得更多内容 [here](https://docs.minio.io/docs/zh-CN/minio-docker-quickstart-guide)
### FreeBSD
```sh
$ chmod 755 minio
$ ./minio --help
$ ./minio server ~/Photos
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
请访问FreeBSD的官网指南获取更多详细信息[here](https://www.freebsd.org/doc/handbook/zfs-quickstart.html)
## 3. 使用浏览器测试minio服务
打开浏览器并输入 http://127.0.0.1:9000 查看在minio服务器上面的所有bucket
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.jpg?raw=true)
## 4. 使用`mc`测试minio服务
按照 [这个](https://docs.minio.io/docs/minio-client-quickstart-guide) 安装mc. 使用 `mc ls` 命令显示所有在minio服务上面的bucket.
```sh
$ mc ls myminio/
[2015-08-05 08:13:22 IST] 0B andoria/
[2015-08-05 06:14:26 IST] 0B deflector/
[2015-08-05 08:13:11 IST] 0B ferenginar/
[2016-03-08 14:56:35 IST] 0B jarjarbing/
[2016-01-20 16:07:41 IST] 0B my.minio.io/
```
查看更多的例子请访问 [Minio Client Complete Guide](https://docs.minio.io/docs/zh-CN/minio-client-complete-guide).
## 5. 更多内容
- [Minio Erasure Code 快速入门](https://docs.minio.io/docs/zh-CN/minio-erasure-code-quickstart-guide)
- [Minio Docker 快速入门](https://docs.minio.io/docs/zh-CN/minio-docker-quickstart-guide)
- [使用`mc`测试 Minio Server](https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide)
- [使用 `aws-cli` 测试 Minio Server](https://docs.minio.io/docs/zh-CN/aws-cli-with-minio)
- [使用 `s3cmd` 测试 Minio Server](https://docs.minio.io/docs/zh-CN/s3cmd-with-minio)
- [使用 `minio-go` SDK ce's测试 Minio Server](https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide)
## 6. 给Minio项目贡献
请按照Minio [贡献者指导手册](https://github.com/minio/minio/blob/master/CONTRIBUTING.md)

106
README_zh_CN.md Normal file
View File

@@ -0,0 +1,106 @@
# MinIO Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口非常适合于存储大容量非结构化的数据例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等而一个对象文件可以是任意大小从几kb到最大5T不等。
MinIO是一个非常轻量的服务,可以很简单的和其他应用的结合,类似 NodeJS, Redis 或者 MySQL。
## Docker 容器
### 稳定版
```
docker pull minio/minio
docker run -p 9000:9000 minio/minio server /data
```
### 尝鲜版
```
docker pull minio/minio:edge
docker run -p 9000:9000 minio/minio:edge server /data
```
更多Docker部署信息请访问 [这里](https://docs.min.io/docs/minio-docker-quickstart-guide)
## macOS
### Homebrew
使用 [Homebrew](http://brew.sh/)安装minio
```sh
brew install minio/stable/minio
minio server /data
```
#### Note
如果你之前使用 `brew install minio`安装过minio, 可以用 `minio/stable/minio` 官方镜像进行重装. 由于golang 1.8的bug,homebrew版本不太稳定。
```
brew uninstall minio
brew install minio/stable/minio
```
### 下载二进制文件
| 操作系统 | CPU架构 | 地址 |
| ---------- | -------- | ------ |
| Apple macOS | 64-bit Intel | https://dl.min.io/server/minio/release/darwin-amd64/minio |
```sh
chmod 755 minio
./minio server /data
```
## GNU/Linux
### 下载二进制文件
| 操作系统 | CPU架构 | 地址 |
| ---------- | -------- | ------ |
| GNU/Linux | 64-bit Intel | https://dl.min.io/server/minio/release/linux-amd64/minio |
```sh
chmod +x minio
./minio server /data
```
## 微软Windows系统
### 下载二进制文件
| 操作系统 | CPU架构 | 地址 |
| ---------- | -------- | ------ |
| 微软Windows系统 | 64位 | https://dl.min.io/server/minio/release/windows-amd64/minio.exe |
```sh
minio.exe server D:\Photos
```
## FreeBSD
### Port
使用 [pkg](https://github.com/freebsd/pkg)进行安装。
```sh
pkg install minio
sysrc minio_enable=yes
sysrc minio_disks=/home/user/Photos
service minio start
```
## 使用源码安装
采用源码安装仅供开发人员和高级用户使用,如果你还没有Golang环境 请参考 [How to install Golang](https://docs.min.io/docs/how-to-install-golang).
```sh
go get -u github.com/minio/minio
```
## 使用MinIO浏览器进行验证
安装后使用浏览器访问[http://127.0.0.1:9000](http://127.0.0.1:9000)如果可以访问则表示minio已经安装成功。
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.png?raw=true)
## 使用MinIO客户端 `mc`进行验证
`mc` 提供了一些UNIX常用命令的替代品像ls, cat, cp, mirror, diff这些。 它支持文件系统和亚马逊S3云存储服务。 更多信息请参考 [mc快速入门](https://docs.min.io/docs/minio-client-quickstart-guide) 。
## 已经存在的数据
当在单块磁盘上部署MinIO server,MinIO server允许客户端访问数据目录下已经存在的数据。比如如果MinIO使用`minio server /mnt/data`启动,那么所有已经在`/mnt/data`目录下的数据都可以被客户端访问到。
上述描述对所有网关后端同样有效。
## 了解更多
- [MinIO纠删码入门](https://docs.min.io/docs/minio-erasure-code-quickstart-guide)
- [`mc`快速入门](https://docs.min.io/docs/minio-client-quickstart-guide)
- [使用 `aws-cli`](https://docs.min.io/docs/aws-cli-with-minio)
- [使用 `s3cmd`](https://docs.min.io/docs/s3cmd-with-minio)
- [使用 `minio-go` SDK](https://docs.min.io/docs/golang-client-quickstart-guide)
- [MinIO文档](https://docs.min.io)
## 如何参与到MinIO项目
请参考 [贡献者指南](https://github.com/minio/minio/blob/master/CONTRIBUTING.md)。欢迎各位中国程序员加到MinIO项目中。

View File

@@ -1,54 +0,0 @@
# version format
version: "{build}"
# Operating system (build VM template)
os: Windows Server 2012 R2
# Platform.
platform: x64
clone_folder: c:\gopath\src\github.com\minio\minio
# Environment variables
environment:
GOPATH: c:\gopath
GO15VENDOREXPERIMENT: 1
# scripts that run after cloning repository
install:
- set PATH=%GOPATH%\bin;c:\go\bin;%PATH%
- go version
- go env
- python --version
# To run your custom scripts instead of automatic MSBuild
build_script:
# Compile
- appveyor AddCompilationMessage "Starting Compile"
- cd c:\gopath\src\github.com\minio\minio
- go run buildscripts/gen-ldflags.go > temp.txt
- set /p BUILD_LDFLAGS=<temp.txt
- go build -ldflags="%BUILD_LDFLAGS%" -o %GOPATH%\bin\minio.exe
- appveyor AddCompilationMessage "Compile Success"
# To run your custom scripts instead of automatic tests
test_script:
# Unit tests
- ps: Add-AppveyorTest "Unit Tests" -Outcome Running
- mkdir build\coverage
- go test -race github.com/minio/minio/cmd...
- go test -race github.com/minio/minio/pkg...
- go test -coverprofile=build\coverage\coverage.txt -covermode=atomic github.com/minio/minio/cmd
- ps: Update-AppveyorTest "Unit Tests" -Outcome Passed
after_test:
- go tool cover -html=build\coverage\coverage.txt -o build\coverage\coverage.html
- ps: Push-AppveyorArtifact build\coverage\coverage.txt
- ps: Push-AppveyorArtifact build\coverage\coverage.html
# Upload coverage report.
- "SET PATH=C:\\Python34;C:\\Python34\\Scripts;%PATH%"
- pip install codecov
- codecov -X gcov -f "build\coverage\coverage.txt"
# to disable deployment
deploy: off

9
browser/.babelrc Normal file
View File

@@ -0,0 +1,9 @@
{
"presets": [
"es2015",
"react"
],
"plugins": [
"transform-object-rest-spread"
]
}

16
browser/.editorconfig Normal file
View File

@@ -0,0 +1,16 @@
# editorconfig.org
root = true
[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[*.json]
indent_size = 2
[*.md]
trim_trailing_whitespace = false

23
browser/.esformatter Normal file
View File

@@ -0,0 +1,23 @@
{
"plugins": [
"esformatter-jsx"
],
// Copied from https://github.com/royriojas/esformatter-jsx
"jsx": {
"formatJSX": true, //Duh! that's the default
"attrsOnSameLineAsTag": false, // move each attribute to its own line
"maxAttrsOnTag": 3, // if lower or equal than 3 attributes, they will be kept on a single line
"firstAttributeOnSameLine": true, // keep the first attribute in the same line as the tag
"formatJSXExpressions": true, // default true, if false jsxExpressions won't be recursively formatted
"JSXExpressionsSingleLine": true, // default true, if false the JSXExpressions might span several lines
"alignWithFirstAttribute": false, // do not align attributes with the first tag
"spaceInJSXExpressionContainers": " ", // default to one space. Make it empty if you don't like spaces between JSXExpressionContainers
"removeSpaceBeforeClosingJSX": false, // default false. if true <React.Something /> => <React.Something/>
"closingTagOnNewLine": false, // default false. if true attributes on multiple lines will close the tag on a new line
"JSXAttributeQuotes": "", // possible values "single" or "double". Leave it as empty string if you don't want to modify the attributes' quotes
"htmlOptions": {
// put here the options for js-beautify.html
}
}
}

20
browser/.gitignore vendored Normal file
View File

@@ -0,0 +1,20 @@
**/*.swp
cover.out
*~
minio
!*/
site/
**/*.test
**/*.sublime-workspace
/.idea/
/Minio.iml
**/access.log
build
vendor/**/*.js
vendor/**/*.json
release
.DS_Store
*.syso
coverage.txt
node_modules
production

3
browser/.prettierrc Normal file
View File

@@ -0,0 +1,3 @@
{
"semi": false
}

72
browser/README.md Normal file
View File

@@ -0,0 +1,72 @@
# MinIO File Browser
``MinIO Browser`` provides minimal set of UI to manage buckets and objects on ``minio`` server. ``MinIO Browser`` is written in javascript and released under [Apache 2.0 License](./LICENSE).
## Installation
### Install yarn
```sh
curl -o- -L https://yarnpkg.com/install.sh | bash
yarn
```
### Install `go-bindata` and `go-bindata-assetfs`
If you do not have a working Golang environment, please follow [Install Golang](https://docs.min.io/docs/how-to-install-golang)
```sh
go get github.com/jteeuwen/go-bindata/...
go get github.com/elazarl/go-bindata-assetfs/...
```
## Generating Assets
### Generate ui-assets.go
```sh
yarn release
```
This generates ui-assets.go in the current directory. Now do `make` in the parent directory to build the minio binary with the newly generated ``ui-assets.go``
### Run MinIO Browser with live reload
```sh
yarn dev
```
Open [http://localhost:8080/minio/](http://localhost:8080/minio/) in your browser to play with the application
### Run MinIO Browser with live reload on custom port
Edit `browser/webpack.config.js`
```diff
diff --git a/browser/webpack.config.js b/browser/webpack.config.js
index 3ccdaba..9496c56 100644
--- a/browser/webpack.config.js
+++ b/browser/webpack.config.js
@@ -58,6 +58,7 @@ var exports = {
historyApiFallback: {
index: '/minio/'
},
+ port: 8888,
proxy: {
'/minio/webrpc': {
target: 'http://localhost:9000',
@@ -97,7 +98,7 @@ var exports = {
if (process.env.NODE_ENV === 'dev') {
exports.entry = [
'webpack/hot/dev-server',
- 'webpack-dev-server/client?http://localhost:8080',
+ 'webpack-dev-server/client?http://localhost:8888',
path.resolve(__dirname, 'app/index.js')
]
}
```
```sh
yarn dev
```
Open [http://localhost:8888/minio/](http://localhost:8888/minio/) in your browser to play with the application

View File

@@ -0,0 +1,98 @@
.page-load {
position: fixed;
width: 100%;
height: 100%;
top: 0;
left: 0;
background: #002a37;
z-index: 100;
transition: opacity 200ms;
-webkit-transition: opacity 200ms;
}
.pl-0{
opacity: 0;
}
.pl-1 {
display: none;
}
.pl-inner {
position: absolute;
width: 100px;
height: 100px;
left: 50%;
margin-left: -50px;
top: 50%;
margin-top: -50px;
text-align: center;
-webkit-animation: fade-in 500ms;
animation: fade-in 500ms;
-webkit-animation-fill-mode: both;
animation-fill-mode: both;
animation-delay: 350ms;
-webkit-animation-delay: 350ms;
-webkit-backface-visibility: visible;
backface-visibility: visible;
}
.pl-inner:before {
content: '';
position: absolute;
width: 100%;
height: 100%;
left: 0;
top: 0;
display: block;
-webkit-animation: spin 1000ms infinite linear;
animation: spin 1000ms infinite linear;
border: 1px solid rgba(255, 255, 255, 0.2);;
border-left-color: #fff;
border-radius: 50%;
}
.pl-inner > img {
width: 30px;
margin-top: 21px;
}
@-webkit-keyframes fade-in {
0% {
opacity: 0;
}
100% {
opacity: 1;
}
}
@keyframes fade-in {
0% {
opacity: 0;
}
100% {
opacity: 1;
}
}
@-webkit-keyframes spin {
0% {
-webkit-transform: rotate(0deg);
transform: rotate(0deg);
}
100% {
-webkit-transform: rotate(360deg);
transform: rotate(360deg);
}
}
@keyframes spin {
0% {
-webkit-transform: rotate(0deg);
transform: rotate(0deg);
}
100% {
-webkit-transform: rotate(360deg);
transform: rotate(360deg);
}
}

Binary file not shown.

Binary file not shown.

View File

@@ -0,0 +1,3 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" preserveAspectRatio="xMidYMid meet" viewBox="139.0389584668397 284.78404581828653 12.617622649141168 6.417622649141265"><defs><path d="M139.04 290.7L144.95 284.78L145.46 285.29L139.54 291.2L139.04 290.7Z" id="NsdmgIWbGe"></path><path d="M145.24 285.29L151.15 291.2L151.66 290.7L145.74 284.78L145.24 285.29Z" id="VqPWmhvQEo"></path></defs><g visibility="inherit"><g><use xlink:href="#NsdmgIWbGe" opacity="1" fill="#000000" fill-opacity="1"></use></g><g><use xlink:href="#VqPWmhvQEo" opacity="1" fill="#000000" fill-opacity="1"></use></g></g></svg>

After

Width:  |  Height:  |  Size: 797 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

View File

@@ -0,0 +1,50 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="469px" height="60px" viewBox="0 0 469 60" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 48.2 (47327) - http://www.bohemiancoding.com/sketch -->
<title>Untitled</title>
<desc>Created with Sketch.</desc>
<defs></defs>
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<path d="M27.9171869,0.94921875 L5.01068171,0.94921875 C2.24925796,0.94921875 0.0106817104,3.187795 0.0106817104,5.94921875 L0.0106817104,54.9836742 C0.0106817104,57.7450979 2.24925796,59.9836742 5.01068171,59.9836742 L43.0384854,59.9836742 C45.7999092,59.9836742 48.0384854,57.7450979 48.0384854,54.9836742 L48.0384854,21.4784647 C48.0384854,20.1754261 47.5298023,18.9238644 46.6207587,17.9902963 L31.4994602,2.46105043 C30.558243,1.49444073 29.2663438,0.94921875 27.9171869,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M87.9171869,0.94921875 L65.0106817,0.94921875 C62.249258,0.94921875 60.0106817,3.187795 60.0106817,5.94921875 L60.0106817,54.9836742 C60.0106817,57.7450979 62.249258,59.9836742 65.0106817,59.9836742 L103.038485,59.9836742 C105.799909,59.9836742 108.038485,57.7450979 108.038485,54.9836742 L108.038485,21.4784647 C108.038485,20.1754261 107.529802,18.9238644 106.620759,17.9902963 L91.4994602,2.46105043 C90.558243,1.49444073 89.2663438,0.94921875 87.9171869,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M147.917187,0.94921875 L125.010682,0.94921875 C122.249258,0.94921875 120.010682,3.187795 120.010682,5.94921875 L120.010682,54.9836742 C120.010682,57.7450979 122.249258,59.9836742 125.010682,59.9836742 L163.038485,59.9836742 C165.799909,59.9836742 168.038485,57.7450979 168.038485,54.9836742 L168.038485,21.4784647 C168.038485,20.1754261 167.529802,18.9238644 166.620759,17.9902963 L151.49946,2.46105043 C150.558243,1.49444073 149.266344,0.94921875 147.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M207.917187,0.94921875 L185.010682,0.94921875 C182.249258,0.94921875 180.010682,3.187795 180.010682,5.94921875 L180.010682,54.9836742 C180.010682,57.7450979 182.249258,59.9836742 185.010682,59.9836742 L223.038485,59.9836742 C225.799909,59.9836742 228.038485,57.7450979 228.038485,54.9836742 L228.038485,21.4784647 C228.038485,20.1754261 227.529802,18.9238644 226.620759,17.9902963 L211.49946,2.46105043 C210.558243,1.49444073 209.266344,0.94921875 207.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M267.917187,0.94921875 L245.010682,0.94921875 C242.249258,0.94921875 240.010682,3.187795 240.010682,5.94921875 L240.010682,54.9836742 C240.010682,57.7450979 242.249258,59.9836742 245.010682,59.9836742 L283.038485,59.9836742 C285.799909,59.9836742 288.038485,57.7450979 288.038485,54.9836742 L288.038485,21.4784647 C288.038485,20.1754261 287.529802,18.9238644 286.620759,17.9902963 L271.49946,2.46105043 C270.558243,1.49444073 269.266344,0.94921875 267.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M327.917187,0.94921875 L305.010682,0.94921875 C302.249258,0.94921875 300.010682,3.187795 300.010682,5.94921875 L300.010682,54.9836742 C300.010682,57.7450979 302.249258,59.9836742 305.010682,59.9836742 L343.038485,59.9836742 C345.799909,59.9836742 348.038485,57.7450979 348.038485,54.9836742 L348.038485,21.4784647 C348.038485,20.1754261 347.529802,18.9238644 346.620759,17.9902963 L331.49946,2.46105043 C330.558243,1.49444073 329.266344,0.94921875 327.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M387.917187,0.94921875 L365.010682,0.94921875 C362.249258,0.94921875 360.010682,3.187795 360.010682,5.94921875 L360.010682,54.9836742 C360.010682,57.7450979 362.249258,59.9836742 365.010682,59.9836742 L403.038485,59.9836742 C405.799909,59.9836742 408.038485,57.7450979 408.038485,54.9836742 L408.038485,21.4784647 C408.038485,20.1754261 407.529802,18.9238644 406.620759,17.9902963 L391.49946,2.46105043 C390.558243,1.49444073 389.266344,0.94921875 387.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M447.917187,0.94921875 L425.010682,0.94921875 C422.249258,0.94921875 420.010682,3.187795 420.010682,5.94921875 L420.010682,54.9836742 C420.010682,57.7450979 422.249258,59.9836742 425.010682,59.9836742 L463.038485,59.9836742 C465.799909,59.9836742 468.038485,57.7450979 468.038485,54.9836742 L468.038485,21.4784647 C468.038485,20.1754261 467.529802,18.9238644 466.620759,17.9902963 L451.49946,2.46105043 C450.558243,1.49444073 449.266344,0.94921875 447.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<g id="excel" transform="translate(434.000000, 28.000000)" fill="#617A8B" fill-rule="nonzero">
<rect id="Rectangle-2" x="0" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="0" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="0" y="10.3086611" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="10.3086611" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="10.3086611" width="5" height="3.09259835"></rect>
</g>
<g id="folder" transform="translate(14.000000, 25.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M7.99999996,0 L2,0 C0.900000087,0 0.00999998996,0.900000093 0.00999998996,2.00000001 L0,14 C0,15.1000003 0.900000087,16 2,16 L18,16 C19.1000002,16 20,15.1000003 20,14 L20,4.00000001 C20,2.90000009 19.1000002,2.00000001 18,2.00000001 L9.99999996,2.00000001 L7.99999996,0 Z" id="Shape"></path>
</g>
<g id="image" transform="translate(72.000000, 24.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M13.590909,-3.76363638e-07 L9.6704545,5.22727238 L12.6500002,9.19999976 L10.9772726,10.4545451 C9.21045494,8.10227239 6.27272722,4.18181782 6.27272722,4.18181782 L-6.27273216e-08,12.5454541 L22.9999999,12.5454541 L13.590909,-3.76363638e-07 Z" id="Shape"></path>
</g>
<g id="pdf" transform="translate(135.000000, 22.500000)" fill="#617A8B" fill-rule="nonzero">
<path d="M12.9434813,11.7078193 C11.5243518,10.2886898 9.92783089,8.15999533 8.50870126,6.2086921 C8.86348367,4.61217127 9.04087488,3.37043286 9.04087488,2.66086804 C9.04087488,-0.886956014 4.78348601,-0.886956014 4.78348601,2.66086804 C4.78348601,3.37043286 5.49305082,4.61217127 6.73478924,6.5634745 C6.20261563,9.04695135 5.49305082,12.0626018 4.6060948,14.5460787 C2.83218278,15.2556434 1.59044436,16.1425995 0.880879547,16.6747731 C-0.00607646688,17.5617291 -0.183467669,18.6260763 0.171314736,19.5130323 C0.526097141,20.3999883 1.41305316,20.932162 2.30000917,20.932162 C2.83218278,20.932162 3.36435639,20.7547707 3.71913879,20.3999883 C3.89652999,20.2225971 4.6060948,19.5130323 5.84783322,15.6104258 C8.15391886,14.7234698 10.6373957,13.8365138 12.4113077,13.3043402 C14.0078286,14.7234698 15.249567,15.4330347 16.4913054,15.4330347 C17.7330438,15.4330347 18.6199998,14.5460787 18.6199998,13.4817314 C18.6199998,12.7721666 18.2652175,12.239993 17.5556526,11.7078193 C16.8460878,11.353037 15.9591318,11.1756458 14.7173934,11.1756458 C14.362611,11.353037 13.8304373,11.353037 12.9434813,11.7078193 Z M2.47740038,19.1582499 C2.30000917,19.3356411 2.30000917,19.3356411 2.30000917,19.3356411 C2.12261797,19.3356411 1.94522676,19.1582499 1.76783556,18.8034675 C1.59044436,18.6260763 1.59044436,18.0939027 2.12261797,17.7391203 C2.30000917,17.5617291 2.83218278,17.2069466 3.89652999,16.6747731 C3.18696518,18.2712939 2.65479157,18.9808587 2.47740038,19.1582499 Z M6.38000684,2.48347683 C6.38000684,1.77391203 6.73478924,1.41912963 6.73478924,1.41912963 C6.73478924,1.41912963 7.08957165,1.77391203 7.08957165,2.48347683 C7.08957165,2.83825924 7.08957165,3.37043286 6.91218044,4.07999767 C6.55739803,3.19304165 6.38000684,2.66086804 6.38000684,2.48347683 Z M6.55739803,13.6591226 C7.08957165,11.8852106 7.62174525,10.1112986 7.97652766,8.33738654 C9.04087488,9.75651615 10.1052221,10.9982546 11.1695693,12.0626018 C9.75043969,12.5947754 8.15391886,12.9495579 6.55739803,13.6591226 Z M17.023479,13.6591226 C17.023479,13.8365138 16.8460878,14.013905 16.4913054,14.013905 C16.3139142,14.013905 15.6043494,13.8365138 14.5400021,12.9495579 C14.7173934,12.9495579 14.7173934,12.9495579 14.8947846,12.9495579 C15.9591318,12.9495579 16.6686966,13.126949 16.8460878,13.3043402 C16.8460878,13.4817314 17.023479,13.6591226 17.023479,13.6591226 Z" id="Shape"></path>
</g>
<g id="video" transform="translate(196.000000, 24.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M13.1249998,-1.68750002e-07 L13.1249998,1.87499983 L11.2499998,1.87499983 L11.2499998,-1.68750002e-07 L3.74999978,-1.68750002e-07 L3.74999978,1.87499983 L1.87499978,1.87499983 L1.87499978,-1.68750002e-07 L-2.25000019e-07,-1.68750002e-07 L-2.25000019e-07,16.8749998 L1.87499978,16.8749998 L1.87499978,14.9999998 L3.74999978,14.9999998 L3.74999978,16.8749998 L11.2499998,16.8749998 L11.2499998,14.9999998 L13.1249998,14.9999998 L13.1249998,16.8749998 L14.9999998,16.8749998 L14.9999998,-1.68750002e-07 L13.1249998,-1.68750002e-07 Z M3.74999955,13.1249991 L1.87499967,13.1249991 L1.87499967,11.2499992 L3.74999955,11.2499992 L3.74999955,13.1249991 Z M3.74999955,9.37499927 L1.87499967,9.37499927 L1.87499967,7.49999938 L3.74999955,7.49999938 L3.74999955,9.37499927 Z M3.74999955,5.62499949 L1.87499967,5.62499949 L1.87499967,3.74999961 L3.74999955,3.74999961 L3.74999955,5.62499949 Z M13.124999,13.1249991 L11.2499991,13.1249991 L11.2499991,11.2499992 L13.124999,11.2499992 L13.124999,13.1249991 Z M13.124999,9.37499927 L11.2499991,9.37499927 L11.2499991,7.49999938 L13.124999,7.49999938 L13.124999,9.37499927 Z M13.124999,5.62499949 L11.2499991,5.62499949 L11.2499991,3.74999961 L13.124999,3.74999961 L13.124999,5.62499949 Z" id="Shape"></path>
</g>
<g id="audio" transform="translate(253.000000, 25.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M16.6899956,0.302885892 C16.4835716,0.100936704 16.2326065,0 15.9375267,0 C15.8194481,0 15.7159842,0.0145115457 15.6276386,0.0433073022 L6.41930364,2.81258763 C6.19803256,2.88469069 6.01719345,3.01271813 5.87698015,3.1965942 C5.73684439,3.38058392 5.66671835,3.58427601 5.66671835,3.80789779 L5.66671835,14.2686004 C5.02469108,13.9872357 4.31641429,13.8465532 3.54169413,13.8465532 C3.20970997,13.8465532 2.8536527,13.8844045 2.47371613,13.9599933 C2.09358575,14.0355821 1.71171093,14.1509926 1.32813045,14.3060728 C0.944356146,14.4611532 0.627141486,14.6793946 0.376215114,14.9604564 C0.125366273,15.2417075 0,15.5628561 0,15.9233335 C0,16.2840761 0.125366273,16.6048457 0.376215114,16.8862105 C0.627141486,17.1673102 0.944317382,17.3855517 1.32813045,17.540594 C1.71186599,17.6956744 2.09374081,17.810971 2.47371613,17.8866736 C2.8536527,17.9622625 3.20970997,18 3.54169413,18 C3.87367828,18 4.22961926,17.9622625 4.60967212,17.8866736 C4.9898025,17.810971 5.37152225,17.6956744 5.7552578,17.540594 C6.13887704,17.3855137 6.4560917,17.1672722 6.70694055,16.8862105 C6.95802198,16.6048457 7.08334949,16.2840381 7.08334949,15.9233335 L7.08334949,8.25381104 L15.5834076,5.69004149 L15.5834076,11.4990929 C14.9412253,11.2178795 14.2330648,11.0770455 13.4583446,11.0770455 C13.1263217,11.0770455 12.7702257,11.1147831 12.3902891,11.1904856 C12.0101975,11.2660745 11.6285553,11.3814848 11.2447422,11.5365651 C10.861123,11.6916454 10.543792,11.909887 10.2927493,12.1912139 C10.0420168,12.4722 9.91626286,12.7932726 9.91626286,13.1538259 C9.91626286,13.5145685 10.0420168,13.8353381 10.2927493,14.1167028 C10.543792,14.3979541 10.8610842,14.6161956 11.2447422,14.7710864 C11.6284002,14.9261668 12.0101975,15.041577 12.3902891,15.1173175 C12.7702257,15.1930201 13.126283,15.230606 13.4583446,15.230606 C13.7902513,15.230606 14.1463085,15.1930201 14.5262451,15.1173175 C14.9062204,15.041577 15.2879788,14.9261668 15.6719083,14.7710864 C16.0555662,14.6161576 16.3726259,14.3979161 16.6236297,14.1167028 C16.8747888,13.8353381 17,13.5145305 17,13.1538259 L17,1.03820069 C16.9998837,0.749788454 16.8969236,0.504721411 16.6899956,0.302885892 Z" id="Shape"></path>
</g>
<g id="code" transform="translate(313.000000, 27.000000)" fill="#617A8B" fill-rule="nonzero">
<polygon id="Shape" transform="translate(17.091428, 6.857142) scale(-1, 1) translate(-17.091428, -6.857142) " points="21.3257133 1.61142788 19.7142851 -4.11428573e-07 12.8571425 6.85714204 19.7142851 13.7142845 21.3257133 12.1028561 16.091428 6.85714204"></polygon>
<polygon id="Shape" points="8.46857015 1.61142788 6.8571419 -4.11428573e-07 -5.48571475e-07 6.85714204 6.8571419 13.7142845 8.46857015 12.1028561 3.23428485 6.85714204"></polygon>
</g>
<g id="presentation" transform="translate(374.464844, 23.500000)" fill="#617A8B" fill-rule="nonzero">
<path d="M8.16304337,0.383534587 L8.16304337,17.4406771 C3.83905758,17.0142486 0.48732923,13.3554907 0.48732923,8.91210585 C0.48732923,4.468721 3.83905799,0.809963151 8.16304337,0.383534587 Z M9.89434308,0.383534587 L9.89434308,8.05071991 L17.5444718,8.05071991 C17.1436294,4.00817738 13.9283578,0.784376849 9.89434308,0.383534587 Z M9.89434308,9.77349173 L9.89434308,17.4406771 C13.9368865,17.0398348 17.1436278,13.8160343 17.5444718,9.77349173 L9.89434308,9.77349173 Z" id="Shape"></path>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 14 KiB

12
browser/app/img/logo.svg Normal file
View File

@@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="93px" height="187px" viewBox="0 0 93 187" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 48.2 (47327) - http://www.bohemiancoding.com/sketch -->
<title>logo</title>
<desc>Created with Sketch.</desc>
<defs></defs>
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g id="logo" transform="translate(0.187500, -0.683594)" fill="#FFFFFF" fill-rule="nonzero">
<path d="M91.49,46.551 C86.7827023,38.7699609 82.062696,30.9966172 77.33,23.231 C74.87,19.231 72.33,15.231 69.88,11.231 C69.57,10.731 69.18,10.291 68.88,9.831 C64.35,2.931 55.44,-1.679 46.73,2.701 C42.9729806,4.51194908 40.0995718,7.75449451 38.7536428,11.7020516 C37.4077139,15.6496086 37.701799,19.9721186 39.57,23.701 C41.08,26.641 43.57,29.121 45.91,31.581 C53.03,39.141 60.38,46.491 67.45,54.111 C72.4175495,59.4492221 74.4526451,66.8835066 72.8965704,74.0075359 C71.3404956,81.1315653 66.390952,87.0402215 59.65,89.821 C59.4938176,89.83842 59.3361824,89.83842 59.18,89.821 L59.18,54.591 C46.6388051,61.0478363 35.3944735,69.759905 26.01,80.291 C11.32,96.671 2.64,117.141 0.01,132.071 L23.96,119.821 C31.96,115.771 39.86,111.821 48.14,107.581 L48.14,175.921 L59.14,187.131 L59.14,101.831 C59.14,101.831 59.39,101.711 60.22,101.261 C63.5480598,99.6738911 66.7772674,97.8873078 69.89,95.911 C77.7130888,90.4306687 82.7479457,81.8029342 83.6709542,72.295947 C84.5939627,62.7889599 81.3127806,53.3538429 74.69,46.471 C66.49,37.891 58.24,29.351 50.05,20.761 C47.67,18.261 47.72,15.101 50.05,12.881 C52.38,10.661 55.56,10.881 57.96,13.331 L61.38,16.781 C64.1,19.681 66.79,22.611 69.53,25.481 C76.4547149,32.7389629 83.3947303,39.9823123 90.35,47.211 C90.7,47.571 91.12,47.871 91.5,48.211 L91.93,47.951 C91.8351945,47.4695902 91.6876376,47.0000911 91.49,46.551 Z M48.11,94.931 C47.9883217,95.5022568 47.6230065,95.9917791 47.11,96.271 C42.72,98.601 38.29,100.871 33.87,103.141 L17.76,111.401 C24.771203,96.7435071 35.1132853,83.9289138 47.96,73.981 C48.08,74.221 48.16,74.301 48.16,74.381 C48.15,81.231 48.17,88.081 48.11,94.931 Z" id="Shape"></path>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.2 KiB

View File

@@ -0,0 +1,3 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" preserveAspectRatio="xMidYMid meet" width="16" height="4" viewBox="4 10 16 4"><defs><path d="M4 12C4 13.1 4.9 14 6 14C7.1 14 8 13.1 8 12C8 10.9 7.1 10 6 10C4.9 10 4 10.9 4 12ZM16 12C16 13.1 16.9 14 18 14C19.1 14 20 13.1 20 12C20 10.9 19.1 10 18 10C16.9 10 16 10.9 16 12ZM10 12C10 13.1 10.9 14 12 14C13.1 14 14 13.1 14 12C14 10.9 13.1 10 12 10C10.9 10 10 10.9 10 12Z" id="mccsKZxKL3"></path></defs><g visibility="visible"><g><use xlink:href="#mccsKZxKL3" opacity="1" fill="#eaeaea" fill-opacity="1"></use><g><use xlink:href="#mccsKZxKL3" opacity="1" fill-opacity="0" stroke="#000000" stroke-width="1" stroke-opacity="0"></use></g></g></g></svg>

After

Width:  |  Height:  |  Size: 894 B

View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24"><path d="M6 10c-1.1 0-2 0.9-2 2s0.9 2 2 2 2-0.9 2-2-0.9-2-2-2zm12 0c-1.1 0-2 0.9-2 2s0.9 2 2 2 2-0.9 2-2-0.9-2-2-2zm-6 0c-1.1 0-2 0.9-2 2s0.9 2 2 2 2-0.9 2-2-0.9-2-2-2z"/></svg>

After

Width:  |  Height:  |  Size: 261 B

View File

@@ -0,0 +1,3 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" preserveAspectRatio="xMidYMid meet" width="9" height="9" viewBox="326.76441742035513 536.0133077721175 13 13"><defs><path d="M339.76 536.01L326.76 549.01L339.76 549.01L339.76 536.01Z" id="kt3PSf43ua"></path></defs><g visibility="visible"><g><use xlink:href="#kt3PSf43ua" opacity="1" fill="#dadada" fill-opacity="1"></use></g></g></svg>

After

Width:  |  Height:  |  Size: 586 B

59
browser/app/index.html Normal file
View File

@@ -0,0 +1,59 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>MinIO Browser</title>
<link rel="icon" type="image/png" sizes="32x32" href="/minio/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="/minio/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="/minio/favicon-16x16.png">
<link rel="stylesheet" href="/minio/loader.css" type="text/css">
</head>
<body>
<div class="page-load">
<div class="pl-inner">
<img src="/minio/logo.svg" alt="">
</div>
</div>
<div id="root"></div>
<!--[if lt IE 11]>
<div class="ie-warning">
<div class="iw-inner">
<i class="iwi-icon fa fa-warning"></i>
You are using Internet Explorer version 12.0 or lower. Due to security issues and lack of support for Web Standards it is highly recommended that you upgrade to a modern browser
<ul>
<li>
<a href="http://www.google.com/chrome/">
<img src="chrome.png" alt="">
<div>Chrome</div>
</a>
</li>
<li>
<a href="https://www.mozilla.org/en-US/firefox/new/">
<img src="firefox.png" alt="">
<div>Firefox</div>
</a>
</li>
<li>
<a href="https://www.apple.com/safari/">
<img src="safari.png" alt="">
<div>Safari</div>
</a>
</li>
</ul>
<div class="iwi-skip">Skip & Continue</div>
</div>
</div>
<![endif]-->
<script>currentUiVersion = 'MINIO_UI_VERSION'</script>
<script src="/minio/index_bundle.js"></script>
</body>
</html>

43
browser/app/index.js Normal file
View File

@@ -0,0 +1,43 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import "babel-polyfill"
import "./less/main.less"
import "font-awesome/css/font-awesome.css"
import "material-design-iconic-font/dist/css/material-design-iconic-font.min.css"
import React from "react"
import ReactDOM from "react-dom"
import { Router, Route } from "react-router-dom"
import { Provider } from "react-redux"
import history from "./js/history"
import configureStore from "./js/store/configure-store"
import hideLoader from "./js/loader"
import App from "./js/App"
const store = configureStore()
ReactDOM.render(
<Provider store={store}>
<Router history={history}>
<App />
</Router>
</Provider>,
document.getElementById("root")
)
hideLoader()

32
browser/app/js/App.js Normal file
View File

@@ -0,0 +1,32 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { Route, Switch, Redirect } from "react-router-dom"
import Browser from "./browser/Browser"
import Login from "./browser/Login"
import web from "./web"
export const App = () => {
return (
<Switch>
<Route path={"/login"} component={Login} />
<Route path={"/:bucket?/*"} component={Browser} />
</Switch>
)
}
export default App

View File

@@ -0,0 +1,65 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { MemoryRouter } from "react-router-dom"
import App from "../App"
jest.mock("../browser/Login", () => () => <div>Login</div>)
jest.mock("../browser/Browser", () => () => <div>Browser</div>)
describe("App", () => {
it("should render without crashing", () => {
shallow(<App />)
})
it("should render Login component for '/login' route", () => {
const wrapper = mount(
<MemoryRouter initialEntries={["/login"]}>
<App />
</MemoryRouter>
)
expect(wrapper.text()).toBe("Login")
})
it("should render Browser component for '/' route", () => {
const wrapper = mount(
<MemoryRouter initialEntries={["/"]}>
<App />
</MemoryRouter>
)
expect(wrapper.text()).toBe("Browser")
})
it("should render Browser component for '/bucket' route", () => {
const wrapper = mount(
<MemoryRouter initialEntries={["/bucket"]}>
<App />
</MemoryRouter>
)
expect(wrapper.text()).toBe("Browser")
})
it("should render Browser component for '/bucket/a/b/c' route", () => {
const wrapper = mount(
<MemoryRouter initialEntries={["/bucket/a/b/c"]}>
<App />
</MemoryRouter>
)
expect(wrapper.text()).toBe("Browser")
})
})

View File

@@ -0,0 +1,41 @@
/*
* MinIO Cloud Storage (C) 2016 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import JSONrpc from "../jsonrpc"
describe("jsonrpc", () => {
it("should fail with invalid endpoint", done => {
try {
let jsonRPC = new JSONrpc({
endpoint: "htt://localhost:9000",
namespace: "Test"
})
} catch (e) {
done()
}
})
it("should succeed with valid endpoint", () => {
let jsonRPC = new JSONrpc({
endpoint: "http://localhost:9000/webrpc",
namespace: "Test"
})
expect(jsonRPC.version).toEqual("2.0")
expect(jsonRPC.host).toEqual("localhost")
expect(jsonRPC.port).toEqual("9000")
expect(jsonRPC.path).toEqual("/webrpc")
expect(jsonRPC.scheme).toEqual("http")
})
})

View File

@@ -0,0 +1,30 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import AlertComponent from "react-bootstrap/lib/Alert"
const Alert = ({ show, type, message, onDismiss }) => (
<AlertComponent
className={"alert animated " + (show ? "fadeInDown" : "fadeOutUp")}
bsStyle={type}
onDismiss={onDismiss}
>
<div className="text-center">{message}</div>
</AlertComponent>
)
export default Alert

View File

@@ -0,0 +1,41 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import Alert from "./Alert"
import * as alertActions from "./actions"
export const AlertContainer = ({ alert, clearAlert }) => {
if (!alert.message) {
return ""
}
return <Alert {...alert} onDismiss={clearAlert} />
}
const mapStateToProps = state => {
return {
alert: state.alert
}
}
const mapDispatchToProps = dispatch => {
return {
clearAlert: () => dispatch(alertActions.clear())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(AlertContainer)

View File

@@ -0,0 +1,34 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import Alert from "../Alert"
describe("Alert", () => {
it("should render without crashing", () => {
shallow(<Alert />)
})
it("should call onDismiss when close button is clicked", () => {
const onDismiss = jest.fn()
const wrapper = mount(
<Alert show={true} type="danger" message="test" onDismiss={onDismiss} />
)
wrapper.find("button").simulate("click", { preventDefault: jest.fn() })
expect(onDismiss).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,34 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { AlertContainer } from "../AlertContainer"
describe("Alert", () => {
it("should render without crashing", () => {
shallow(
<AlertContainer alert={{ show: true, type: "danger", message: "Test" }} />
)
})
it("should render nothing if message is empty", () => {
const wrapper = shallow(
<AlertContainer alert={{ show: true, type: "danger", message: "" }} />
)
expect(wrapper.find("Alert").length).toBe(0)
})
})

View File

@@ -0,0 +1,69 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import configureStore from "redux-mock-store"
import thunk from "redux-thunk"
import * as actionsAlert from "../actions"
const middlewares = [thunk]
const mockStore = configureStore(middlewares)
jest.useFakeTimers()
describe("Alert actions", () => {
it("creates alert/SET action", () => {
const store = mockStore()
const expectedActions = [
{
type: "alert/SET",
alert: { id: 0, message: "Test alert", type: "danger" }
}
]
store.dispatch(actionsAlert.set({ message: "Test alert", type: "danger" }))
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
it("creates alert/CLEAR action for non danger alerts", () => {
const store = mockStore()
const expectedActions = [
{
type: "alert/SET",
alert: { id: 1, message: "Test alert" }
},
{
type: "alert/CLEAR",
alert: { id: 1 }
}
]
store.dispatch(actionsAlert.set({ message: "Test alert" }))
jest.runAllTimers()
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
it("creates alert/CLEAR action directly", () => {
const store = mockStore()
const expectedActions = [
{
type: "alert/CLEAR"
}
]
store.dispatch(actionsAlert.clear())
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
})

View File

@@ -0,0 +1,87 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import reducer from "../reducer"
import * as actionsAlert from "../actions"
describe("alert reducer", () => {
it("should return the initial state", () => {
expect(reducer(undefined, {})).toEqual({
show: false,
type: "danger"
})
})
it("should handle SET_ALERT", () => {
expect(
reducer(undefined, {
type: actionsAlert.SET,
alert: { id: 1, type: "danger", message: "Test message" }
})
).toEqual({
show: true,
id: 1,
type: "danger",
message: "Test message"
})
})
it("should clear alert if id not passed", () => {
expect(
reducer(
{ show: true, type: "danger", message: "Test message" },
{
type: actionsAlert.CLEAR
}
)
).toEqual({
show: false,
type: "danger"
})
})
it("should clear alert if id is matching", () => {
expect(
reducer(
{ show: true, id: 1, type: "danger", message: "Test message" },
{
type: actionsAlert.CLEAR,
alert: { id: 1 }
}
)
).toEqual({
show: false,
type: "danger"
})
})
it("should not clear alert if id is not matching", () => {
expect(
reducer(
{ show: true, id: 1, type: "danger", message: "Test message" },
{
type: actionsAlert.CLEAR,
alert: { id: 2 }
}
)
).toEqual({
show: true,
id: 1,
type: "danger",
message: "Test message"
})
})
})

View File

@@ -0,0 +1,46 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export const SET = "alert/SET"
export const CLEAR = "alert/CLEAR"
export let alertId = 0
export const set = alert => {
const id = alertId++
return (dispatch, getState) => {
if (alert.type !== "danger" || alert.autoClear) {
setTimeout(() => {
dispatch({
type: CLEAR,
alert: {
id
}
})
}, 5000)
}
dispatch({
type: SET,
alert: Object.assign({}, alert, {
id
})
})
}
}
export const clear = () => {
return { type: CLEAR }
}

View File

@@ -0,0 +1,41 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import * as actionsAlert from "./actions"
const initialState = {
show: false,
type: "danger"
}
export default (state = initialState, action) => {
switch (action.type) {
case actionsAlert.SET:
return {
show: true,
id: action.alert.id,
type: action.alert.type,
message: action.alert.message
}
case actionsAlert.CLEAR:
if (action.alert && action.alert.id != state.id) {
return state
} else {
return initialState
}
default:
return state
}
}

View File

@@ -0,0 +1,64 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { Modal } from "react-bootstrap"
import logo from "../../img/logo.svg"
export const AboutModal = ({ serverInfo, hideAbout }) => {
const { version, memory, platform, runtime } = serverInfo
return (
<Modal
className="modal-about modal-dark"
animation={false}
show={true}
onHide={hideAbout}
>
<button className="close" onClick={hideAbout}>
<span>×</span>
</button>
<div className="ma-inner">
<div className="mai-item hidden-xs">
<a href="https://min.io" target="_blank">
<img className="maii-logo" src={logo} alt="" />
</a>
</div>
<div className="mai-item">
<ul className="maii-list">
<li>
<div>Version</div>
<small>{version}</small>
</li>
<li>
<div>Memory</div>
<small>{memory}</small>
</li>
<li>
<div>Platform</div>
<small>{platform}</small>
</li>
<li>
<div>Runtime</div>
<small>{runtime}</small>
</li>
</ul>
</div>
</div>
</Modal>
)
}
export default AboutModal

View File

@@ -0,0 +1,40 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import classNames from "classnames"
import { connect } from "react-redux"
import SideBar from "./SideBar"
import MainContent from "./MainContent"
import AlertContainer from "../alert/AlertContainer"
class Browser extends React.Component {
render() {
return (
<div
className={classNames({
"file-explorer": true
})}
>
<SideBar />
<MainContent />
<AlertContainer />
</div>
)
}
}
export default connect(state => state)(Browser)

View File

@@ -0,0 +1,156 @@
/*
* MinIO Cloud Storage (C) 2016, 2017, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import { Dropdown } from "react-bootstrap"
import * as browserActions from "./actions"
import web from "../web"
import history from "../history"
import AboutModal from "./AboutModal"
import ChangePasswordModal from "./ChangePasswordModal"
export class BrowserDropdown extends React.Component {
constructor(props) {
super(props)
this.state = {
showAboutModal: false,
showChangePasswordModal: false
}
}
showAbout(e) {
e.preventDefault()
this.setState({
showAboutModal: true
})
}
hideAbout() {
this.setState({
showAboutModal: false
})
}
showChangePassword(e) {
e.preventDefault()
this.setState({
showChangePasswordModal: true
})
}
hideChangePassword() {
this.setState({
showChangePasswordModal: false
})
}
componentDidMount() {
const { fetchServerInfo } = this.props
fetchServerInfo()
}
fullScreen(e) {
e.preventDefault()
let el = document.documentElement
if (el.requestFullscreen) {
el.requestFullscreen()
}
if (el.mozRequestFullScreen) {
el.mozRequestFullScreen()
}
if (el.webkitRequestFullscreen) {
el.webkitRequestFullscreen()
}
if (el.msRequestFullscreen) {
el.msRequestFullscreen()
}
}
logout(e) {
e.preventDefault()
web.Logout()
history.replace("/login")
}
render() {
const { serverInfo } = this.props
return (
<li>
<Dropdown pullRight id="top-right-menu">
<Dropdown.Toggle noCaret>
<i className="fa fa-reorder" />
</Dropdown.Toggle>
<Dropdown.Menu className="dropdown-menu-right">
<li>
<a target="_blank" href="https://github.com/minio/minio">
GitHub <i className="fa fa-github" />
</a>
</li>
<li>
<a href="" onClick={this.fullScreen}>
Fullscreen <i className="fa fa-expand" />
</a>
</li>
<li>
<a target="_blank" href="https://docs.min.io/">
Documentation <i className="fa fa-book" />
</a>
</li>
<li>
<a target="_blank" href="https://slack.min.io">
Ask for help <i className="fa fa-question-circle" />
</a>
</li>
<li>
<a href="" id="show-about" onClick={this.showAbout.bind(this)}>
About <i className="fa fa-info-circle" />
</a>
{this.state.showAboutModal && (
<AboutModal
serverInfo={serverInfo}
hideAbout={this.hideAbout.bind(this)}
/>
)}
</li>
<li>
<a href="" onClick={this.showChangePassword.bind(this)}>
Change Password <i className="fa fa-cog" />
</a>
{this.state.showChangePasswordModal && (
<ChangePasswordModal
serverInfo={serverInfo}
hideChangePassword={this.hideChangePassword.bind(this)}
/>
)}
</li>
<li>
<a href="" id="logout" onClick={this.logout}>
Sign Out <i className="fa fa-sign-out" />
</a>
</li>
</Dropdown.Menu>
</Dropdown>
</li>
)
}
}
const mapStateToProps = state => {
return {
serverInfo: state.browser.serverInfo
}
}
const mapDispatchToProps = dispatch => {
return {
fetchServerInfo: () => dispatch(browserActions.fetchServerInfo())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(BrowserDropdown)

View File

@@ -0,0 +1,284 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import web from "../web"
import * as alertActions from "../alert/actions"
import { getRandomAccessKey, getRandomSecretKey } from "../utils"
import jwtDecode from "jwt-decode"
import classNames from "classnames"
import { Modal, ModalBody, ModalHeader } from "react-bootstrap"
import InputGroup from "./InputGroup"
import { ACCESS_KEY_MIN_LENGTH, SECRET_KEY_MIN_LENGTH } from "../constants"
export class ChangePasswordModal extends React.Component {
constructor(props) {
super(props)
this.state = {
currentAccessKey: "",
currentSecretKey: "",
currentSecretKeyVisible: false,
newAccessKey: "",
newSecretKey: "",
newSecretKeyVisible: false
}
}
// When its shown, it loads the access key from JWT token
componentWillMount() {
const token = jwtDecode(web.GetToken())
this.setState({
currentAccessKey: token.sub,
newAccessKey: token.sub
})
}
// Save the auth params and set them.
setAuth(e) {
const { showAlert } = this.props
if (this.canUpdateCredentials()) {
const currentAccessKey = this.state.currentAccessKey
const currentSecretKey = this.state.currentSecretKey
const newAccessKey = this.state.newAccessKey
const newSecretKey = this.state.newSecretKey
web
.SetAuth({
currentAccessKey,
currentSecretKey,
newAccessKey,
newSecretKey
})
.then(data => {
showAlert({
type: "success",
message: "Credentials updated successfully."
})
})
.catch(err => {
showAlert({
type: "danger",
message: err.message
})
})
}
}
generateAuth(e) {
const { serverInfo } = this.props
// Generate random access key only for root user
if (!serverInfo.userInfo.isIAMUser) {
this.setState({
newAccessKey: getRandomAccessKey()
})
}
this.setState({
newSecretKey: getRandomSecretKey(),
newSecretKeyVisible: true
})
}
canChangePassword() {
const { serverInfo } = this.props
// Password change is not allowed in WORM mode
if (serverInfo.info.isWorm) {
return false
}
// When credentials are set on ENV, password change not allowed for owner
if (serverInfo.info.isEnvCreds && !serverInfo.userInfo.isIAMUser) {
return false
}
return true
}
canUpdateCredentials() {
return (
this.state.currentAccessKey.length > 0 &&
this.state.currentSecretKey.length > 0 &&
this.state.newAccessKey.length >= ACCESS_KEY_MIN_LENGTH &&
this.state.newSecretKey.length >= SECRET_KEY_MIN_LENGTH
)
}
render() {
const { hideChangePassword, serverInfo } = this.props
const allowChangePassword = this.canChangePassword()
if (!allowChangePassword) {
return (
<Modal bsSize="sm" animation={false} show={true}>
<ModalHeader>Change Password</ModalHeader>
<ModalBody>
Credentials of this user cannot be updated through MinIO Browser.
</ModalBody>
<div className="modal-footer">
<button
id="cancel-change-password"
className="btn btn-link"
onClick={hideChangePassword}
>
Close
</button>
</div>
</Modal>
)
}
return (
<Modal bsSize="sm" animation={false} show={true}>
<ModalHeader>Change Password</ModalHeader>
<ModalBody className="m-t-20">
<div className="has-toggle-password">
<InputGroup
value={this.state.currentAccessKey}
id="currentAccessKey"
label="Current Access Key"
name="currentAccesskey"
type="text"
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
readonly={true}
/>
<i
onClick={() => {
this.setState({
currentSecretKeyVisible: !this.state.currentSecretKeyVisible
})
}}
className={
"toggle-password fa fa-eye " +
(this.state.currentSecretKeyVisible ? "toggled" : "")
}
/>
<InputGroup
value={this.state.currentSecretKey}
onChange={e => {
this.setState({ currentSecretKey: e.target.value })
}}
id="currentSecretKey"
label="Current Secret Key"
name="currentSecretKey"
type={this.state.currentSecretKeyVisible ? "text" : "password"}
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
/>
</div>
<div className="has-toggle-password m-t-30">
{!serverInfo.userInfo.isIAMUser && (
<InputGroup
value={this.state.newAccessKey}
id="newAccessKey"
label={"New Access Key"}
name="newAccesskey"
type="text"
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
onChange={e => {
this.setState({ newAccessKey: e.target.value })
}}
readonly={serverInfo.userInfo.isIAMUser}
/>
)}
<i
onClick={() => {
this.setState({
newSecretKeyVisible: !this.state.newSecretKeyVisible
})
}}
className={
"toggle-password fa fa-eye " +
(this.state.newSecretKeyVisible ? "toggled" : "")
}
/>
<InputGroup
value={this.state.newSecretKey}
onChange={e => {
this.setState({ newSecretKey: e.target.value })
}}
id="newSecretKey"
label="New Secret Key"
name="newSecretKey"
type={this.state.newSecretKeyVisible ? "text" : "password"}
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
onChange={e => {
this.setState({ newSecretKey: e.target.value })
}}
/>
</div>
</ModalBody>
<div className="modal-footer">
<button
id="generate-keys"
className={"btn btn-primary"}
onClick={this.generateAuth.bind(this)}
>
Generate
</button>
<button
id="update-keys"
className={classNames({
btn: true,
"btn-success": this.canUpdateCredentials()
})}
disabled={!this.canUpdateCredentials()}
onClick={this.setAuth.bind(this)}
>
Update
</button>
<button
id="cancel-change-password"
className="btn btn-link"
onClick={hideChangePassword}
>
Cancel
</button>
</div>
</Modal>
)
}
}
const mapStateToProps = state => {
return {
serverInfo: state.browser.serverInfo
}
}
const mapDispatchToProps = dispatch => {
return {
showAlert: alert => dispatch(alertActions.set(alert))
}
}
export default connect(
mapStateToProps,
mapDispatchToProps
)(ChangePasswordModal)

View File

@@ -0,0 +1,57 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { Modal, ModalBody } from "react-bootstrap"
let ConfirmModal = ({
baseClass,
icon,
text,
sub,
okText,
cancelText,
okHandler,
cancelHandler,
show
}) => {
return (
<Modal
bsSize="small"
animation={false}
show={show}
className={"modal-confirm " + (baseClass || "")}
>
<ModalBody>
<div className="mc-icon">
<i className={icon} />
</div>
<div className="mc-text">{text}</div>
<div className="mc-sub">{sub}</div>
</ModalBody>
<div className="modal-footer">
<button className="btn btn-danger" onClick={okHandler}>
{okText}
</button>
<button className="btn btn-link" onClick={cancelHandler}>
{cancelText}
</button>
</div>
</Modal>
)
}
export default ConfirmModal

View File

@@ -0,0 +1,43 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import Path from "../objects/Path"
import StorageInfo from "./StorageInfo"
import BrowserDropdown from "./BrowserDropdown"
import web from "../web"
import { minioBrowserPrefix } from "../constants"
export const Header = () => {
const loggedIn = web.LoggedIn()
return (
<header className="fe-header">
<Path />
{loggedIn && <StorageInfo />}
<ul className="feh-actions">
{loggedIn ? (
<BrowserDropdown />
) : (
<a className="btn btn-danger" href={minioBrowserPrefix + "/login"}>
Login
</a>
)}
</ul>
</header>
)
}
export default Header

View File

@@ -0,0 +1,26 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
export const Host = () => (
<div className="fes-host">
<i className="fa fa-globe" />
<a href="/">{window.location.host}</a>
</div>
)
export default Host

View File

@@ -0,0 +1,70 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
let InputGroup = ({
label,
id,
name,
value,
onChange,
type,
spellCheck,
required,
readonly,
autoComplete,
align,
className
}) => {
var input = (
<input
id={id}
name={name}
value={value}
onChange={onChange}
className="ig-text"
type={type}
spellCheck={spellCheck}
required={required}
autoComplete={autoComplete}
/>
)
if (readonly)
input = (
<input
id={id}
name={name}
value={value}
onChange={onChange}
className="ig-text"
type={type}
spellCheck={spellCheck}
required={required}
autoComplete={autoComplete}
disabled
/>
)
return (
<div className={"input-group " + align + " " + className}>
{input}
<i className="ig-helpers" />
<label className="ig-label">{label}</label>
</div>
)
}
export default InputGroup

View File

@@ -0,0 +1,154 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import classNames from "classnames"
import logo from "../../img/logo.svg"
import Alert from "../alert/Alert"
import * as actionsAlert from "../alert/actions"
import InputGroup from "./InputGroup"
import web from "../web"
import { Redirect } from "react-router-dom"
export class Login extends React.Component {
constructor(props) {
super(props)
this.state = {
accessKey: "",
secretKey: ""
}
}
// Handle field changes
accessKeyChange(e) {
this.setState({
accessKey: e.target.value
})
}
secretKeyChange(e) {
this.setState({
secretKey: e.target.value
})
}
handleSubmit(event) {
event.preventDefault()
const { showAlert, clearAlert, history } = this.props
let message = ""
if (this.state.accessKey === "") {
message = "Access Key cannot be empty"
}
if (this.state.secretKey === "") {
message = "Secret Key cannot be empty"
}
if (message) {
showAlert("danger", message)
return
}
web
.Login({
username: this.state.accessKey,
password: this.state.secretKey
})
.then(res => {
// Clear alerts from previous login attempts
clearAlert()
history.push("/")
})
.catch(e => {
showAlert("danger", e.message)
})
}
componentWillMount() {
const { clearAlert } = this.props
// Clear out any stale message in the alert of previous page
clearAlert()
document.body.classList.add("is-guest")
}
componentWillUnmount() {
document.body.classList.remove("is-guest")
}
render() {
const { clearAlert, alert } = this.props
if (web.LoggedIn()) {
return <Redirect to={"/"} />
}
let alertBox = <Alert {...alert} onDismiss={clearAlert} />
// Make sure you don't show a fading out alert box on the initial web-page load.
if (!alert.message) alertBox = ""
return (
<div className="login">
{alertBox}
<div className="l-wrap">
<form onSubmit={this.handleSubmit.bind(this)}>
<InputGroup
value={this.state.accessKey}
onChange={this.accessKeyChange.bind(this)}
className="ig-dark"
label="Access Key"
id="accessKey"
name="username"
type="text"
spellCheck="false"
required="required"
autoComplete="username"
/>
<InputGroup
value={this.state.secretKey}
onChange={this.secretKeyChange.bind(this)}
className="ig-dark"
label="Secret Key"
id="secretKey"
name="password"
type="password"
spellCheck="false"
required="required"
autoComplete="new-password"
/>
<button className="lw-btn" type="submit">
<i className="fa fa-sign-in" />
</button>
</form>
</div>
<div className="l-footer">
<a className="lf-logo" href="">
<img src={logo} alt="" />
</a>
<div className="lf-server">{window.location.host}</div>
</div>
</div>
)
}
}
const mapDispatchToProps = dispatch => {
return {
showAlert: (type, message) =>
dispatch(actionsAlert.set({ type: type, message: message })),
clearAlert: () => dispatch(actionsAlert.clear())
}
}
export default connect(
state => state,
mapDispatchToProps
)(Login)

View File

@@ -0,0 +1,106 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import { Dropdown, OverlayTrigger, Tooltip } from "react-bootstrap"
import web from "../web"
import * as actionsBuckets from "../buckets/actions"
import * as uploadsActions from "../uploads/actions"
import { getPrefixWritable } from "../objects/selectors"
export const MainActions = ({
prefixWritable,
uploadFile,
showMakeBucketModal
}) => {
const uploadTooltip = <Tooltip id="tt-upload-file">Upload file</Tooltip>
const makeBucketTooltip = (
<Tooltip id="tt-create-bucket">Create bucket</Tooltip>
)
const onFileUpload = e => {
e.preventDefault()
let files = e.target.files
let filesToUploadCount = files.length
for (let i = 0; i < filesToUploadCount; i++) {
uploadFile(files.item(i))
}
e.target.value = null
}
const loggedIn = web.LoggedIn()
if (loggedIn || prefixWritable) {
return (
<Dropdown dropup className="feb-actions" id="fe-action-toggle">
<Dropdown.Toggle noCaret className="feba-toggle">
<span>
<i className="fa fa-plus" />
</span>
</Dropdown.Toggle>
<Dropdown.Menu>
<OverlayTrigger placement="left" overlay={uploadTooltip}>
<a href="#" className="feba-btn feba-upload">
<input
type="file"
onChange={onFileUpload}
style={{ display: "none" }}
id="file-input"
multiple={true}
/>
<label htmlFor="file-input">
{" "}
<i className="fa fa-cloud-upload" />{" "}
</label>
</a>
</OverlayTrigger>
{loggedIn && (
<OverlayTrigger placement="left" overlay={makeBucketTooltip}>
<a
href="#"
id="show-make-bucket"
className="feba-btn feba-bucket"
onClick={e => {
e.preventDefault()
showMakeBucketModal()
}}
>
<i className="fa fa-hdd-o" />
</a>
</OverlayTrigger>
)}
</Dropdown.Menu>
</Dropdown>
)
} else {
return <noscript />
}
}
const mapStateToProps = state => {
return {
prefixWritable: getPrefixWritable(state)
}
}
const mapDispatchToProps = dispatch => {
return {
uploadFile: file => dispatch(uploadsActions.uploadFile(file)),
showMakeBucketModal: () => dispatch(actionsBuckets.showMakeBucketModal())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(MainActions)

View File

@@ -0,0 +1,43 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import MobileHeader from "./MobileHeader"
import Header from "./Header"
import ObjectsSection from "../objects/ObjectsSection"
import MainActions from "./MainActions"
import BucketPolicyModal from "../buckets/BucketPolicyModal"
import MakeBucketModal from "../buckets/MakeBucketModal"
import UploadModal from "../uploads/UploadModal"
import ObjectsBulkActions from "../objects/ObjectsBulkActions"
import Dropzone from "../uploads/Dropzone"
export const MainContent = () => (
<div className="fe-body">
<ObjectsBulkActions />
<MobileHeader />
<Dropzone>
<Header />
<ObjectsSection />
</Dropzone>
<MainActions />
<BucketPolicyModal />
<MakeBucketModal />
<UploadModal />
</div>
)
export default MainContent

View File

@@ -0,0 +1,60 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import classNames from "classnames"
import { connect } from "react-redux"
import logo from "../../img/logo.svg"
import * as actionsCommon from "./actions"
export const MobileHeader = ({ sidebarOpen, toggleSidebar }) => (
<header className="fe-header-mobile hidden-lg hidden-md">
<div
id="sidebar-toggle"
className={
"feh-trigger " +
classNames({
"feht-toggled": sidebarOpen
})
}
onClick={e => {
e.stopPropagation()
toggleSidebar()
}}
>
<div className="feht-lines">
<div className="top" />
<div className="center" />
<div className="bottom" />
</div>
</div>
<img className="mh-logo" src={logo} alt="" />
</header>
)
const mapStateToProps = state => {
return {
sidebarOpen: state.browser.sidebarOpen
}
}
const mapDispatchToProps = dispatch => {
return {
toggleSidebar: () => dispatch(actionsCommon.toggleSidebar())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(MobileHeader)

View File

@@ -0,0 +1,65 @@
/*
* MinIO Cloud Storage (C) 2016, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import classNames from "classnames"
import ClickOutHandler from "react-onclickout"
import { connect } from "react-redux"
import logo from "../../img/logo.svg"
import Dropdown from "react-bootstrap/lib/Dropdown"
import BucketSearch from "../buckets/BucketSearch"
import BucketList from "../buckets/BucketList"
import Host from "./Host"
import * as actionsCommon from "./actions"
import web from "../web"
export const SideBar = ({ sidebarOpen, clickOutside }) => {
return (
<ClickOutHandler onClickOut={clickOutside}>
<div
className={classNames({
"fe-sidebar": true,
toggled: sidebarOpen
})}
>
<div className="fes-header clearfix hidden-sm hidden-xs">
<img src={logo} alt="" />
<h2>MinIO Browser</h2>
</div>
<div className="fes-list">
{web.LoggedIn() && <BucketSearch />}
<BucketList />
</div>
<Host />
</div>
</ClickOutHandler>
)
}
const mapStateToProps = state => {
return {
sidebarOpen: state.browser.sidebarOpen
}
}
const mapDispatchToProps = dispatch => {
return {
clickOutside: () => dispatch(actionsCommon.closeSidebar())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(SideBar)

View File

@@ -0,0 +1,57 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import humanize from "humanize"
import * as actionsCommon from "./actions"
export class StorageInfo extends React.Component {
componentWillMount() {
const { fetchStorageInfo } = this.props
fetchStorageInfo()
}
render() {
const { used } = this.props.storageInfo
return (
<div className="feh-used">
<div className="fehu-chart">
<div style={{ width: 0 }} />
</div>
<ul>
<li>
<span>Used: </span>
{humanize.filesize(used)}
</li>
</ul>
</div>
)
}
}
const mapStateToProps = state => {
return {
storageInfo: state.browser.storageInfo
}
}
const mapDispatchToProps = dispatch => {
return {
fetchStorageInfo: () => dispatch(actionsCommon.fetchStorageInfo())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(StorageInfo)

View File

@@ -0,0 +1,41 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { AboutModal } from "../AboutModal"
describe("AboutModal", () => {
const serverInfo = {
version: "test",
memory: "test",
platform: "test",
runtime: "test"
}
it("should render without crashing", () => {
shallow(<AboutModal serverInfo={serverInfo} />)
})
it("should call hideAbout when close button is clicked", () => {
const hideAbout = jest.fn()
const wrapper = shallow(
<AboutModal serverInfo={serverInfo} hideAbout={hideAbout} />
)
wrapper.find("button").simulate("click")
expect(hideAbout).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,29 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import Browser from "../Browser"
import configureStore from "redux-mock-store"
const mockStore = configureStore()
describe("Browser", () => {
it("should render without crashing", () => {
const store = mockStore()
shallow(<Browser store={store}/>)
})
})

View File

@@ -0,0 +1,63 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { BrowserDropdown } from "../BrowserDropdown"
describe("BrowserDropdown", () => {
const serverInfo = {
version: "test",
memory: "test",
platform: "test",
runtime: "test"
}
it("should render without crashing", () => {
shallow(
<BrowserDropdown serverInfo={serverInfo} fetchServerInfo={jest.fn()} />
)
})
it("should call fetchServerInfo after its mounted", () => {
const fetchServerInfo = jest.fn()
const wrapper = shallow(
<BrowserDropdown
serverInfo={serverInfo}
fetchServerInfo={fetchServerInfo}
/>
)
expect(fetchServerInfo).toHaveBeenCalled()
})
it("should show AboutModal when About link is clicked", () => {
const wrapper = shallow(
<BrowserDropdown serverInfo={serverInfo} fetchServerInfo={jest.fn()} />
)
wrapper.find("#show-about").simulate("click", { preventDefault: jest.fn() })
wrapper.update()
expect(wrapper.state("showAboutModal")).toBeTruthy()
expect(wrapper.find("AboutModal").length).toBe(1)
})
it("should logout and redirect to /login when logout is clicked", () => {
const wrapper = shallow(
<BrowserDropdown serverInfo={serverInfo} fetchServerInfo={jest.fn()} />
)
wrapper.find("#logout").simulate("click", { preventDefault: jest.fn() })
expect(window.location.pathname.endsWith("/login")).toBeTruthy()
})
})

View File

@@ -0,0 +1,181 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { ChangePasswordModal } from "../ChangePasswordModal"
import jwtDecode from "jwt-decode"
jest.mock("jwt-decode")
jwtDecode.mockImplementation(() => ({ sub: "minio" }))
jest.mock("../../web", () => ({
GenerateAuth: jest.fn(() => {
return Promise.resolve({ accessKey: "gen1", secretKey: "gen2" })
}),
SetAuth: jest.fn(
({ currentAccessKey, currentSecretKey, newAccessKey, newSecretKey }) => {
if (
currentAccessKey == "minio" &&
currentSecretKey == "minio123" &&
newAccessKey == "test" &&
newSecretKey == "test1234"
) {
return Promise.resolve({})
} else {
return Promise.reject({
message: "Error"
})
}
}
),
GetToken: jest.fn(() => "")
}))
jest.mock("../../utils", () => ({
getRandomAccessKey: () => "raccesskey",
getRandomSecretKey: () => "rsecretkey"
}))
describe("ChangePasswordModal", () => {
const serverInfo = {
version: "test",
memory: "test",
platform: "test",
runtime: "test",
info: { isEnvCreds: false },
userInfo: { isIAMUser: false }
}
it("should render without crashing", () => {
shallow(<ChangePasswordModal serverInfo={serverInfo} />)
})
it("should not allow changing password when isWorm is true", () => {
const newServerInfo = { ...serverInfo, info: { isWorm: true } }
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
expect(
wrapper
.find("ModalBody")
.childAt(0)
.text()
).toBe("Credentials of this user cannot be updated through MinIO Browser.")
})
it("should not allow changing password when isEnvCreds is true and not IAM user", () => {
const newServerInfo = {
...serverInfo,
info: { isEnvCreds: true },
userInfo: { isIAMUser: false }
}
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
expect(
wrapper
.find("ModalBody")
.childAt(0)
.text()
).toBe("Credentials of this user cannot be updated through MinIO Browser.")
})
it("should generate accessKey and secretKey when Generate buttons is clicked", () => {
const wrapper = shallow(<ChangePasswordModal serverInfo={serverInfo} />)
wrapper.find("#generate-keys").simulate("click")
setImmediate(() => {
expect(wrapper.state("newAccessKey")).toBe("raccesskey")
expect(wrapper.state("newSecretKey")).toBe("rsecretkey")
})
})
it("should not generate accessKey for IAM User", () => {
const newServerInfo = {
...serverInfo,
userInfo: { isIAMUser: true }
}
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
wrapper.find("#generate-keys").simulate("click")
setImmediate(() => {
expect(wrapper.state("newAccessKey")).toBe("minio")
expect(wrapper.state("newSecretKey")).toBe("rsecretkey")
})
})
it("should not show new accessKey field for IAM User", () => {
const newServerInfo = {
...serverInfo,
userInfo: { isIAMUser: true }
}
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
expect(wrapper.find("#newAccesskey").exists()).toBeFalsy()
})
it("should disble Update button for invalid accessKey or secretKey", () => {
const showAlert = jest.fn()
const wrapper = shallow(
<ChangePasswordModal serverInfo={serverInfo} showAlert={showAlert} />
)
wrapper
.find("#currentAccessKey")
.simulate("change", { target: { value: "minio" } })
wrapper
.find("#currentSecretKey")
.simulate("change", { target: { value: "minio123" } })
wrapper.find("#newAccessKey").simulate("change", { target: { value: "t" } })
wrapper
.find("#newSecretKey")
.simulate("change", { target: { value: "t1" } })
expect(wrapper.find("#update-keys").prop("disabled")).toBeTruthy()
})
it("should update accessKey and secretKey when Update button is clicked", () => {
const showAlert = jest.fn()
const wrapper = shallow(
<ChangePasswordModal serverInfo={serverInfo} showAlert={showAlert} />
)
wrapper
.find("#currentAccessKey")
.simulate("change", { target: { value: "minio" } })
wrapper
.find("#currentSecretKey")
.simulate("change", { target: { value: "minio123" } })
wrapper
.find("#newAccessKey")
.simulate("change", { target: { value: "test" } })
wrapper
.find("#newSecretKey")
.simulate("change", { target: { value: "test1234" } })
expect(wrapper.find("#update-keys").prop("disabled")).toBeFalsy()
wrapper.find("#update-keys").simulate("click")
setImmediate(() => {
expect(showAlert).toHaveBeenCalledWith({
type: "success",
message: "Credentials updated successfully."
})
})
})
it("should call hideChangePassword when Cancel button is clicked", () => {
const hideChangePassword = jest.fn()
const wrapper = shallow(
<ChangePasswordModal
serverInfo={serverInfo}
hideChangePassword={hideChangePassword}
/>
)
wrapper.find("#cancel-change-password").simulate("click")
expect(hideChangePassword).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,42 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import Header from "../Header"
jest.mock("../../web", () => ({
LoggedIn: jest
.fn(() => true)
.mockReturnValueOnce(true)
.mockReturnValueOnce(false)
}))
describe("Header", () => {
it("should render without crashing", () => {
shallow(<Header />)
})
it("should render Login button when the user has not LoggedIn", () => {
const wrapper = shallow(<Header />)
expect(wrapper.find("a").text()).toBe("Login")
})
it("should render StorageInfo and BrowserDropdown when the user has LoggedIn", () => {
const wrapper = shallow(<Header />)
expect(wrapper.find("Connect(BrowserDropdown)").length).toBe(1)
expect(wrapper.find("Connect(StorageInfo)").length).toBe(1)
})
})

View File

@@ -0,0 +1,25 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import Host from "../Host"
describe("Host", () => {
it("should render without crashing", () => {
shallow(<Host />)
})
})

View File

@@ -0,0 +1,105 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { Login } from "../Login"
import web from "../../web"
jest.mock('../../web', () => ({
Login: jest.fn(() => {
return Promise.resolve({ token: "test", uiVersion: "2018-02-01T01:17:47Z" })
}),
LoggedIn: jest.fn()
}))
describe("Login", () => {
const dispatchMock = jest.fn()
const showAlertMock = jest.fn()
const clearAlertMock = jest.fn()
it("should render without crashing", () => {
shallow(<Login
dispatch={dispatchMock}
alert={{ show: false, type: "danger"}}
showAlert={showAlertMock}
clearAlert={clearAlertMock}
/>)
})
it("should initially have the is-guest class", () => {
const wrapper = shallow(
<Login
dispatch={dispatchMock}
alert={{ show: false, type: "danger"}}
showAlert={showAlertMock}
clearAlert={clearAlertMock}
/>,
{ attachTo: document.body }
)
expect(document.body.classList.contains("is-guest")).toBeTruthy()
})
it("should throw an alert if the keys are empty in login form", () => {
const wrapper = mount(
<Login
dispatch={dispatchMock}
alert={{ show: false, type: "danger"}}
showAlert={showAlertMock}
clearAlert={clearAlertMock}
/>
)
// case where both keys are empty - displays the second warning
wrapper.find("form").simulate("submit")
expect(showAlertMock).toHaveBeenCalledWith("danger", "Secret Key cannot be empty")
// case where access key is empty
wrapper.setState({
accessKey: "",
secretKey: "secretKey"
})
wrapper.find("form").simulate("submit")
expect(showAlertMock).toHaveBeenCalledWith("danger", "Access Key cannot be empty")
// case where secret key is empty
wrapper.setState({
accessKey: "accessKey",
secretKey: ""
})
wrapper.find("form").simulate("submit")
expect(showAlertMock).toHaveBeenCalledWith("danger", "Secret Key cannot be empty")
})
it("should call web.Login with correct arguments if both keys are entered", () => {
const wrapper = mount(
<Login
dispatch={dispatchMock}
alert={{ show: false, type: "danger"}}
showAlert={showAlertMock}
clearAlert={clearAlertMock}
/>
)
wrapper.setState({
accessKey: "accessKey",
secretKey: "secretKey"
})
wrapper.find("form").simulate("submit")
expect(web.Login).toHaveBeenCalledWith({
"username": "accessKey",
"password": "secretKey"
})
})
})

View File

@@ -0,0 +1,82 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { MainActions } from "../MainActions"
jest.mock("../../web", () => ({
LoggedIn: jest
.fn(() => true)
.mockReturnValueOnce(true)
.mockReturnValueOnce(false)
.mockReturnValueOnce(false)
}))
describe("MainActions", () => {
it("should render without crashing", () => {
shallow(<MainActions />)
})
it("should not show any actions when user has not LoggedIn and prefixWritable is false", () => {
const wrapper = shallow(<MainActions />)
expect(wrapper.find("#show-make-bucket").length).toBe(0)
expect(wrapper.find("#file-input").length).toBe(0)
})
it("should show only file upload action when user has not LoggedIn and prefixWritable is true", () => {
const wrapper = shallow(<MainActions prefixWritable={true} />)
expect(wrapper.find("#show-make-bucket").length).toBe(0)
expect(wrapper.find("#file-input").length).toBe(1)
})
it("should show make bucket upload file actions when user has LoggedIn", () => {
const wrapper = shallow(<MainActions />)
expect(wrapper.find("#show-make-bucket").length).toBe(1)
expect(wrapper.find("#file-input").length).toBe(1)
})
it("should call showMakeBucketModal when create bucket icon is clicked", () => {
const showMakeBucketModal = jest.fn()
const wrapper = shallow(
<MainActions showMakeBucketModal={showMakeBucketModal} />
)
wrapper
.find("#show-make-bucket")
.simulate("click", { preventDefault: jest.fn() })
expect(showMakeBucketModal).toHaveBeenCalled()
})
it("should call uploadFile when a file is selected for upload", () => {
const uploadFile = jest.fn()
const wrapper = shallow(<MainActions uploadFile={uploadFile} />)
const files = [new Blob(["file content"], { type: "text/plain" })]
const input = wrapper.find("#file-input")
const event = {
preventDefault: jest.fn(),
target: {
files: {
length: files.length,
item: function(index) {
return files[index]
}
}
}
}
input.simulate("change", event)
expect(uploadFile).toHaveBeenCalledWith(files[0])
})
})

View File

@@ -0,0 +1,25 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import MainContent from "../MainContent"
describe("MainContent", () => {
it("should render without crashing", () => {
shallow(<MainContent />)
})
})

View File

@@ -0,0 +1,36 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { MobileHeader } from "../MobileHeader"
describe("Bucket", () => {
it("should render without crashing", () => {
shallow(<MobileHeader sidebarOpen={false} />)
})
it("should toggleSidebar when trigger is clicked", () => {
const toggleSidebar = jest.fn()
const wrapper = shallow(
<MobileHeader sidebarOpen={false} toggleSidebar={toggleSidebar} />
)
wrapper
.find("#sidebar-toggle")
.simulate("click", { stopPropagation: jest.fn() })
expect(toggleSidebar).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,41 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { SideBar } from "../SideBar"
jest.mock("../../web", () => ({
LoggedIn: jest.fn(() => false).mockReturnValueOnce(true)
}))
describe("SideBar", () => {
it("should render without crashing", () => {
shallow(<SideBar />)
})
it("should not render BucketSearch for non LoggedIn users", () => {
const wrapper = shallow(<SideBar />)
expect(wrapper.find("Connect(BucketSearch)").length).toBe(0)
})
it("should call clickOutside when the user clicks outside the sidebar", () => {
const clickOutside = jest.fn()
const wrapper = shallow(<SideBar clickOutside={clickOutside} />)
wrapper.simulate("clickOut", { preventDefault: jest.fn() })
expect(clickOutside).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,41 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { StorageInfo } from "../StorageInfo"
describe("StorageInfo", () => {
it("should render without crashing", () => {
shallow(
<StorageInfo
storageInfo={{ used: 60 }}
fetchStorageInfo={jest.fn()}
/>
)
})
it("should fetchStorageInfo before component is mounted", () => {
const fetchStorageInfo = jest.fn()
shallow(
<StorageInfo
storageInfo={{ used: 60 }}
fetchStorageInfo={fetchStorageInfo}
/>
)
expect(fetchStorageInfo).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,70 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import configureStore from "redux-mock-store"
import thunk from "redux-thunk"
import * as actionsCommon from "../actions"
jest.mock("../../web", () => ({
StorageInfo: jest.fn(() => {
return Promise.resolve({ storageInfo: { Used: 60 } })
}),
ServerInfo: jest.fn(() => {
return Promise.resolve({
MinioVersion: "test",
MinioMemory: "test",
MinioPlatform: "test",
MinioRuntime: "test",
MinioGlobalInfo: "test"
})
})
}))
const middlewares = [thunk]
const mockStore = configureStore(middlewares)
describe("Common actions", () => {
it("creates common/SET_STORAGE_INFO after fetching the storage details ", () => {
const store = mockStore()
const expectedActions = [
{ type: "common/SET_STORAGE_INFO", storageInfo: { used: 60 } }
]
return store.dispatch(actionsCommon.fetchStorageInfo()).then(() => {
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
})
it("creates common/SET_SERVER_INFO after fetching the server details", () => {
const store = mockStore()
const expectedActions = [
{
type: "common/SET_SERVER_INFO",
serverInfo: {
version: "test",
memory: "test",
platform: "test",
runtime: "test",
info: "test"
}
}
]
return store.dispatch(actionsCommon.fetchServerInfo()).then(() => {
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
})
})

View File

@@ -0,0 +1,92 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import reducer from "../reducer"
import * as actionsCommon from "../actions"
describe("common reducer", () => {
it("should return the initial state", () => {
expect(reducer(undefined, {})).toEqual({
sidebarOpen: false,
storageInfo: {
total: 0,
free: 0
},
serverInfo: {}
})
})
it("should handle TOGGLE_SIDEBAR", () => {
expect(
reducer(
{ sidebarOpen: false },
{
type: actionsCommon.TOGGLE_SIDEBAR
}
)
).toEqual({
sidebarOpen: true
})
})
it("should handle CLOSE_SIDEBAR", () => {
expect(
reducer(
{ sidebarOpen: true },
{
type: actionsCommon.CLOSE_SIDEBAR
}
)
).toEqual({
sidebarOpen: false
})
})
it("should handle SET_STORAGE_INFO", () => {
expect(
reducer(
{},
{
type: actionsCommon.SET_STORAGE_INFO,
storageInfo: { total: 100, free: 40 }
}
)
).toEqual({
storageInfo: { total: 100, free: 40 }
})
})
it("should handle SET_SERVER_INFO", () => {
expect(
reducer(undefined, {
type: actionsCommon.SET_SERVER_INFO,
serverInfo: {
version: "test",
memory: "test",
platform: "test",
runtime: "test",
info: "test"
}
}).serverInfo
).toEqual({
version: "test",
memory: "test",
platform: "test",
runtime: "test",
info: "test"
})
})
})

View File

@@ -0,0 +1,68 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import web from "../web"
export const TOGGLE_SIDEBAR = "common/TOGGLE_SIDEBAR"
export const CLOSE_SIDEBAR = "common/CLOSE_SIDEBAR"
export const SET_STORAGE_INFO = "common/SET_STORAGE_INFO"
export const SET_SERVER_INFO = "common/SET_SERVER_INFO"
export const toggleSidebar = () => ({
type: TOGGLE_SIDEBAR
})
export const closeSidebar = () => ({
type: CLOSE_SIDEBAR
})
export const fetchStorageInfo = () => {
return function(dispatch) {
return web.StorageInfo().then(res => {
const storageInfo = {
total: res.storageInfo.Total,
used: res.storageInfo.Used
}
dispatch(setStorageInfo(storageInfo))
})
}
}
export const setStorageInfo = storageInfo => ({
type: SET_STORAGE_INFO,
storageInfo
})
export const fetchServerInfo = () => {
return function(dispatch) {
return web.ServerInfo().then(res => {
const serverInfo = {
version: res.MinioVersion,
memory: res.MinioMemory,
platform: res.MinioPlatform,
runtime: res.MinioRuntime,
info: res.MinioGlobalInfo,
userInfo: res.MinioUserInfo
}
dispatch(setServerInfo(serverInfo))
})
}
}
export const setServerInfo = serverInfo => ({
type: SET_SERVER_INFO,
serverInfo
})

View File

@@ -0,0 +1,45 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import * as actionsCommon from "./actions"
export default (
state = {
sidebarOpen: false,
storageInfo: { total: 0, free: 0 },
serverInfo: {}
},
action
) => {
switch (action.type) {
case actionsCommon.TOGGLE_SIDEBAR:
return Object.assign({}, state, {
sidebarOpen: !state.sidebarOpen
})
case actionsCommon.CLOSE_SIDEBAR:
return Object.assign({}, state, {
sidebarOpen: false
})
case actionsCommon.SET_STORAGE_INFO:
return Object.assign({}, state, {
storageInfo: action.storageInfo
})
case actionsCommon.SET_SERVER_INFO:
return { ...state, serverInfo: action.serverInfo }
default:
return state
}
}

View File

@@ -0,0 +1,45 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import classNames from "classnames"
import BucketDropdown from "./BucketDropdown"
export const Bucket = ({ bucket, isActive, selectBucket }) => {
return (
<li
className={classNames({
active: isActive
})}
onClick={e => {
e.preventDefault()
selectBucket(bucket)
}}
>
<a
href=""
className={classNames({
"fesli-loading": false
})}
>
{bucket}
</a>
<BucketDropdown bucket={bucket}/>
</li>
)
}
export default Bucket

View File

@@ -0,0 +1,35 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import * as actionsBuckets from "./actions"
import { getCurrentBucket } from "./selectors"
import Bucket from "./Bucket"
const mapStateToProps = (state, ownProps) => {
return {
isActive: getCurrentBucket(state) === ownProps.bucket
}
}
const mapDispatchToProps = dispatch => {
return {
selectBucket: bucket => dispatch(actionsBuckets.selectBucket(bucket))
}
}
export default connect(mapStateToProps, mapDispatchToProps)(Bucket)

View File

@@ -0,0 +1,92 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import classNames from "classnames"
import { connect } from "react-redux"
import * as actionsBuckets from "./actions"
import { getCurrentBucket } from "./selectors"
import Dropdown from "react-bootstrap/lib/Dropdown"
export class BucketDropdown extends React.Component {
constructor(props) {
super(props)
this.state = {
showBucketDropdown: false
}
}
toggleDropdown() {
if (this.state.showBucketDropdown) {
this.setState({
showBucketDropdown: false
})
} else {
this.setState({
showBucketDropdown: true
})
}
}
render() {
const { bucket, showBucketPolicy, deleteBucket, currentBucket } = this.props
return (
<Dropdown
open = {this.state.showBucketDropdown}
onToggle = {this.toggleDropdown.bind(this)}
className="bucket-dropdown"
id="bucket-dropdown"
>
<Dropdown.Toggle noCaret>
<i className="zmdi zmdi-more-vert" />
</Dropdown.Toggle>
<Dropdown.Menu className="dropdown-menu-right">
<li>
<a
onClick={e => {
e.stopPropagation()
this.toggleDropdown()
showBucketPolicy()
}}
>
Edit policy
</a>
</li>
<li>
<a
onClick={e => {
e.stopPropagation()
this.toggleDropdown()
deleteBucket(bucket)
}}
>
Delete
</a>
</li>
</Dropdown.Menu>
</Dropdown>
)
}
}
const mapDispatchToProps = dispatch => {
return {
deleteBucket: bucket => dispatch(actionsBuckets.deleteBucket(bucket)),
showBucketPolicy: () => dispatch(actionsBuckets.showBucketPolicy())
}
}
export default connect(state => state, mapDispatchToProps)(BucketDropdown)

View File

@@ -0,0 +1,78 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import { Scrollbars } from "react-custom-scrollbars"
import * as actionsBuckets from "./actions"
import { getVisibleBuckets } from "./selectors"
import BucketContainer from "./BucketContainer"
import web from "../web"
import history from "../history"
import { pathSlice } from "../utils"
export class BucketList extends React.Component {
componentWillMount() {
const { fetchBuckets, setBucketList, selectBucket } = this.props
if (web.LoggedIn()) {
fetchBuckets()
} else {
const { bucket, prefix } = pathSlice(history.location.pathname)
if (bucket) {
setBucketList([bucket])
selectBucket(bucket, prefix)
} else {
history.replace("/login")
}
}
}
render() {
const { visibleBuckets } = this.props
return (
<div className="fesl-inner">
<Scrollbars
renderTrackVertical={props => <div className="scrollbar-vertical" />}
>
<ul>
{visibleBuckets.map(bucket => (
<BucketContainer key={bucket} bucket={bucket} />
))}
</ul>
</Scrollbars>
</div>
)
}
}
const mapStateToProps = state => {
return {
visibleBuckets: getVisibleBuckets(state)
}
}
const mapDispatchToProps = dispatch => {
return {
fetchBuckets: () => dispatch(actionsBuckets.fetchBuckets()),
setBucketList: buckets => dispatch(actionsBuckets.setList(buckets)),
selectBucket: (bucket, prefix) =>
dispatch(actionsBuckets.selectBucket(bucket, prefix))
}
}
export default connect(
mapStateToProps,
mapDispatchToProps
)(BucketList)

View File

@@ -0,0 +1,61 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import { Modal, ModalHeader } from "react-bootstrap"
import * as actionsBuckets from "./actions"
import PolicyInput from "./PolicyInput"
import Policy from "./Policy"
export const BucketPolicyModal = ({ showBucketPolicy, currentBucket, hideBucketPolicy, policies }) => {
return (
<Modal className="modal-policy"
animation={ false }
show={ showBucketPolicy }
onHide={ hideBucketPolicy }
>
<ModalHeader>
Bucket Policy (
{ currentBucket })
<button className="close close-alt" onClick={ hideBucketPolicy }>
<span>×</span>
</button>
</ModalHeader>
<div className="pm-body">
<PolicyInput />
{ policies.map((policy, i) => <Policy key={ i } prefix={ policy.prefix } policy={ policy.policy } />
) }
</div>
</Modal>
)
}
const mapStateToProps = state => {
return {
currentBucket: state.buckets.currentBucket,
showBucketPolicy: state.buckets.showBucketPolicy,
policies: state.buckets.policies
}
}
const mapDispatchToProps = dispatch => {
return {
hideBucketPolicy: () => dispatch(actionsBuckets.hideBucketPolicy())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(BucketPolicyModal)

View File

@@ -0,0 +1,44 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import * as actionsBuckets from "./actions"
export const BucketSearch = ({ onChange }) => (
<div
className="input-group ig-dark ig-left ig-search"
style={{ display: "block" }}
>
<input
className="ig-text"
type="text"
onChange={e => onChange(e.target.value)}
placeholder="Search Buckets..."
/>
<i className="ig-helpers" />
</div>
)
const mapDispatchToProps = dispatch => {
return {
onChange: filter => {
dispatch(actionsBuckets.setFilter(filter))
}
}
}
export default connect(undefined, mapDispatchToProps)(BucketSearch)

View File

@@ -0,0 +1,90 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { connect } from "react-redux"
import { Modal, ModalBody } from "react-bootstrap"
import * as actionsBuckets from "./actions"
export class MakeBucketModal extends React.Component {
constructor(props) {
super(props)
this.state = {
bucketName: ""
}
}
onSubmit(e) {
e.preventDefault()
const { makeBucket } = this.props
const bucket = this.state.bucketName
if (bucket) {
makeBucket(bucket)
this.hideModal()
}
}
hideModal() {
this.setState({
bucketName: ""
})
this.props.hideMakeBucketModal()
}
render() {
const { showMakeBucketModal } = this.props
return (
<Modal
className="modal-create-bucket"
bsSize="small"
animation={false}
show={showMakeBucketModal}
onHide={this.hideModal.bind(this)}
>
<button className="close close-alt" onClick={this.hideModal.bind(this)}>
<span>×</span>
</button>
<ModalBody>
<form onSubmit={this.onSubmit.bind(this)}>
<div className="input-group">
<input
className="ig-text"
type="text"
placeholder="Bucket Name"
value={this.state.bucketName}
onChange={e => this.setState({ bucketName: e.target.value })}
autoFocus
/>
<i className="ig-helpers" />
</div>
</form>
</ModalBody>
</Modal>
)
}
}
const mapStateToProps = state => {
return {
showMakeBucketModal: state.buckets.showMakeBucketModal
}
}
const mapDispatchToProps = dispatch => {
return {
makeBucket: bucket => dispatch(actionsBuckets.makeBucket(bucket)),
hideMakeBucketModal: () => dispatch(actionsBuckets.hideMakeBucketModal())
}
}
export default connect(mapStateToProps, mapDispatchToProps)(MakeBucketModal)

View File

@@ -0,0 +1,96 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { READ_ONLY, WRITE_ONLY, READ_WRITE, NONE } from '../constants'
import React from "react"
import { connect } from "react-redux"
import classnames from "classnames"
import * as actionsBuckets from "./actions"
import * as actionsAlert from "../alert/actions"
import web from "../web"
export class Policy extends React.Component {
removePolicy(e) {
e.preventDefault()
const {currentBucket, prefix, fetchPolicies, showAlert} = this.props
web.
SetBucketPolicy({
bucketName: currentBucket,
prefix: prefix,
policy: 'none'
})
.then(() => {
fetchPolicies(currentBucket)
})
.catch(e => showAlert('danger', e.message))
}
render() {
const {policy, prefix} = this.props
let newPrefix = prefix
if (newPrefix === '')
newPrefix = '*'
if (policy === NONE) {
return <noscript />
} else {
return (
<div className="pmb-list">
<div className="pmbl-item">
{ newPrefix }
</div>
<div className="pmbl-item">
<select className="form-control"
disabled
value={ policy }>
<option value={ READ_ONLY }>
Read Only
</option>
<option value={ WRITE_ONLY }>
Write Only
</option>
<option value={ READ_WRITE }>
Read and Write
</option>
</select>
</div>
<div className="pmbl-item">
<button className="btn btn-block btn-danger" onClick={ this.removePolicy.bind(this) }>
Remove
</button>
</div>
</div>
)
}
}
}
const mapStateToProps = state => {
return {
currentBucket: state.buckets.currentBucket
}
}
const mapDispatchToProps = dispatch => {
return {
fetchPolicies: bucket => dispatch(actionsBuckets.fetchPolicies(bucket)),
showAlert: (type, message) =>
dispatch(actionsAlert.set({ type: type, message: message }))
}
}
export default connect(mapStateToProps, mapDispatchToProps)(Policy)

View File

@@ -0,0 +1,115 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { READ_ONLY, WRITE_ONLY, READ_WRITE } from '../constants'
import React from "react"
import { connect } from "react-redux"
import classnames from "classnames"
import * as actionsBuckets from "./actions"
import * as actionsAlert from "../alert/actions"
import web from "../web"
export class PolicyInput extends React.Component {
componentDidMount() {
const { currentBucket, fetchPolicies } = this.props
fetchPolicies(currentBucket)
}
componentWillUnmount() {
const { setPolicies } = this.props
setPolicies([])
}
handlePolicySubmit(e) {
e.preventDefault()
const { currentBucket, fetchPolicies, showAlert } = this.props
if (this.prefix.value === "*")
this.prefix.value = ""
let policyAlreadyExists = this.props.policies.some(
elem => this.prefix.value === elem.prefix && this.policy.value === elem.policy
)
if (policyAlreadyExists) {
showAlert("danger", "Policy for this prefix already exists.")
return
}
web.
SetBucketPolicy({
bucketName: currentBucket,
prefix: this.prefix.value,
policy: this.policy.value
})
.then(() => {
fetchPolicies(currentBucket)
this.prefix.value = ''
})
.catch(e => showAlert("danger", e.message))
}
render() {
return (
<header className="pmb-list">
<div className="pmbl-item">
<input
type="text"
ref={ prefix => this.prefix = prefix }
className="form-control"
placeholder="Prefix"
/>
</div>
<div className="pmbl-item">
<select ref={ policy => this.policy = policy } className="form-control">
<option value={ READ_ONLY }>
Read Only
</option>
<option value={ WRITE_ONLY }>
Write Only
</option>
<option value={ READ_WRITE }>
Read and Write
</option>
</select>
</div>
<div className="pmbl-item">
<button className="btn btn-block btn-primary" onClick={ this.handlePolicySubmit.bind(this) }>
Add
</button>
</div>
</header>
)
}
}
const mapStateToProps = state => {
return {
currentBucket: state.buckets.currentBucket,
policies: state.buckets.policies
}
}
const mapDispatchToProps = dispatch => {
return {
fetchPolicies: bucket => dispatch(actionsBuckets.fetchPolicies(bucket)),
setPolicies: policies => dispatch(actionsBuckets.setPolicies(policies)),
showAlert: (type, message) =>
dispatch(actionsAlert.set({ type: type, message: message }))
}
}
export default connect(mapStateToProps, mapDispatchToProps)(PolicyInput)

View File

@@ -0,0 +1,39 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { Bucket } from "../Bucket"
describe("Bucket", () => {
it("should render without crashing", () => {
shallow(<Bucket />)
})
it("should call selectBucket when clicked", () => {
const selectBucket = jest.fn()
const wrapper = shallow(
<Bucket bucket={"test"} selectBucket={selectBucket} />
)
wrapper.find("li").simulate("click", { preventDefault: jest.fn() })
expect(selectBucket).toHaveBeenCalledWith("test")
})
it("should highlight the selected bucket", () => {
const wrapper = shallow(<Bucket bucket={"test"} isActive={true} />)
expect(wrapper.find("li").hasClass("active")).toBeTruthy()
})
})

View File

@@ -0,0 +1,52 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import BucketContainer from "../BucketContainer"
import configureStore from "redux-mock-store"
const mockStore = configureStore()
describe("BucketContainer", () => {
let store
beforeEach(() => {
store = mockStore({
buckets: {
currentBucket: "Test"
}
})
store.dispatch = jest.fn()
})
it("should render without crashing", () => {
shallow(<BucketContainer store={store}/>)
})
it('maps state and dispatch to props', () => {
const wrapper = shallow(<BucketContainer store={store}/>)
expect(wrapper.props()).toEqual(expect.objectContaining({
isActive: expect.any(Boolean),
selectBucket: expect.any(Function)
}))
})
it('maps selectBucket to dispatch action', () => {
const wrapper = shallow(<BucketContainer store={store}/>)
wrapper.props().selectBucket()
expect(store.dispatch).toHaveBeenCalled()
})
})

View File

@@ -0,0 +1,62 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { BucketDropdown } from "../BucketDropdown"
describe("BucketDropdown", () => {
it("should render without crashing", () => {
shallow(<BucketDropdown />)
})
it("should call toggleDropdown on dropdown toggle", () => {
const spy = jest.spyOn(BucketDropdown.prototype, 'toggleDropdown')
const wrapper = shallow(
<BucketDropdown />
)
wrapper
.find("Uncontrolled(Dropdown)")
.simulate("toggle")
expect(spy).toHaveBeenCalled()
spy.mockReset()
spy.mockRestore()
})
it("should call showBucketPolicy when Edit Policy link is clicked", () => {
const showBucketPolicy = jest.fn()
const wrapper = shallow(
<BucketDropdown showBucketPolicy={showBucketPolicy} />
)
wrapper
.find("li a")
.at(0)
.simulate("click", { stopPropagation: jest.fn() })
expect(showBucketPolicy).toHaveBeenCalled()
})
it("should call deleteBucket when Delete link is clicked", () => {
const deleteBucket = jest.fn()
const wrapper = shallow(
<BucketDropdown bucket={"test"} deleteBucket={deleteBucket} />
)
wrapper
.find("li a")
.at(1)
.simulate("click", { stopPropagation: jest.fn() })
expect(deleteBucket).toHaveBeenCalledWith("test")
})
})

View File

@@ -0,0 +1,57 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import history from "../../history"
import { BucketList } from "../BucketList"
jest.mock("../../web", () => ({
LoggedIn: jest
.fn(() => false)
.mockReturnValueOnce(true)
.mockReturnValueOnce(true)
}))
describe("BucketList", () => {
it("should render without crashing", () => {
const fetchBuckets = jest.fn()
shallow(<BucketList visibleBuckets={[]} fetchBuckets={fetchBuckets} />)
})
it("should call fetchBuckets before component is mounted", () => {
const fetchBuckets = jest.fn()
const wrapper = shallow(
<BucketList visibleBuckets={[]} fetchBuckets={fetchBuckets} />
)
expect(fetchBuckets).toHaveBeenCalled()
})
it("should call setBucketList and selectBucket before component is mounted when the user has not loggedIn", () => {
const setBucketList = jest.fn()
const selectBucket = jest.fn()
history.push("/bk1/pre1")
const wrapper = shallow(
<BucketList
visibleBuckets={[]}
setBucketList={setBucketList}
selectBucket={selectBucket}
/>
)
expect(setBucketList).toHaveBeenCalledWith(["bk1"])
expect(selectBucket).toHaveBeenCalledWith("bk1", "pre1")
})
})

View File

@@ -0,0 +1,43 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { BucketPolicyModal } from "../BucketPolicyModal"
import { READ_ONLY, WRITE_ONLY, READ_WRITE } from "../../constants"
describe("BucketPolicyModal", () => {
it("should render without crashing", () => {
shallow(<BucketPolicyModal policies={[]}/>)
})
it("should call hideBucketPolicy when close button is clicked", () => {
const hideBucketPolicy = jest.fn()
const wrapper = shallow(
<BucketPolicyModal hideBucketPolicy={hideBucketPolicy} policies={[]} />
)
wrapper.find("button").simulate("click")
expect(hideBucketPolicy).toHaveBeenCalled()
})
it("should include the PolicyInput and Policy components when there are any policies", () => {
const wrapper = shallow(
<BucketPolicyModal policies={ [{prefix: "test", policy: READ_ONLY}] } />
)
expect(wrapper.find("Connect(PolicyInput)").length).toBe(1)
expect(wrapper.find("Connect(Policy)").length).toBe(1)
})
})

View File

@@ -0,0 +1,32 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow } from "enzyme"
import { BucketSearch } from "../BucketSearch"
describe("BucketSearch", () => {
it("should render without crashing", () => {
shallow(<BucketSearch />)
})
it("should call onChange with search text", () => {
const onChange = jest.fn()
const wrapper = shallow(<BucketSearch onChange={onChange} />)
wrapper.find("input").simulate("change", { target: { value: "test" } })
expect(onChange).toHaveBeenCalledWith("test")
})
})

View File

@@ -0,0 +1,80 @@
/*
* MinIO Cloud Storage (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react"
import { shallow, mount } from "enzyme"
import { MakeBucketModal } from "../MakeBucketModal"
describe("MakeBucketModal", () => {
it("should render without crashing", () => {
shallow(<MakeBucketModal />)
})
it("should call hideMakeBucketModal when close button is clicked", () => {
const hideMakeBucketModal = jest.fn()
const wrapper = shallow(
<MakeBucketModal hideMakeBucketModal={hideMakeBucketModal} />
)
wrapper.find("button").simulate("click")
expect(hideMakeBucketModal).toHaveBeenCalled()
})
it("bucketName should be cleared before hiding the modal", () => {
const hideMakeBucketModal = jest.fn()
const wrapper = shallow(
<MakeBucketModal hideMakeBucketModal={hideMakeBucketModal} />
)
wrapper.find("input").simulate("change", {
target: { value: "test" }
})
expect(wrapper.state("bucketName")).toBe("test")
wrapper.find("button").simulate("click")
expect(wrapper.state("bucketName")).toBe("")
})
it("should call makeBucket when the form is submitted", () => {
const makeBucket = jest.fn()
const hideMakeBucketModal = jest.fn()
const wrapper = shallow(
<MakeBucketModal
makeBucket={makeBucket}
hideMakeBucketModal={hideMakeBucketModal}
/>
)
wrapper.find("input").simulate("change", {
target: { value: "test" }
})
wrapper.find("form").simulate("submit", { preventDefault: jest.fn() })
expect(makeBucket).toHaveBeenCalledWith("test")
})
it("should call hideMakeBucketModal and clear bucketName after the form is submited", () => {
const makeBucket = jest.fn()
const hideMakeBucketModal = jest.fn()
const wrapper = shallow(
<MakeBucketModal
makeBucket={makeBucket}
hideMakeBucketModal={hideMakeBucketModal}
/>
)
wrapper.find("input").simulate("change", {
target: { value: "test" }
})
wrapper.find("form").simulate("submit", { preventDefault: jest.fn() })
expect(hideMakeBucketModal).toHaveBeenCalled()
expect(wrapper.state("bucketName")).toBe("")
})
})

Some files were not shown because too many files have changed in this diff Show More