mirrors/minio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate config to KV data format (#8392)

Browse Source 
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
This commit is contained in:
Harshavardhana
2019-10-22 22:59:13 -07:00
committed by kannappanr
parent 8836d57e3c
commit ee4a6a823d
185 changed files with 8228 additions and 3597 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/sts/README.md
View File

@@ -34,7 +34,7 @@ Make sure we have followed the previous step and configured each software indepe
```
export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=minio123
export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
minio server /mnt/data
```
@@ -46,7 +46,7 @@ Make sure we have followed the previous step and configured each software indepe
```
export MINIO_ACCESS_KEY=aws_access_key
export MINIO_SECRET_KEY=aws_secret_key
export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
export MINIO_ETCD_ENDPOINTS=http://localhost:2379
minio gateway s3
```

24
docs/sts/client-grants.md
View File

@@ -93,25 +93,15 @@ http://minio.cluster:9000?Action=AssumeRoleWithClientGrants&DurationSeconds=3600
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
$ export MINIO_IAM_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
$ export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
$ export MINIO_POLICY_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
$ minio server /mnt/export
$ mc admin config get myminio
...
{
"openid": {
"jwks": {
"url": "https://localhost:9443/oauth2/jwks"
}
}
"policy": {
"opa": {
"url": "http://localhost:8181/v1/data/httpapi/authz",
"authToken": ""
}
}
}
$ mc admin config get myminio identity_openid
identity_openid config_url="https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration" state="on"
$ mc admin config get myminio policy_opa
policy_opa auth_token="" state="on" url="http://localhost:8181/v1/data/httpapi/authz"
```
Testing with an example

69
docs/sts/keycloak.md Normal file
View File

@@ -0,0 +1,69 @@
# Keycloak Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
Keycloak is an open source Identity and Access Management solution aimed at modern applications and services, this document covers configuring Keycloak to be used as an identity provider for MinIO server STS API.
## 1. Prerequisites
- JAVA 1.8 and above installed
- Download and start Keycloak server by following the [installation guide](https://www.keycloak.org/docs/latest/getting_started/index.html) (finish upto section 3.4)
## 2. Configure Keycloak
- Go to Users -> Click on the user -> Attribute, add a new attribute `Key` is `policy`, `Value` is name of the policy in minio (ex: `readwrite`). Click Add and then Save.
- Go to Clients -> Click on `account` -> Settings, set `Valid Redirect URIs` to `*`, expand `Advanced Settings` and set `Access Token Lifespan` to `1 Hours`, then Save.
- Go to Clients -> Client on `account` -> Mappers -> Create, `Name` can be any text, `Mapper Type` is `User Attribute`, `User Attribute` is `policy`, `Token Claim Name` is `policy`, `Claim JSON Type` is `string`, then Save.
- Open http://localhost:8080/auth/realms/demo/.well-known/openid-configuration and see if it has `authorization_endpoint` and `jwks_uri`
## 3. Configure MinIO
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ minio server /mnt/export
```
Set `identity_openid` config and restart MinIO
```
~ mc admin config set myminio identity_openid config_url="http://localhost:8080/auth/realms/demo/.well-known/openid-configuration" state="on"
```
Once successfully set restart the MinIO instance.
```
mc admin service restart myminio
```
## 4. Using WebIdentiy API
Client ID and Client Secret can be found by clicking any of the clients listed [here](http://localhost:8080/auth/admin/master/console/#/realms/demo/clients). If you have followed the above steps docs, the default Client ID will be `account` and Client Secret can be found under `Credentials` tab.
```
$ go run web-identity.go -cid account -csec e61cb282-745b-4113-bece-29b921c735f0 -auth-ep http://localhost:8080/auth/realms/demo/protocol/openid-connect/auth -token-ep http://localhost:8080/auth/realms/demo/protocol/openid-connect/token -port 8888
2018/12/26 17:49:36 listening on http://localhost:8888/
```
This will open the login page of keycloak, upon successful login, STS credentials will be printed on the screen, for example
```
##### Credentials
{
"accessKey": "6N2BALX7ELO827DXS3GK",
"secretKey": "23JKqAD+um8ObHqzfIh+bfqwG9V8qs9tFY6MqeFR",
"expiration": "2019-10-01T07:22:34Z",
"sessionToken": "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiI2TjJCQUxYN0VMTzgyN0RYUzNHSyIsImFjciI6IjAiLCJhdWQiOiJhY2NvdW50IiwiYXV0aF90aW1lIjoxNTY5OTEwNTUyLCJhenAiOiJhY2NvdW50IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJleHAiOjE1Njk5MTQ1NTQsImlhdCI6MTU2OTkxMDk1NCwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL2F1dGgvcmVhbG1zL2RlbW8iLCJqdGkiOiJkOTk4YTBlZS01NDk2LTQ4OWYtYWJlMi00ZWE5MjJiZDlhYWYiLCJuYmYiOjAsInBvbGljeSI6InJlYWR3cml0ZSIsInByZWZlcnJlZF91c2VybmFtZSI6Im5ld3VzZXIxIiwic2Vzc2lvbl9zdGF0ZSI6IjJiYTAyYTI2LWE5MTUtNDUxNC04M2M1LWE0YjgwYjc4ZTgxNyIsInN1YiI6IjY4ZmMzODVhLTA5MjItNGQyMS04N2U5LTZkZTdhYjA3Njc2NSIsInR5cCI6IklEIn0._UG_-ZHgwdRnsp0gFdwChb7VlbPs-Gr_RNUz9EV7TggCD59qjCFAKjNrVHfOSVkKvYEMe0PvwfRKjnJl3A_mBA"
}
```
These credentials can now be used to perform MinIO API operations.
## 5. Using MinIO Browser
- Open MinIO url on the browser, for example `http://localhost:9000`
- Click on `Log in with OpenID`
- Provide `Client ID` and press ENTER
- Now the user will be redirected to the Keycloak login page, upon successful login the user will be redirected to MinIO page and logged in automatically
## Explore Further
- [MinIO STS Quickstart Guide](https://docs.min.io/docs/minio-sts-quickstart-guide)
- [The MinIO documentation website](https://docs.min.io)

43
docs/sts/ldap.md
View File

@@ -40,26 +40,31 @@ MinIO can be configured to find the groups of a user from AD/LDAP by specifying
LDAP is configured via the following environment variables:
| Variable | Required? | Purpose |
|----------------------------------------------|---------------------------|--------------------------------------------------------|
| **MINIO_IDENTITY_LDAP_SERVER_ADDR** | **YES** | AD/LDAP server address |
| **MINIO_IDENTITY_LDAP_USERNAME_FORMAT** | **YES** | Format of full username DN |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN** | **NO** | Base DN in AD/LDAP hierarchy to use in search requests |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER** | **NO** | Search filter to find groups of a user |
| **MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE** | **NO** | Attribute of search results to use as group name |
| **MINIO_IDENTITY_LDAP_STS_EXPIRY** | **NO** (default: "1h") | STS credentials validity duration |
| Variable | Required? | Purpose |
|----------------------------------------------|-------------------------|-------------------------------------------------------------------------|
| **MINIO_IDENTITY_LDAP_STATE** | **YES** | Enable or disable ldap identity |
| **MINIO_IDENTITY_LDAP_SERVER_ADDR** | **YES** | AD/LDAP server address |
| **MINIO_IDENTITY_LDAP_USERNAME_FORMAT** | **YES** | Format of full username DN |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN** | **NO** | Base DN in AD/LDAP hierarchy to use in search requests |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER** | **NO** | Search filter to find groups of a user |
| **MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE** | **NO** | Attribute of search results to use as group name |
| **MINIO_IDENTITY_LDAP_STS_EXPIRY** | **NO** (default: "1h") | STS credentials validity duration |
| **MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY** | **NO** (default: "off") | Set this to 'on', to disable client verification of server certificates |
Please note that MinIO will only access the AD/LDAP server over TLS. If a self-signed certificate is being used, the certificate can be added to MinIO's certificates directory, so it can be trusted by the server.
An example setup for development or experimentation:
``` shell
export MINIO_IDENTITY_LDAP_STATE="on"
export MINIO_IDENTITY_LDAP_SERVER_ADDR=myldapserver.com:636
export MINIO_IDENTITY_LDAP_USERNAME_FORMAT="uid=${username},cn=accounts,dc=myldapserver,dc=com"
export MINIO_IDENTITY_LDAP_USERNAME_FORMAT="uid={username},cn=accounts,dc=myldapserver,dc=com"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN="dc=myldapserver,dc=com"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member=${usernamedn}))"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member={usernamedn})$)"
export MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE="cn"
export MINIO_IDENTITY_LDAP_STS_EXPIRY=60
export MINIO_IDENTITY_LDAP_STS_EXPIRY=60h
export MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY="on"
```
### Variable substitution in AD/LDAP configuration strings
@@ -99,18 +104,20 @@ member: CN=John,CN=Users,DC=minioad,DC=local
...
```
The lines with "..." represent skipped content not shown here from brevity. Based on the output above, we see that the username format variable looks like `cn=${username},cn=users,dc=minioad,dc=local`.
The lines with "..." represent skipped content not shown here from brevity. Based on the output above, we see that the username format variable looks like `cn={username},cn=users,dc=minioad,dc=local`.
The group search filter looks like `(&(objectclass=group)(member=${usernamedn}))` and the group name attribute is clearly `cn`.
The group search filter looks like `(&(objectclass=group)(member={usernamedn}))` and the group name attribute is clearly `cn`.
Thus the key configuration parameters look like:
```
MINIO_IDENTITY_LDAP_STATE="on"
MINIO_IDENTITY_LDAP_SERVER_ADDR='my.ldap-active-dir-server.com:636'
MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn=${username},cn=users,dc=minioad,dc=local'
MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn={username},cn=users,dc=minioad,dc=local'
MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN='dc=minioad,dc=local'
MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member=${usernamedn}))'
MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member={usernamedn}))'
MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE='cn'
MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY="on"
```
## Managing User/Group Access Policy
@@ -211,10 +218,11 @@ http://minio.cluster:9000?Action=AssumeRoleWithLDAPIdentity&LDAPUsername=foouser
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ export MINIO_IDENTITY_LDAP_STATE="on"
$ export MINIO_IDENTITY_LDAP_SERVER_ADDR='my.ldap-active-dir-server.com:636'
$ export MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn=${username},cn=users,dc=minioad,dc=local'
$ export MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn={username},cn=users,dc=minioad,dc=local'
$ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN='dc=minioad,dc=local'
$ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member=${usernamedn}))'
$ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member={usernamedn}))'
$ export MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE='cn'
$ minio server ~/test
```
@@ -230,4 +238,3 @@ $ go run ldap.go -u foouser -p foopassword
"sessionToken": "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJOVUlCT1JaWVRWMkhHMkJNUlNYUiIsImF1ZCI6IlBvRWdYUDZ1Vk80NUlzRU5SbmdEWGo1QXU1WWEiLCJhenAiOiJQb0VnWFA2dVZPNDVJc0VOUm5nRFhqNUF1NVlhIiwiZXhwIjoxNTM0ODk2NjI5LCJpYXQiOjE1MzQ4OTMwMjksImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0Ojk0NDMvb2F1dGgyL3Rva2VuIiwianRpIjoiNjY2OTZjZTctN2U1Ny00ZjU5LWI0MWQtM2E1YTMzZGZiNjA4In0.eJONnVaSVHypiXKEARSMnSKgr-2mlC2Sr4fEGJitLcJF_at3LeNdTHv0_oHsv6ZZA3zueVGgFlVXMlREgr9LXA"
}
```

4
docs/sts/opa.md
View File

@@ -61,9 +61,9 @@ curl -X PUT --data-binary @putobject.rego \
```
### 4. Setup MinIO with OPA
MinIO server expects environment variable for OPA http API url as `MINIO_IAM_OPA_URL`, this environment variable takes a single entry.
MinIO server expects environment variable for OPA http API url as `MINIO_POLICY_OPA_URL`, this environment variable takes a single entry.
```
export MINIO_IAM_OPA_URL=http://localhost:8181/v1/data/httpapi/authz/allow
export MINIO_POLICY_OPA_URL=http://localhost:8181/v1/data/httpapi/authz/allow
minio server /mnt/data
```

4
docs/sts/sts.env
View File

@@ -1,5 +1,5 @@
export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=minio123
export MINIO_IAM_JWKS_URL=http://localhost:9763/oauth2/jwks
export MINIO_IAM_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
export MINIO_POLICY_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
export MINIO_ETCD_ENDPOINTS=http://localhost:2379

11
docs/sts/web-identity.go
View File

@@ -78,6 +78,7 @@ var (
tokenEndpoint string
clientID string
clientSecret string
port int
)
func init() {
@@ -86,6 +87,7 @@ func init() {
flag.StringVar(&tokenEndpoint, "token-ep", googleOAuth2.Endpoint.TokenURL, "Token endpoint")
flag.StringVar(&clientID, "cid", "", "Client ID")
flag.StringVar(&clientSecret, "csec", "", "Client secret")
flag.IntVar(&port, "port", 8080, "Port")
}
func main() {
@@ -104,7 +106,7 @@ func main() {
AuthURL: authEndpoint,
TokenURL: tokenEndpoint,
},
RedirectURL: "http://localhost:8080/oauth2/callback",
RedirectURL: fmt.Sprintf("http://localhost:%v/oauth2/callback", port),
Scopes: []string{"openid", "profile", "email"},
}
@@ -140,7 +142,7 @@ func main() {
}
u.RawQuery = v.Encode()
req, err := http.NewRequest("POST", u.String(), nil)
req, err := http.NewRequest(http.MethodPost, u.String(), nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
@@ -173,6 +175,7 @@ func main() {
}
})
log.Printf("listening on http://%s/", "localhost:8080")
log.Fatal(http.ListenAndServe("localhost:8080", nil))
address := fmt.Sprintf("localhost:%v", port)
log.Printf("listening on http://%s/", address)
log.Fatal(http.ListenAndServe(address, nil))
}

36
docs/sts/web-identity.md
View File

@@ -14,6 +14,7 @@
- [Sample Response](#sample-response)
- [Testing](#testing)
- [Authorization Flow](#authorization-flow)
- [MinIO Browser](#minio-browser)
## Introduction
@@ -92,18 +93,11 @@ http://minio.cluster:9000?Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ export MINIO_IAM_JWKS_URL=https://www.googleapis.com/oauth2/v3/certs
$ export MINIO_IDENTITY_OPENID_CONFIG_URL=https://accounts.google.com/.well-known/openid-configuration
$ minio server /mnt/export
$ mc admin config get myminio
...
{
"openid": {
"jwks": {
"url": "https://www.googleapis.com/oauth2/v3/certs"
}
}
}
$ mc admin config get myminio identity_openid
identity_openid config_url="https://accounts.google.com/.well-known/openid-configuration" state="on"
```
Testing with an example
@@ -121,3 +115,25 @@ $ go run web-identity.go -cid 204367807228-ok7601k6gj1pgge7m09h7d79co8p35xx.apps
- Using the access token the callback handler further talks to Google OAuth2 Token URL to obtain an JWT id_token.
- Once obtained the JWT id_token is further sent to STS endpoint i.e MinIO to retrive temporary credentials.
- Temporary credentials are displayed on the browser upon successful retrieval.
## MinIO Browser
To support WebIdentity login on MinIO Browser
1. Set openid configuration and restart MinIO
```
mc admin config set myminio identity_openid jwks_url="<JWKS_URL>" config_url="<CONFIG_URL>"
mc admin service restart myminio
```
Sample URLs for Keycloak are
`config_url` - `http://localhost:8080/auth/realms/demo/.well-known/openid-configuration`,
`jwks_url` - `http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs`
JWT token returned by the Identity Provider should include a custom claim for the policy, this is required to create a STS user in MinIO. The name of the custom claim could be either `policy` or `<NAMESPACE_PREFIX>policy`.
If there is no namespace then `policy_claim_prefix` can be ingored. For example if the custom claim name is `https://min.io/policy` then, `policy_claim_prefix` should be set as `https://min.io/`
2. Open MinIO Browser and click `Log in with OpenID`
3. Enter the `Client ID` obtained from Identity Provider and press ENTER
4. The user will be redirected to the Identity Provider login page
5. Upon successful login on Identity Provider page the user will be automatically logged into MinIO Browser

6
docs/sts/wso2.md
View File

@@ -66,10 +66,10 @@ Using the above `access_token` we can perform an STS request to MinIO to get tem
**We recommend setting `policy` as a custom claim for the JWT service provider follow [here](https://docs.wso2.com/display/IS550/Configuring+Claims+for+a+Service+Provider) and [here](https://docs.wso2.com/display/IS550/Handling+Custom+Claims+with+the+JWT+Bearer+Grant+Type) for relevant docs on how to configure claims for a service provider.**
### 5. Setup MinIO with JWKS URL
MinIO server expects environment variable for JWKS url as `MINIO_IAM_JWKS_URL`, this environment variable takes a single entry.
### 5. Setup MinIO with OpenID configuration URL
MinIO server expects environment variable for OpenID configuration url as `MINIO_IDENTITY_OPENID_CONFIG_URL`, this environment variable takes a single entry.
```
export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
minio server /mnt/data
```