12. Managing External Repositories¶
In the previous chapter we looked at how one can snapshot and clone the entire application stack. Snapshots are stored on the same disks from which the storage for the PersistentVolume is allocated. Over time, as more snapshots are taken, they start consuming more and more disk space on the primary storage device. To avoid this, yet preserve as many snapshots as needed, Robin CNS can attach an external secondary storage repository onto which older snapshots are copied. This way, a user can setup a policy to retain (for example) only the last 2 snapshots on the primary storage device, and copy the older snapshots to a secondary storage repository.
Recall that Robin CNS is a highly sophisticated product, in that snapshots are not just snapshots of the PersistentVolumeClaims, but they capture all the relevant Kubernetes resources that make up that application. Hence with Robin CNS users can:
Backup applications and data to an external storage repository such as AWS S3, Google Cloud Storage etc.
Make applications and data portable across Kubernetes clusters spanning on-prem, public and hybrid cloud environments.
This is done by registering a common external Repository (or repo in short) that resides outside the Robin CNS cluster. This repository is typically an object storage system such as an Amazon S3 bucket, Google Cloud Storage (GCS) bucket or Azure Blob Storage.
The workflow to register and use the repository (abbreviated to repo
) is as follows:
Create / register a Repo by passing access credentials to it
Attach the repo to an app
Backup the app and data to the attached repo
Register the same repo to a different Kubernetes cluster
Export an app backup from the first cluster
Import the app backup into the second cluster
Browse the contents of the repo on the second cluster, pick an app, and launch it
Topics covered in this chapter:
|
Register a new external storage repository (S3, GCS, …) |
|
Unregister a storage repo |
|
List all storage repos |
|
Show details about a specific storage repo |
|
Show the contents of a storage repo |
|
Purge/delete entries in a storage repo |
|
Reclaim storage in a storage repo previously used by deleted backups |
|
Show status of transfers to and from a storage repo |
|
Share a storage repo with one or more tenants |
|
Stop sharing a storage repo with one or more tenants |
|
Share a storage repo with one or more users |
|
Stop sharing a repo with one or more users |
|
List information about tenants and users a storage repo has been shared with |
12.1. Register a repo¶
A repo is registered with this command:
# robin repo register <reponame> s3|gcs://bucket[/path/to/folder] <credentials> <readwrite | readonly>
|
A name that would be assigned to the repo |
|
Type of the repo. It can be one of these values: ‘s3’ representing AWS S3 and gcs representing Google Cloud Storage |
|
S3/GCS bucket name and a folder within that bucket into which the application snapshots would be pushed (backed up) |
|
Path to a JSON file with credentials to access the repo. The format for different repo types is shown below |
|
Specifies the permissions with which the repo is registered with. The ‘readwrite’ permission creates the bucket and the folder hierarchy in the repo if it doesn’t already exist. This assumes that the supplied credentials have bucket creation privileges. With the ‘readonly’ permission the repo is registered in read-only mode - it can only be used to pull already pushed application snapshots from the repo. New snapshots cannot be written to this repo |
Format of the credentials file for an AWS S3 type repo
The format of the json file passed via the <credentials>
parameter
for AWS S3 type repos is:
{
"aws_access_key_id" : "AKIAIC9YAL8TKCSEO43A",
"aws_secret_access_key" : "66HXkA3mjQEbmrUo1Aw02bcdYEWjsuSPHvLEMNfZ",
"end_point": "s3.amazonaws.com",
}
In addition to the mandatory parameters detailed above, the following parameters may also be utilized within the credentials JSON file:
region: <region_name>
- A string specifying the region in which to create the bucket. The default value is ‘us-east-1’. Note this option is not needed if the bucket already exists.tls: [yes|no]
- A boolean value indicating whether or not the HTTPs protocol should be used or not.validate_certs: [yes|no]
- A boolean value indicating whether or not certificate validation should be performed or not.ca_file: <file_path>
- A string into the path at which to find the CA chain certificate to verify the S3 server certificate. Note this certificate is only used when HTTPs protocol usage is enabled.
Example:
# robin repo register mybackups s3://db-backup-bucket/mybackups aws.json readwrite
AWS Access Key and Secret Keys can be obtained this way:
Log in to your AWS Management Console.
Click on your user name at the top right of the page.
Click on the Security Credentials link from the drop-down menu.
Find the Access Credentials section, and copy the latest Access Key ID.
Click on the Show link in the same row, and copy the Secret Access Key.
Format of credentials file for a Google Cloud Storage (GCS) type repo
The format of the JSON file passed via the <credentials>
parameter for Google Cloud Storage (GCS) type repos is:
{
"type": "service_account",
"project_id": "rock-range-805623",
"privbte_key_id": "c9fde8c819735439248147457629895ebbcc1f21",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgpqkkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDXlyNg3997PJlA\nyS9doXSfX5lEpDgmk7nATyOrKQkl1D3/bWvoFMR372i0WKopj5FvLNY42jrcRzjF\nTlbJuwP4WAU652ss3qEvnUgd+mD7CcjOWd9bAA6vPJAJDo/TtTleZNQWd96y2WbT\nH/BqI9UCxdWNfsuYHQtpVViPdviizU3AxFtD6NTN/KmUX9mlM6RuF2RXAc5N/p4X\nhbeSV+rSfEgb9PA2U+fwobGzeR97V4SwK15btPpSN5twf0Wy49zGQLONFcmfwhRc\n+3r3gpPfqb+cb1xWiiD5fgO4yWYcWQXJJgUumVtVZxikc6k+9Vbko9zw20NNzRgp\nKfZdtm1jAgMBAAECggEABx+l2h4bntFWSQc3yu26UkfQ4y0/37pe4WVcCtxMwpS3\nUEbV+7Gv+mbYwVjKqpvlVNSY3YD4f+3UiOR5RIzK6UpTRep+ppoPzGh4iREMYk3k\n9PiQQkwCsDSil3IE65xp3F4die6FC3jWNFdSVNeBQtmxoD7H0GtpRJ4+0mK+fXbW\nr5f6O3WES4nOTNRonOdg9bIJJKklt3GSjtd1X5JWfGU53sbrksvy31+hL71pXCQn\nUZlkoilc3KYGnesd3KwIpxX9Pi5TldNUWuSnibvgXnDjM05PuQ0YI9VEfVR+eDfI\nBZAw2b/dmCWyU3QVAMtRbhwISmbxB33rZ133gTVN/QKBgQD5ZTZSnf7/b0YdKcAD\nP3bevN3/0s9EXO+D62MTRdrNeyb46hAcbNHmzAvgqtB74bQxiF5hs6RbmTmUpVpL\nSyCl6lqf3eMzP2gdECRLmkeZKpiWtfbPNbTefGVvbvWG6vi3E6+b/5cKPVHMLSV6\n4LIbqpVyWFWM905MXU9cgFgJDwKBgQDdTL5Yl3G/91HnyFieQ9pju6C9v1S3277k\nN10hcmldkJWludgT3WzDsNmbcAdCp+KOtQusf6KMKT5sd5ickMmAI3IZkzsgb7rS\n7AVBf3bpLG+3kOGJ4PVGKXfTEONw7hSo1UwLQxSZj1SMTtOd63pFcVea1hYQXifr\nADUrkwhObQKBgQCTb9pljTIrIEV7CCuTwChTv6QekSonaCnQ+19fDXUE9UFc9kMA\nCvUsVITRFSqbkhHXlp71c5Y+3J6x2e3/g/KRI7Lfv/WJXnrWc6yBZXveeOgscPaK\nQ13iCfiNndX5JQBXb+vo3iEU1Jt+3VGvCxc/3IjtSHuLEskfLCq2rMle0QKBgQCX\nsTJM0gGKT8KCCc/M9J/vez6Msmkk+mkYUGbzNVTKQQCDMCfQPhh+72vKY3lmlGP1\nBF7zKC5Iu0DB4xzmPU0SG/DBzS1bZ5r9V7GmmvPsk3wkrRgcheo65NPxBwOQdnIM\n5OCSW7H0LM58us/N0UG+Zhnx1cwb/h6ItIS90LSB5QKBgCwZPxhDhlKXGn3J8sLn\nuZ9WjFTBlabt8mksHpnE4xgj5BeOXee29CtaX+mPIcPEOO9ynsbizaSv7I2kBM+d\n9DZt9XiM6NcjrqtP7FCdgvZeQV0GIHjlqj2zH0UDxImGmENyHeXA1X2d0fZwzuJ+\n5dEZAfYh4QAnnPiAF5K9QFEi\n-----END PRIVATE KEY-----\n",
"client_email": "jamesbond-cloud-repo@rock-range-805623.iam.gserviceaccount.com",
"client_id": "1815195052297641083965",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/jamesbond-cloud-repo%40rock-range-805623.iam.gserviceaccount.com"
}
Example:
# robin repo register datadump gcs://datadump-bucket/dumps gcs.json readwrite
To download a GCS Key file from the Google Cloud Platform portal:
Open the IAM & Admin page in the GCP Console.
Select your project and click Continue.
In the left nav, click Service accounts.
Look for the service account for which you wish to create a key, click the More
:
button in that row, and then click Create key.Select a Key type and click Create.
More details can be found here.
Note
The ability to register a repo is subject to Role Based Access Control (RBAC). By default, only Cluster Administrators (users having the superadmin role) have permission to do so. See the section on User Management for details on how RBAC works.
Only one repo with a given bucket name and path can be registered in a Robin CNS cluster.
To provide some users with readonly access to a repo, share the
repo with them for the view
operation only. For users that
should have readwrite access to a repo, share the repo with them
for view
and push
operations. In order for this to work,
the repo must have been regsitered with readwrite permisison. See
Share a repo with tenants for details.
12.2. Unregister a repo¶
A registered repo can be unregistered by issuing the following command:
# robin repo unregister <reponame>
--force
|
A name that would be assigned to the repo |
|
Force removal of a repo regardless of backups cataloged |
Example:
# robin repo unregister demo-backup --wait
Job: 719 Name: StorageRepoRemove State: PROCESSED Error: 0
Job: 719 Name: StorageRepoRemove State: COMPLETED Error: 0
Note
The ability to unregister a repo is subject to Role Based Access Control (RBAC). By default, only Cluster Administrators (users having the superadmin role) have permission to do so. See the section on User Management for details on how RBAC works.
There are a number of conditions that will prevent a repo from being unregistered:
A repo cannot be unregistered if it has attached applications (see
robin app attach-repo
command). Use therobin app detach-repo
command to detach each app before running therobin repo unregister
command. See documentation forrobin repo detach
to understand what happens to an app’s data when it is detached from the repo.A repo cannot be unregistered if there are any backup entries in the repo’s catalog. All backup entries must be purged from the repo’s catalog before running the
robin repo unregister
command. Backup entries can be purged from the repo catalog only, or they can be purged from the catalog and the external repository. See documentation forrobin repo purge
androbin backup delete
for details.A repo cannot be unregistered if it is shared with any users (see
robin repo share
command). Use therobin repo unshare
command to remove any user shares before running therobin repo unregister
command.
Unregisters a repo from Robin such that it cannot be used until it is registered again.
End Point: /api/v3/robin_server/storage_repo/<repo_name>
Method: DELETE
URL Parameters: None
Data Parameters:
force: true
- Utilizing this parameter results in the forceful removal of a repo regardless of the backups cataloged.
Port: RCM Port (default value is 29442)
Headers:
Authorization: <auth_token>
: Authorization token to identify which user is sending the request. The token can be acquired from the login API.
Success Response Code: 202
Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error), 404 (Not Found Error), 400 (Invalid API Usage Error)
Example Response:
Output
{
"jobid":717
}
12.3. List repos¶
Repos registered with a Robin CNS cluster can be listed like this:
# robin repo list --full
|
Show additional information about each repo (e.g., tenant and user shares) |
Example 1:
# robin repo list
+----------+------+----------------------+--------------+-----------------+--------+-------------+
| Name | Type | Owner/Tenant | BackupTarget | Bucket | Path | Permissions |
+----------+------+----------------------+--------------+-----------------+--------+-------------+
| datadump | GCS | robin/Administrators | 1 | datadump-bucket | dumps/ | readwrite |
+----------+------+----------------------+--------------+-----------------+--------+-------------+
Example 2: Repo listing with additional information
# robin repo list --full
+----------+------+----------------------+--------------+-----------------------+-----------+-------------+
| Name | Type | Owner/Tenant | BackupTarget | Bucket | Path | Permissions |
+----------+------+----------------------+--------------+-----------------------+-----------+-------------+
| testrepo | GCS | robin/Administrators | 1 | testbucket-1029384756 | dev/test/ | readwrite |
+----------+------+----------------------+--------------+-----------------------+-----------+-------------+
Tenant Shares: ALL_TENANTS
User Shares:
t1:admin1: update, delete, push, scan, purge, reclaim, share
t1:all_tenant_users: view
Note
The ability to view repos is subject to Role Based Access Control (RBAC). Cluster Administrators (users having the superadmin role) are able to view all registered repos in a cluster. Cluster users (users having the user role), on the other hand, are only able to view repos that have been explicitly shared with them, or that have been shared with all cluster users (see Share a repo with tenants for details). Cluster users are also able to view any repos they register or create, but they must first be assigned the capability to do so. See the section on User Management for details on how RBAC works.
Returns information on all repos registered with a cluster including details on the associated bucket, path and read-write permissions.
End Point: /api/v3/robin_server/storage_repo?sub-command=list
Method: GET
URL Parameters: None
Data Parameters: None
Port: RCM Port (default value is 29442)
Headers:
Authorization: <auth_token>
: Authorization token to identify which user is sending the request. The token can be acquired from the login API.
Success Response Code: 200
Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error)
Example Response:
Output
{
"items":[
{
"tenant":"Administrators",
"backup_target":1,
"username":"robin",
"scan_end":1599634520,
"user_shares":{
},
"permissions":"readwrite",
"repo_type":"GCS",
"path":"my-appyx-bucket\/",
"user_id":3,
"repo_params":"<REDACTED>",
"repo_id":2,
"scan_diff":{
"added":0,
"removed":0
},
"name":"app-repofpom",
"share_with_all_tenants":false,
"scan_start":1599634519,
"tenant_shares":{
},
"scan_state":"SUCCESS",
"scan_error":"-",
"bucket":"ramendra",
"tenant_id":1
},
{
"tenant":"Administrators",
"backup_target":1,
"username":"robin",
"scan_end":1599641890,
"user_shares":{
},
"permissions":"readwrite",
"repo_type":"GCS",
"path":"my-appzu-bucket\/",
"user_id":3,
"repo_params":"<REDACTED>",
"repo_id":6,
"scan_diff":{
"added":0,
"removed":0
},
"name":"app-repotmcj",
"share_with_all_tenants":false,
"scan_start":1599641890,
"tenant_shares":{
},
"scan_state":"SUCCESS",
"scan_error":"-",
"bucket":"ramendra",
"tenant_id":1
}
]
}
12.4. Get information about a specific repo¶
In order to attain details of a specific repo issue the following command:
# robin repo info <reponame>
|
Name of the repo |
Example:
# robin repo info datadump
Name : datadump
Type : GCS
Bucket : datadump-bucket
Path : dumps/
Permissions : readwrite
Scan Details
-------
Scan State : SUCCESS
Start Time : 2019-04-08 16:05:35
End Time : 2019-04-08 16:05:35
Time Taken : 0s
Scan Error : -
Scan Diff : {'removed': 0, 'added': 0}
Apps attached: 1
helm/mydb
Note
The ability to view information about a specific repo is subject to Role Based Access Control (RBAC). Cluster administrators (users having the superadmin role) are able to view information about all registered repos in a cluster. Cluster users (users having the user role), on the other hand, are only able to view information about repos that have been explicitly shared with them or that have been shared with all cluster users (see Share a repo with tenants for details). See the section on User Management for details on how RBAC works.
Returns detailed information about a repo, such as the permissions associated with the repo, details about the last run scan and the applications attached to the repo.
End Point: /api/v3/robin_server/storage_repo/<repo_name>?sub-command=info
Method: GET
URL Parameters: None
Data Parameters: None
Port: RCM Port (default value is 29442)
Headers:
Authorization: <auth_token>
: Authorization token to identify which user is sending the request. The token can be acquired from the login API.
Success Response Code: 200
Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error), 404 (Not Found Error)
Example Response:
Output
{
"apps":[
"helm\/my-appyx",
"flexapp\/my-import1ry",
"robin\/demo"
],
"tenant":"Administrators",
"backup_target":1,
"username":"robin",
"scan_end":1599634520,
"user_shares":{
},
"permissions":"readwrite",
"repo_type":"GCS",
"path":"my-appyx-bucket\/",
"user_id":3,
"repo_params":"<REDACTED>",
"repo_id":2,
"scan_diff":{
"added":0,
"removed":0
},
"bucket":"ramendra",
"name":"app-repofpom",
"share_with_all_tenants":false,
"scan_start":1599634519,
"tenant_shares":{
},
"scan_error":"-",
"scan_state":"SUCCESS",
"tenant_id":1
}
12.5. List the contents of a repo (Browse Backup Catalog)¶
To browse the contents of a repo (a.k.a. browsing the backup catalog) use the following command:
# robin repo contents <reponame>
--zoneid <zoneid>
--backupid <backupid>
--refresh
--managed
|
Name of the repo to browse |
|
The zone ID to use when scanning repo contents (disaster recovery only) |
|
The backup Id to use when scanning repo contents (disaster recovery only) |
|
When specified the repo would be scanned again to build the catalog |
|
Take ownership of an orphaned backup in an external storage repo (disaster recovery only) |
Example:
# robin repo contents datadump --refresh
+----------------------------------+------------+------+------------+
| BackupID | ZoneID | App | Snapshot |
+----------------------------------+------------+------+------------+
| 5eccc5285a5211e9a1c3417886f14cc5 | 1554764031 | mydb | mydb_snap1 |
+----------------------------------+------------+------+------------+
Note
The ability to view the contents of a repo (view the repo catalog)
is subject to Role Based Access Control (RBAC). By default, only
Cluster Administrators (users having the superadmin role) have
permission to do so. Cluster Users can use the robin backup
list
command to view information about app backups that are
stored in a repo, at least for those backups they have view access
to. See the section on User Management for details on how
RBAC works.
In general, it should not be necessary to issue a robin repo
contents
command with the --refresh
command line argument.
All backups pushed to a repo from a cluster will automatically get
added to the repo’s catalog.
To perform disaster recovery (recover orphaned backups from an
external storage repo), issue the robin repo contents
command
with the --zoneid <zoneid>
command line argument. This will
allow all backups from the destroyed cluster to be added to the
repo catalog in the local Robin CNS cluster. If only a single
backup is required, then include the --backupid <backupid>
command line option as well. Note that recovered backups are
functionally identical to backups that have been imported from
another cluster. To take ownership of the recovered backups (have
the ability to issue a robin repo purge
command on them),
include the --managed
command line option.
12.6. Purge (delete) a backup from a repo¶
A snapshot that has been backed up to a repo can be purged (deleted from the repo) by issuing the following command:
# robin repo purge <reponame> <backupid>
|
Name of the repo to attach |
|
Id of application backup to purge |
Example:
# robin repo purge testrepo 930a489e4f6e11eab782034b0b612bc8 --wait
Are you sure you want to delete backup id: 930a489e4f6e11eab782034b0b612bc8 from repo: testrepo [y/n] ? y
Job: 200 Name: StorageRepoPurge State: PROCESSED Error: 0
Job: 200 Name: StorageRepoPurge State: COMPLETED Error: 0
Purges (deletes) a backup from a repo.
End Point: /api/v3/robin_server/storage_repo
Method: PUT
URL Parameters: None
Data Parameters:
action: purge
- This mandatory field within the payload specifies that the purge operation is to be performed.backupid: <backupid>
- This mandatory field within the payload specifies that the UUID of the backup to be purged.repo_name: <repo_name>
- This mandatory field within the payload specifies the repo from which the backup will be purged.
Port: RCM Port (default value is 29442)
Headers:
Authorization: <auth_token>
: Authorization token to identify which user is sending the request. The token can be acquired from the login API.
Success Response Code: 202
Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error), 404 (Not Found Error), 400 (Invalid API Usage Error)
Example Response:
Output
{
"jobid":704
}
12.7. Monitor the status of a repo¶
The status of transfers to and from a repo by issuing the following command:
# robin repo status <reponame>
|
Name of the repo to monitor |
Example:
# robin repo status datadump
+---------+------------------------------------+-----------------+
| Name | TransferStatus | Schedule/Manual |
+---------+------------------------------------+-----------------+
| backup1 | 20% [####................] 480.0MB | MANUAL |
+---------+------------------------------------+-----------------+
After a few seconds. We can run the command again to see:
# robin repo status datadump
+---------+-------------------------------------+-----------------+
| Name | TransferStatus | Schedule/Manual |
+---------+-------------------------------------+-----------------+
| backup1 | 100% [####################] 480.0MB | MANUAL |
+---------+-------------------------------------+-----------------+
Returns the transfer status of backups to a particular repo.
End Point: /api/v3/robin_server/k8s_app/
Method: GET
URL Parameters: None
Data Parameters:
action: get_k8s_backup_ops
- This mandatory field within the payload specifies that the status of backups should be returned.all: [true|false]
- This mandatory field within the payload specifies whether or not the status of all backups (both in progress and completed) within the repo should be returned.repo_name: <repo_name>
- This mandatory field within the payload specifies the repo to which the backups are being transferred.
Port: RCM Port (default value is 29442)
Headers:
Authorization: <auth_token>
: Authorization token to identify which user is sending the request. The token can be acquired from the login API.
Success Response Code: 200
Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error)
Example Response:
Output
{
"items":{
"26596ca6f27a11ea90fe3b3589dcc8cb":{
"snap_name":"my-appzu_my-s1px",
"bytes_transferred":503316480,
"state":"Pushed",
"bytes_count":503316480,
"op_trigger":"MANUAL",
"bytes_skipped":0
},
"be94d1d0f25e11eaa72edd449692af76":{
"snap_name":"my-appyx_my-s1tr",
"bytes_transferred":369098752,
"state":"Pushed",
"bytes_count":369098752,
"op_trigger":"MANUAL",
"bytes_skipped":0
}
}
}
12.9. Stop sharing a repo with tenants¶
In order to stop sharing a storage repo with one or more tenants, issue the following command:
# robin repo tenant-unshare <reponame> [tenant_list]
--all-tenants
|
Name of the repo to stop sharing |
|
List of tenants to stop sharing the repo with |
|
Stop sharing a repo (for specified operations) with all users |
Example:
# robin repo tenant-unshare testrepo t1 --wait
Job: 208 Name: StorageRepoUnshare State: PROCESSED Error: 0
Job: 208 Name: StorageRepoUnshare State: COMPLETED Error: 0
Note
The ability to stop sharing a storage repo with tenants is subject to Role Based Access Control (RBAC). Only Cluster Administrators (users having the superadmin role) have permission to do so. See the section on User Management for details on how RBAC works.
12.11. Stop sharing a repo with users¶
In otder stop sharing a storage repo with one or more users, issue the following command:
# robin repo user-unshare <reponame> [user_list]
--tenant <tenant>
--operations <operations>
--all-tenant-users
|
Name of the repo to stop sharing |
|
List of users to stop sharing the repo with |
|
Tenant user share should be removed from |
|
List of operations the user will no longer be allowed to perform. To allow all valid operations to be performed, specify ‘ALL_OPERATIONS’ on the command line (default value is ‘view’) |
|
Stop sharing a repo (for specified operations) with all users |
Example:
# robin repo user-unshare testrepo t1 admin1 --operations share --wait
Job: 205 Name: StorageRepoUnshare State: PROCESSED Error: 0
Job: 205 Name: StorageRepoUnshare State: COMPLETED Error: 0
Note
The ability to stop sharing a storage repo with users is subject to Role Based Access Control (RBAC). Only Cluster Administrators (users having the superadmin role) and Tenant Administrators (users having the tenantadmin role) have permission to do so. Tenant Administrators are only allowed to stop sharring a storage repo with users from their tenant. See the section on User Management for details on how RBAC works.