Use Case/ Scenario
During cloud infrastructure migration from one AWS account to another, services like VPC, EC2, etc and the content in storage services like S3 buckets should also be moved. However, application-related configuration settings that were applied to objects over time may or may not have been documented at the time of migration. The lack of clarity on configuration settings, each of the millions of objects in an S3 bucket, presents a problem of plenty where we have to diligently copy the object permissions of each object from a source bucket to a target bucket.
Copying objects across buckets while maintaining all application-relevant attributes of each object for a successful migration comes with some hurdles during the migration activity, and those are summarized in the following section.
Migration challenges
Various tools can be used to copy keys and objects from source to target S3 buckets, and each of them comes with its own set of issues -- either due to the maturity level of the tool to handle all S3 features, or through use cases created by customers using S3.
Past migration experience with the various tools used, and further tool feature(s) evaluation to overcome migration issues are tabulated below. This is not a complete feature comparison of the various tools used; it is a comparison of abilities to handle migration issues of objects in the said circumstances.
Tool capability matrix:
| Windows AWS CLI | Linux AWS CLI | CloudBerry | s3CMD | S3express | Bucket explorer | s3s3mirror |
---|---|---|---|---|---|---|---|
Copy object recursively | Yes | Yes | Yes | Yes | No | Yes | Yes |
Copy Object ACL | No | No | No | No | Yes | No | No |
Set copied object “bucket owner” value | No | No | Yes | No | No | No | No |
Copy keys with no objects | No | No | Yes | Yes | No |
| Yes |
Copy objects with "/"character in object names | No | Yes | No |
|
|
|
|
Copy keys which do not have a value set name(null) | Yes | Yes | No |
|
|
|
|
Ongoing differential Sync | No | No | No | No | No | No | No |
Proper log files regarding copy activity | No | No | No | No | No | No | No |
Bucket level actions | No | No | No | No | No | No | No |
Solution
AWS also provides API and SDKs to copy keys and objects across S3 in separate accounts. In the use case of a migration activity where the count of keys and objects within an S3 bucket run into the millions, automation is more suited for copying, comparing, reporting, as well as for ongoing differential sync activities. The following section describes the high-level features of automation achieved using AWS services (S3 events, Lambda, DynamoDB, SNS), python, and boto3. Note: The python implementation matures with time with respect to performance and features.
Limitation
Contact Us Today
Let’s Talk
x