Use Case/ Scenario
During cloud infrastructure migration from one AWS account to another, services like VPC, EC2, etc and the content in storage services like S3 buckets should also be moved. However, application-related configuration settings that were applied to objects over time may or may not have been documented at the time of migration. The lack of clarity on configuration settings, each of the millions of objects in an S3 bucket, presents a problem of plenty where we have to diligently copy the object permissions of each object from a source bucket to a target bucket.
Copying objects across buckets while maintaining all application-relevant attributes of each object for a successful migration comes with some hurdles during the migration activity, and those are summarized in the following section.
Migration challenges
Various tools can be used to copy keys and objects from source to target S3 buckets, and each of them comes with its own set of issues -- either due to the maturity level of the tool to handle all S3 features, or through use cases created by customers using S3.
Past migration experience with the various tools used, and further tool feature(s) evaluation to overcome migration issues are tabulated below. This is not a complete feature comparison of the various tools used; it is a comparison of abilities to handle migration issues of objects in the said circumstances.
Tool capability matrix:
| Windows AWS CLI | Linux AWS CLI | CloudBerry | s3CMD | S3express | Bucket explorer | s3s3mirror |
|---|---|---|---|---|---|---|---|
Copy object recursively | Yes | Yes | Yes | Yes | No | Yes | Yes |
Copy Object ACL | No | No | No | No | Yes | No | No |
Set copied object “bucket owner” value | No | No | Yes | No | No | No | No |
Copy keys with no objects | No | No | Yes | Yes | No |
| Yes |
Copy objects with "/"character in object names | No | Yes | No |
|
|
|
|
Copy keys which do not have a value set name(null) | Yes | Yes | No |
|
|
|
|
Ongoing differential Sync | No | No | No | No | No | No | No |
Proper log files regarding copy activity | No | No | No | No | No | No | No |
Bucket level actions | No | No | No | No | No | No | No |
Solution
AWS also provides API and SDKs to copy keys and objects across S3 in separate accounts. In the use case of a migration activity where the count of keys and objects within an S3 bucket run into the millions, automation is more suited for copying, comparing, reporting, as well as for ongoing differential sync activities. The following section describes the high-level features of automation achieved using AWS services (S3 events, Lambda, DynamoDB, SNS), python, and boto3. Note: The python implementation matures with time with respect to performance and features.
- Automated copy of bucket-level ACL, bucket policy, and CORS:
- Get ACL, bucket policy, and CORS from source S3 bucket
- Apply the copied ACL, bucket policy, and CORS to target S3 bucket
- Creates log of the above steps in a log file
- Copy keys and objects from source to destination bucket:
- Get a list of keys and objects from source S3 bucket
- Copy each key and object to target S3 bucket
- Read ACL of each object in source bucket and apply it to corresponding target object along with appropriate ‘owner’ value of object in target S3 bucket
- Creates log of the above steps in a log file
- Differential sync of objects using lambda and S3 events:
- When an object is uploaded (PUT) or removed (DELETE) from source S3 bucket, S3 provides an event notification that triggers a custom Lambda function to replicate the action in the target S3 bucket
- The Lambda function:
- Retrieves the key and object name from S3 event
- Copies that object from source to target bucket
- Reads ACL of the source object; applies it to corresponding object in the target bucket
- Creates log of the action in DynamoDB
Limitation
- If only the ACL of an existing object in source bucket is changed, that updated ACL is not replicated to the destination
