Overcoming cloud migration obstacles

Use Case/ Scenario

During cloud infrastructure migration from one AWS account to another,  services like VPC, EC2, etc and the content in storage services like S3 buckets should also be moved. However, application-related configuration settings that were applied to objects over time may or may not have been documented at the time of migration. The lack of clarity on configuration settings, each of the millions of objects in an S3 bucket, presents a problem of plenty where we have to diligently copy the object permissions of each object from a source bucket to a target bucket.

Copying objects across buckets while maintaining all application-relevant attributes of each object for a successful migration comes with some hurdles during the migration activity, and those are summarized in the following section.

Migration challenges

Various tools can be used to copy keys and objects from source to target S3 buckets, and each of them comes with its own set of issues -- either due to the maturity level of the tool to handle all S3 features, or through use cases created by customers using S3.

Past migration experience with the various tools used, and further tool feature(s) evaluation to overcome migration issues are tabulated below. This is not a complete feature comparison of the various tools used; it is a comparison of abilities to handle migration issues of objects in the said circumstances.

Tool capability matrix:

 

Windows AWS CLI

 Linux AWS    CLI 

CloudBerry

s3CMD

S3express

Bucket explorer

s3s3mirror

Copy object recursively

Yes

Yes

Yes

Yes

No

Yes

Yes

Copy Object ACL

No

No

No

No

Yes

No

No

Set copied object “bucket owner” value

No

No

Yes

No

No

No

No

Copy keys with no objects

No

No

Yes

Yes

No

 

Yes

Copy objects with "/"character in object names

No

Yes

No

 

 

 

 

Copy keys which do not have a value set name(null)

Yes

Yes

No

 

 

 

 

Ongoing differential Sync

No

No

No

No

No

No

No

Proper log files regarding copy activity

No

No

No

No

No

No

No

Bucket level actions

No

No

No

No

No

No

No

 

Solution

AWS also provides API and SDKs to copy keys and objects across S3 in separate accounts. In the use case of a migration activity where the count of keys and objects within an S3 bucket run into the millions, automation is more suited for copying, comparing, reporting, as well as for ongoing differential sync activities. The following section describes the high-level features of automation achieved using AWS services (S3 events, Lambda, DynamoDB, SNS), python, and boto3. Note: The python implementation matures with time with respect to performance and features.

  1. Automated copy of bucket-level ACL, bucket policy, and CORS:
  • Get ACL, bucket policy, and CORS from source S3 bucket
  • Apply the copied ACL, bucket policy, and CORS to target S3 bucket
  • Creates log of the above steps in a log file
  1. Copy keys and objects from source to destination bucket:
  • Get a list of keys and objects from source S3 bucket
  • Copy each key and object to target S3 bucket
  • Read ACL of each object in source bucket and apply it to corresponding target object along with appropriate ‘owner’ value of object in target S3 bucket
  • Creates log of the above steps in a log file
  1. Differential sync of objects using lambda and S3 events:
  • When an object is uploaded (PUT) or removed (DELETE) from source S3 bucket, S3 provides an event notification that triggers a custom Lambda function to replicate the action in the target S3 bucket
  • The Lambda function:
    • Retrieves the key and object name from S3 event
    • Copies that object from source to target bucket
    • Reads ACL of the source object; applies it to corresponding object in the target bucket
    • Creates log of the action in DynamoDB

Limitation

  • If only the ACL of an existing object in source bucket is changed, that updated ACL is not replicated to the destination

You might also like...

Persistent Cloud Cover_0

Finance, healthcare, manufacturing or retail – whichever the industry, enterprises today are sitting on huge volumes of data. And what is data without insights? Which is why all stats lead to…

Continue Reading >

Hybrid-Cloud-for-Data-Driven-Decision-Making_0

Cloud computing today has emerged as one of the top items on the enterprise digitalization agenda. With adoption rates continuously increasing, the public cloud market keeps expanding in size and…

Continue Reading >

Driving-Digital-Transformation-with-Hybrid-Cloud_0

Agility and innovation today have become the mantra for business success.

Continue Reading >