Tuesday, May 7, 2019

Crib-Notes: Offline Delta-Syncs of S3 Buckets

In the normal world, synchronizing two buckets is as simple as doing `aws s3 sync <SYNC_OPTIONS> <SOURCE_BUCKET> <DESTINATION_BUCKET>`. However, due to the information security needs of some of my customers, it's occasionally necessary to perform data-synchronizations between two S3 buckets, but using methods that amount to "offline" transfers.

To illustrate what is meant by "offline":
  1. Create a transfer-archive from a data source
  2. Copy the transfer-archive across a security boundary
  3. Unpack the transfer-archive to its final destination

Note that things are a bit more involved than the summary of the process – but this gives you the gist of the major effort-points.

The first time you do an offline bucket sync, transferring the entirety of a bucket is typically the goal. However, for a refresh-sync – particularly for a bucket of greater than a trivial content-size, this can be sub-ideal. For example, it might be necessary to do monthly syncs of a bucket that grows by a few Gigabytes per month. After a year, a full sync can mean having to move tens to hundreds of gigabytes. A better way is to only sync the deltas – copying only what's changed between the current and immediately-prior sync-tasks (a few GiB rather than tens to hundreds).

The AWS CLI tools don't really have a "sync only the files that have been added/modified since <DATE>". That said, it's not super difficult to work around that gap. A simple shell script like the following works a trick:

for FILE in $( aws s3 ls --recursive s3://<SOURCE_BUCKET>/  | \
   awk '$1 > "2019-03-01 00:00:00" {print $4}' )
do
   echo "Downloading ${FILE}"
   install -bDm 000644 <( aws s3 cp "s3://<SOURCE_BUCKET>/${FILE}" - ) \
     "<STAGING_DIR>/${FILE}"
done

To explain the above:

  1. Create a list of files to iterate:
    1. Invoke a subprocess using the $() notation. Within that subprocess...
    2. Invoke the AWS CLI's S3 module to recursively list the source-bucket's contents (`aws s3 ls --recursive`)
    3. Pipe the output to `awk` – looking for any date-string that's newer than the value in s3 ls's first output-column (the file-modification date column) and print out only the fourth column (the S3 object-path)
    The output from the subprocess is captured that output as an iterable list-structure
  2. Use a for loop-method to iterate the previously-assembled list, assigning each S3 object-path to the ${FILE} variable
  3. Since I hate sending programs off to do things in silence (I don't trust them to not hang), my first looped-command is to say what's happening via the echo "Downloading ${FILE}" directive.
  4. The install line makes use of some niftiness within both BASH and the AWS CLI's S3 command:
    1. By specifying "-" as the "destination" for the file-copy operation, you tell the S3 command to write the fetched object-contents to STDOUT.
    2. BASH allows you take a stream of output and assign a file-handle to it by surrounding the output-producing command with <( ).
    3. Invoking the install command with the -D flag tells the command to "create all necessary path-elements to place the source 'file' in the desired location within the filesystem, even if none of the intervening directory structure exists, yet."
    Putting it all together, the install operation takes the streamed s3 cp output, and installs it as a file (with mode 000644) at the location derived from the STAGING_DIR plus the S3 object-path ...thus preserving the SOURCE_BUCKET's content-structure within the STAGING_DIR
Obviously, this method really only works for additive/substitutive deltas. If you need to account for deletions and/or moves, this approach will be insufficient.

9 comments:

  1. This blog covers some most asked topics in the Azure Data Engineer Interview Questions with fully explained answers and details dp203

    ReplyDelete
  2. Addressing titular discrepancies in home decor is crucial for harmony. By consulting insightful COFFEE TABLE BOOKS, homeowners can find inspiration, ensuring cohesion and elegance. These books not only guide but also elevate the aesthetic, ensuring a unified and sophisticated living space

    ReplyDelete
  3. Crib-Notes enlightens the tech realm, akin to the way we illuminate homes with our Organic Online Grocery Store. Just as syncing data ensures precision, we guarantee the accuracy and freshness of our organic products. Explore the synchronization of technology and nature - where seamless data meets the pure essence of organics, only at our store.

    ReplyDelete
  4. Offline delta-syncs of S3 buckets" refers to updating files in Amazon S3 storage with only the changes made since the last synchronization, reducing bandwidth usage. Implementing this efficiently requires technical expertise. social media marketing agency in delhi might utilize this for managing and updating client media assets securely and cost-effectively

    ReplyDelete
  5. Experience the epitome of luxury with our Pashmina Shawl. Crafted from the finest Pashmina wool, it exudes timeless elegance and sophistication. Perfect for adding a touch of glamour to any outfit, this shawl offers both warmth and style. Whether you're attending a formal event or simply cozying up at home, our Pashmina Shawl is the ultimate accessory to elevate your look and embrace unparalleled comfort in exquisite fashion.

    ReplyDelete