Direct Upload to S3 (with a little help from jQuery) [Updated]

The contents of this article has been replaced by a PHP Composer package, hope you find it useful.
View on Github

Whist working on a new web app recently, running on Heroku, we wanted users to be able to upload large files (I’m talking 50mb-150mb) onto our site. We started by having the standard html form post the data to our back-end code, which processed it, etc, etc. This was fine for small files, but large files would hit the maximum request time of 30 seconds for any request on heroku, which then in turn messed up the dyno it was running on.

The solution was to upload the file directly to Amazon’s S3 (Simple Storage) and not to the back-end code at all. This method comes with advantages and disadvantages of course. One of the disadvantages is the small matter of tracking the file once it’s been uploaded. How do you add a file/row to your database to track the file if you aren’t calling any back-end code? We’ll take a look at these problem and how we can bypass it in the tutorial.

Edit 1st Feb 14: Complete code update thanks to comments by Matt Painter and Ian Mackay.

HTML#

To upload a file directly to S3 and have the ability to then track it within your app the front-end must have two separate forms (they can look like they’re one, but we need two action locations).

Edit 18th Feb 14: Form action attribute changed from //<?php echo $bucket; ?>.s3.amazonaws.com to //s3-eu-west-1.amazonaws.com/<?php echo $bucket; ?>/ due to a comment by Matt and it not working over HTTPS. Make sure, however, that your S3 bucket is in fact based in: s3-eu-west-1 (or change appropriately).

<!-- Direct Upload to S3 -->
<!-- URL prefix (//) means either HTTP or HTTPS (depending on which is being currently used) -->
<form action="//s3-eu-west-1.amazonaws.com/<?php echo $bucket; ?>/" method="POST" enctype="multipart/form-data" class="direct-upload">

    <!-- We'll specify these variables with PHP -->
    <!-- Note: Order of these is Important -->
    <input type="hidden" name="key" value="${filename}">
    <input type="hidden" name="AWSAccessKeyId" value="<?php echo $accesskey; ?>">
    <input type="hidden" name="acl" value="private">
    <input type="hidden" name="success_action_status" value="201">
    <input type="hidden" name="policy" value="<?php echo $base64Policy; ?>">
    <input type="hidden" name="signature" value="<?php echo $signature; ?>">

    <input type="file" name="file" />

    <!-- Progress Bar to show upload completion percentage -->
    <div class="progress"><div class="bar"></div></div>

</form>

<!-- Used to Track Upload within our App -->
<form action="process-form-data.php" method="POST">
    <input type="hidden" name="upload_original_name" />

    <label for="upload_custom_name">Name:</label><br />
    <input type="text" name="upload_custom_name" id="upload_custom_name" /><br />

    <input type="submit" value="Save" />
</form>

The important bit of this code is in the first form within the hidden input parameters.

name	desc
key	This is the name given to it when saving it on S3, by default it will be ${filename} which will take name it has on upload, but you can give it unique name by hashing it or adding in a prefix folder name.
AWSAccessKeyId	This is your AWS access key, given to you through the S3 interface.
acl	This is the access control list, which can be private (only giving the owner rights), public-read (giving AllUsers read capabilities), public-read-write (giving AllUsers both read and write capabilities). For a full list see the AWS documentation.
success_action_status	This tells S3 what to do on success, it’s possible to also redirect on success but we’ll be using AJAX and XHR requests so we’ll be wanting it to return a successful HTTP status code.
policy	This is the upload policy, we’ll talk about a little bit more about later on, but is a base 64 encoded string containing important information related to the upload.
signature	The signature is an encoded string which includes both your S3 secret and the upload policy.

PHP#

Although the file will upload directly to S3, we need to use a little bit of server-side code (PHP in this case) to generate some necessary hidden input parameters. For any direct upload to succeed on S3, a policy has to be specified — which contains information like the request expiry, acl (whether it’s private or public) and what happens on success.

So with PHP we’ll both specify a base64 encoded policy and add our S3 secret (also encoded).

For this to work you’ll need to change the $bucket variable to an already created S3 bucket; the $accesskey to your S3 key you will have been given; and the $secret variable to your AWS secret. Make sure to not expose your secret to the public.

Edit 10th Sept 14: Update expiry to include timezone & increase to 6 hours, thanks to S. Kristoff’s comment.

<?php

// Fill These In!
$bucket = "";
$accesskey = "";
$secret = "";

$policy = json_encode(array(
    'expiration' => date('Y-m-d\TG:i:s\Z', strtotime('+6 hours')),
    'conditions' => array(
        array(
            'bucket' => $bucket
        ),
        array(
            'acl' => 'private'
        ),
        array(
            'starts-with',
            '$key',
            ''
        ),
        array(
            'success_action_status' => '201'
        )
    )
));
$base64Policy = base64_encode($policy);
$signature = base64_encode(hash_hmac("sha1", $base64Policy, $secret, $raw_output = true));
?>

Edit 7th Jan 15: It’s possible to expand the policy to limit the file size being uploaded, using content-length-range. You can also specify the content-type of the upload file, if you’re sticking to a single file type. These are explained in Daniel Katzan’s comment.

S3 Configuration – Allow CORS#

All modern browsers now allow cross origin resource sharing (CORS). This means that by default you can’t post random data from your form to another website. It is possible however to explicitly allow it, so the next step is to go into your S3 account, find the bucket you are working with and open it’s properties. In the properties you’ll see an ‘Add CORS Configuration’ button. Within this you can paste XML allowing your bucket to receive POST data from your domain.

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedHeader>*</AllowedHeader>  
    </CORSRule>
</CORSConfiguration>

jQuery#

Now we’ll just want to add the finishing touches by using jQuery, jQuery UI and the jQuery File Upload plugin (which you’ll need to download). The iframe-transport-js and fileupload.js should be the only files needed.

This jQuery was created with help from a tutorial by Pierre Jambet which is geared towards Ruby development.

HTML

<script src="//code.jquery.com/jquery-1.11.2.min.js"></script>
<script src="//code.jquery.com/ui/1.10.4/jquery-ui.js"></script>
<script src="fileupload/jquery.fileupload.js"></script>

JS

$(document).ready( function() {
    $('.direct-upload').each( function() {
        var form = $(this);

        form.fileupload({
            url: form.attr('action'),
            type: 'POST',
            datatype: 'xml',
            add: function (event, data) {

                // Message on unLoad.
                window.onbeforeunload = function() {
                    return 'You have unsaved changes.';
                };

                // Submit
                data.submit();
            },
            send: function(e, data) {
                // onSend
            },
            progress: function(e, data){
                // This is what makes everything really cool, thanks to that callback
                // you can now update the progress bar based on the upload progress.
                var percent = Math.round((data.loaded / data.total) * 100);
                $('.bar').css('width', percent + '%');
            },
            fail: function(e, data) {
                // Remove 'unsaved changes' message.
                window.onbeforeunload = null;
            },
            success: function(data) {
                // onSuccess
            },
            done: function (event, data) {
                // Fill the name field with the file's name.
                $('#upload_original_name').val(data.originalFiles[0].name);
                $('#upload_custom_name').val(data.originalFiles[0].name);
            },
        });
    });
});

CSS (Optional)#

We’ll also add in a little CSS to style a custom made progress bar (it’s nothing special) which we’ll interact with through jQuery.

.progress {
    position: relative;
    width: 100%;
    height: 15px;
    background: #C7DA9F;
    border-radius: 10px;
    overflow: hidden;
}

.bar {
    position: absolute;
    top: 0;
    left: 0;
    width: 0;
    height: 15px;
    background: #85C220;
}

This article has been update to use the AWS Signature V4 within a different post.
It’s recommended you use the code from the new article instead.

View Shiny New Article