Problems when uploading large files to Amazon S3

By : Alexey
Source: Stackoverflow.com
Question!

I tried to use Amazon-SDK(Java) sample code S3TransferProgressSample.java to upload large files to Amazon-S3 storage (also posted here on AWS docs).

But when I am trying to upload 11 GB files, the upload is getting stuck at different points with the error message:

Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot).

It looks like that after IOException occurs SDK is not able to retry the request (see below).

Does anyone encounter this? What is the best-practice to resolve this? Any code is appreciated.

INFO: Received successful response: 200, AWS Request ID: 2B66E7669E24DA75
Jan 15, 2011 6:44:46 AM com.amazonaws.http.HttpClient execute
INFO: Sending Request: PUT s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters: (uploadId: m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )
Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:12 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:13 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:17 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
....
Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient handleResponse
INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007
Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient execute
...
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
INFO: Sending Request: DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
...
Jan 15, 2011 6:47:01 AM com.amazonaws.http.HttpClient handleErrorResponse
INFO: Received error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595, AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

By : Alexey


Answers

The answer of Geoff Appleford works for me. However, I would add a

By : aviad


I wanted to add a comment to Geoff Appleford's answer but SO wouldn't allow me to. In general his answer to use low level API works fine but even if we do now have a do-while loop the way for loop is designed there is in-built retry logic. In his code snippet the file position increases only when there is a success otherwise you are uploading the same part again.

By : SurenderS


Are you trying to upload a single file of size 11GB? Or the size of all your files combined is 11GB? Because Maximum File Size limit on S3 is 5GB.



This video can help you solving your question :)
By: admin