API gateway has payload limits. docs.aws.amazon.com/apigateway/lat... 10MB for input payloads. Also lambda will have similar payload limits, so usually you want to sidestep this by only passing metadata about big files, similar to how you'd want to model databases for files.
An alternative to this URL-passing strategy is to utilize AWS cognito. docs.aws.amazon.com/IAM/latest/Use... Cognito users can be granted IAM permissions allowing them to write to restricted areas of S3 buckets. I'm not a huge fan of cognito user pools, and getting custom auth working through cognito for IAM is a pretty huge task.
One more thing to note: I typically tell people "don't request the presigned URL unless you are going to use it literally ASAP". This is important for when people present download links or upload links. The link should not pre-populate with presigned urls, but instead should be JS that fetches the URL and immediately fires it. I use a 300s time limit on all presigned URLs and have no issues. It's important to know that the time limit is only on the START of the request, not the end -- so people with slow connections only need to start the call in 300s, not finish it (good to know for uploads). Of course, when you use put-part, you should get each part URL right before putting the part -- not all at the beginning.
did you mean 300ms? i'm having the same problem with downloading large S3 objects via Api gateway and we are currently using a pre-signed URLs returned from the backend to the UI. However we were concerned with potential security breach if somebody malicious managed to get hold of the pre-signed URL. We currently set the time limit expiry to 5 minutes.
The alternative approach of using Cognito seemed like a large task to undertake given this should help mitigate the security risk
No, I meant 300s (5m). 300ms is far too low and you'd certainly have issues with people not having time to execute it. Keep in mind you're minting this URL and then sending it over network to a client who is expected to then call S3 within the time you have allotted, which includes the wire time to send it to them and their wire time to request it from s3. If I'm remembering correctly, S3 doesn't even deal in milliseconds, the lowest value you can set is 1 second which I think is still too low.
I think if your front end is using a pre-signed URL and it's never shown to a user you should ask yourself, what is the attack vector that you're worried about? A hacker that can get ahold of a 5min URL can certainly get ahold of a 1min URL or even a 30s URL. The closer you get to real world latencies on cell networks, the more likely your users are going to see failures when following the presigned URL. We decided 5minutes was low enough to mitigate risk, but I'd be cautious recommending anything less than 1 minute or like 30 seconds if you have cell users.
From experience though, it does seem like the timeout is until the user calls S3, not finishes the s3 call, so s3 won't hang up on users who have slow connections, which was something we worried would happen when we selected 5min. It is possible that their connection naturally times out or disconnects, which may have some recovery implications with lower presigned URL durations.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
API gateway has payload limits. docs.aws.amazon.com/apigateway/lat... 10MB for input payloads. Also lambda will have similar payload limits, so usually you want to sidestep this by only passing metadata about big files, similar to how you'd want to model databases for files.
An alternative to this URL-passing strategy is to utilize AWS cognito. docs.aws.amazon.com/IAM/latest/Use... Cognito users can be granted IAM permissions allowing them to write to restricted areas of S3 buckets. I'm not a huge fan of cognito user pools, and getting custom auth working through cognito for IAM is a pretty huge task.
One more thing to note: I typically tell people "don't request the presigned URL unless you are going to use it literally ASAP". This is important for when people present download links or upload links. The link should not pre-populate with presigned urls, but instead should be JS that fetches the URL and immediately fires it. I use a 300s time limit on all presigned URLs and have no issues. It's important to know that the time limit is only on the START of the request, not the end -- so people with slow connections only need to start the call in 300s, not finish it (good to know for uploads). Of course, when you use put-part, you should get each part URL right before putting the part -- not all at the beginning.
Hope this helps!
did you mean 300ms? i'm having the same problem with downloading large S3 objects via Api gateway and we are currently using a pre-signed URLs returned from the backend to the UI. However we were concerned with potential security breach if somebody malicious managed to get hold of the pre-signed URL. We currently set the time limit expiry to 5 minutes.
The alternative approach of using Cognito seemed like a large task to undertake given this should help mitigate the security risk
No, I meant 300s (5m). 300ms is far too low and you'd certainly have issues with people not having time to execute it. Keep in mind you're minting this URL and then sending it over network to a client who is expected to then call S3 within the time you have allotted, which includes the wire time to send it to them and their wire time to request it from s3. If I'm remembering correctly, S3 doesn't even deal in milliseconds, the lowest value you can set is 1 second which I think is still too low.
I think if your front end is using a pre-signed URL and it's never shown to a user you should ask yourself, what is the attack vector that you're worried about? A hacker that can get ahold of a 5min URL can certainly get ahold of a 1min URL or even a 30s URL. The closer you get to real world latencies on cell networks, the more likely your users are going to see failures when following the presigned URL. We decided 5minutes was low enough to mitigate risk, but I'd be cautious recommending anything less than 1 minute or like 30 seconds if you have cell users.
From experience though, it does seem like the timeout is until the user calls S3, not finishes the s3 call, so s3 won't hang up on users who have slow connections, which was something we worried would happen when we selected 5min. It is possible that their connection naturally times out or disconnects, which may have some recovery implications with lower presigned URL durations.