This post describes how to mount an S3 bucket to all the nodes in an EKS cluster and make it available to pods as a hostPath volume. Yes, we're awa...
For further actions, you may consider blocking this person and/or reporting abuse
Cool setup. Have you tested its speed?
No, speed wasn't a consideration here. Main motivation here was providing an easy and transparent way to upload files and make them accessible to pods. S3 gives users an easy and secure UI for that. Goofys is supposedly quite performant compared to other FUSE implementations (i.e s3fs). But we haven't benchmarked this ourselves.
Thx! Would love to know numbers if you ever do try it :-)
Hi, Can set multiple bucketName?
I need to interact with few s3 buckets for different tasks
hi @oleksiihead
no support for this right now.
to add this one would need to do smthng like:
If you get to do this - please submit a PR.
Amazing approach!
Thx for sharing in details.
I tried to use the sharing method to complete the entire demo, but unfortunately this didn't work. Because the goofys mount directory( /var/s3fs) in daemonset is not the same as the directory I want to share with the host(/var/s3fs:shared);
Is there any configuration I missed?
Daemonset.yaml
What's your node OS? Is mount propagation enabled in the container runtime? See this note here: kubernetes.io/docs/concepts/storag...
I have tried this and the other similar option mentioned in this blog. blog.meain.io/2020/mounting-s3-buc.... In neither case, the mounting to hostPath was successful for the cluster managed by AWS EKS.
Hi @dirai09 , this was originally tested on AWS EKS. I haven't tested it since but it should in theory still work. What is the error you're getting when trying to mount the hostPath?
Also - can you share your config in a gist?
Nice approach. However, you might want to have a look at JuiceFS:
github.com/juicedata/juicefs
That has quite a good performance due to the combination with Redis and it is made with Kubernetes in mind.
Thanks for the wonderful suggestion @randy
In high number of files its fail you due to nature of s3 api for small files the http response will be bigger than files.
I think mounting s3 is a bad idea, if you have enough developing resources its better to write a client for code to connect directly to s3 and cache list of s3 files ... For better performance.
But its a fun thing to do, also cephfs with rados gateway will give you better performance in kubernetes
good to know. not an issue in our case - we have a small number of large files there. And I agree it's not such a great idea in general - both performance wise and because of the hidden complexity. But it solved our specific itch and may help others solve it.
Hi,
I don't think I am able to mount the volumes on the hostPath. Am I missing something here.
I ran into an issue where goofys doesn't reload the content of a small txt file. It updates the timestamp though. Do you know what could be wrong?
I have goofys run inside a container.