Hello, in this tutorial we will learn how to upload files directly to MongoDB using GridFS specification.
If you think TLDR; just check finish code here.
The official docs explain when to use this specification for uploading files. Which is summarized in the following:
If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as needed.
When you want to access information from portions of large files without having to load whole files into memory, you can use GridFS to recall sections of files without reading the entire file into memory.
When you want to keep your files and metadata automatically synced and deployed across a number of systems and facilities, you can use GridFS. When using geographically distributed replica sets, MongoDB can distribute files and their metadata automatically to a number of mongod instances and facilities.
Since, GridFS stores files in chunks. Following are the collections created:
- chunks stores the binary chunks.
- files stores the file’s metadata.
Prerequisites
- NodeJS LTS
- MongoDB installed on your local machine
- a Code Editor
Setting up a local NodeJS server
Go to your command line, and type
npm init -y
This will generate an package.json file with default values.
Then install all the dependencies required for this project
npm install express mongoose ejs multer multer-gridfs-storage
Create a file named app.js in the root of the project. Require the necessary packages for creating a server.
const express = require("express");
const app = express();
app.use(express.json());
app.set("view engine", "ejs");
const port = 5001;
app.listen(port, () => {
console.log("server started on " + port);
});
It will be better for us to create scripts to run the web app from the command line, go to your package.json file and on the scripts key, add the following:
"scripts": {
"start": "node app.js",
"dev": "nodemon app.js"
}
then run, npm start and the server should start on the port 5001. You should see one log on the command line stating that server started on 5001.
Connecting to Database, Initializing GridFsStorage and Creating a Storage
Require all the necessary packages
const crypto = require("crypto");
const path = require("path");
const mongoose = require("mongoose");
const multer = require("multer");
const GridFsStorage = require("multer-gridfs-storage");
Mongoose is an ORM for MongoDB which will be used for this tutorial. Multer is a NodeJS middleware which facilitates file uploads. And GridFsStorage is GridFS storage engine for Multer to store uploaded files directly to MongoDB. Crypto and Path will be used to create unique name for the file uploaded.
// DB
const mongoURI = "mongodb://localhost:27017/node-file-upl";
// connection
const conn = mongoose.createConnection(mongoURI, {
useNewUrlParser: true,
useUnifiedTopology: true
});
Now, Initializing the GridFsStorage
// init gfs
let gfs;
conn.once("open", () => {
// init stream
gfs = new mongoose.mongo.GridFSBucket(conn.db, {
bucketName: "uploads"
});
});
Here we are using the native nodejs-mongodb-drive which the mongoose uses and creating a GridFSBucket, we are passing the db to the bucket, you can see we are giving one bucket name, this bucket name will be used a name of a collection.
// Storage
const storage = new GridFsStorage({
url: mongoURI,
file: (req, file) => {
return new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
const filename = buf.toString("hex") + path.extname(file.originalname);
const fileInfo = {
filename: filename,
bucketName: "uploads"
};
resolve(fileInfo);
});
});
}
});
const upload = multer({
storage
});
Now we are initializing the storage as per Multer GridFS and creating random bytes using the randomBytes method present on the crypto library.
Here we are using the Promise constructor to create a promise, which then resolves with the fileInfo object. This step is optional as you can only pass a url key and the bucket will work just fine and not change the file name. For example you can use like the following :
const storage = new GridFsStorage({ url : mongoURI})
Next lets set up our frontend with a template engine and configure express to render the template.
Creating the view
Create a new folder named views in the root of the folder, and inside it create a file named index.ejs. Here we will store our front end view. I will not bore you guys will the HTML creation and just post the code for it. I am using bootstrap for fast prototyping.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
<title>Mongo File Upload</title>
</head>
<body>
<div class="container">
<div class="row">
<div class="col-md-6 m-auto">
<h1 class="my-4">Lets upload some stuff</h1>
<form action="/upload" method="post" enctype="multipart/form-data">
<div class="custom-file mb-3">
<input type="file" class="custom-file-input" name="file" id="file1" onchange="readSingleFile(this.files)">
<label class="custom-file-label" for="file1" id="file-label">Choose file</label>
</div>
<input type="submit" value="Submit" class="btn btn-primary btn-block">
</form>
</div>
</div>
</div>
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
<script>
function readSingleFile(e) {
const name = e[0].name;
document.getElementById("file-label").textContent = name;
}
</script>
</body>
</html>
Setting up the express app to render the view. Set up the view engine middleware to ejs
....
app.use(express.json());
app.set("view engine", "ejs");
....
app.get("/", (req, res) => {
res.render("index")
})
Then start the server again, go to the browser and open http://localhost:5001, and you should see one page rendered with the view we just created.
Create Request to handle the form submit and upload the file
app.post("/upload", upload.single("file"), (req, res) => {
res.redirect("/");
});
As we already did most of our heavy lifting while creating a storage bucket and multer take cares of the rest. We just need to pass the middleware and then just redirect to the same url.
The tricky part is to download or in this case stream the data from the GridFS storage bucket and render the image, for that we will create a route for showing an image that will take the name of the file as an argument or passed as a route param.
app.get("/image/:filename", (req, res) => {
// console.log('id', req.params.id)
const file = gfs
.find({
filename: req.params.filename
})
.toArray((err, files) => {
if (!files || files.length === 0) {
return res.status(404).json({
err: "no files exist"
});
}
gfs.openDownloadStreamByName(req.params.filename).pipe(res);
});
});
On the gridfs bucket we get access to many methods one such is find, which is very similar to normal find in MongoDB and accepts a filename as a first argument and then we are converting the result to an array and check if there is any file with such filename and if there is we use another method which is present on the gridfs bucket called openDownloadStreamByName which then again takes the filename and then we use the pipe to return the response to the client.
Now up until now, we can get the image with the above route but no way to render it on our view, so let's create a method inside the route where we were rending our index.ejs page.
....
app.get("/", (req, res) => {
if(!gfs) {
console.log("some error occured, check connection to db");
res.send("some error occured, check connection to db");
process.exit(0);
}
gfs.find().toArray((err, files) => {
// check if files
if (!files || files.length === 0) {
return res.render("index", {
files: false
});
} else {
const f = files
.map(file => {
if (
file.contentType === "image/png" ||
file.contentType === "image/jpeg"
) {
file.isImage = true;
} else {
file.isImage = false;
}
return file;
})
.sort((a, b) => {
return (
new Date(b["uploadDate"]).getTime() -
new Date(a["uploadDate"]).getTime()
);
});
return res.render("index", {
files: f
});
}
});
});
....
Here you can see a lot of optional code like the sorting of the array and you can skip those.
Now, On the template, we loop over the files sent and then show the images below the form. We will only render the files which are of type jpg or png, that check can be upgraded by using a regex and depends on the personal preference.
<hr>
<% if(files) { %>
<% files.forEach(function(file) {%>
<div class="card mb-3">
<div class="card-header">
<div class="card-title">
<%= file.filename %>
</div>
</div>
<div class="card-body">
<% if (file.isImage) { %>
<img src="image/<%= file.filename %>" width="250" alt="" class="img-responsive">
<%} else { %>
<p><% file.filename %></p>
<% } %>
</div>
<div class="card-footer">
<form action="/files/del/<%= file._id %>" method="post">
<button type="submit" class="btn btn-danger">Remove</button>
</form>
</div>
</div>
<%}) %>
<% } else { %>
<p>No files to show</p>
<% } %>
You can see there is one remove button on the above code, so let us create one delete route to remove the file from the database.
// files/del/:id
// Delete chunks from the db
app.post("/files/del/:id", (req, res) => {
gfs.delete(new mongoose.Types.ObjectId(req.params.id), (err, data) => {
if (err) return res.status(404).json({ err: err.message });
res.redirect("/");
});
});
Here we get the id as a string so that needs to be converted into a mongodb objectid and then only the bucket method can delete the file with the corresponding id. I kept things simple by not using the delete HTTP method here you are free to use it if you feel like, a post request just works fine here.
Conclusion
As we can see MongoDB provides a nice solution to store files on the database and can come handy while creating WebApps with less storage facility, but keep in mind you can only store documents up to 16mb.
Give the post a like and star the repo if it helped you.
Top comments (28)
The only thing that i can do is thank you for such amazing post . Today I was learning how to manage the images files with nodejs and mongodb , I saw some tutorials but some of them didn´t work and there were others that I couldn´t understand , so I was a little stuck . But your post solved me a lot of doubts and luckily I figure out and could do it .
The only thing that i didn´t get was this part : function readSingleFile(e) {const name = e[0].name;document.getElementById("file-label").textContent = name;}
I´ve not get why is this for . Anyway , the post is amazing , thank you .
Thanks glad this could help.
Basically what that part is doing is that i am getting the file name and displaying it on the input element, its not required just some code to complete the tutorial, can be skipped.
Thanks for the answer !
Error: Cannot find module 'ejs '
Thanks alot. This is very helpful tutorial for me.I am new in this field. I am facing above error and have installed ejs by npm and following line by line your code.
I can upload files but can't retrieve it . After writing ejs code of retrieving files i am getting this error.I need your help.
did you try installing ejs? it can be done using npm, you can run
npm install ejsto install ejsYes I have done that already and even tried to install it globally.Actually, it just works fine if I pass only json data to index.ejs but error occurs only after typing the below code in index.ejs file:
Code link : dev-to-uploads.s3.amazonaws.com/i/...
Error link : dev-to-uploads.s3.amazonaws.com/i/...
Can you share your GitHub repo, if you have one where you where trying it, i can maybe take a look for you 😀
Sure . Here it is
github.com/krishna-y2000/Uploading...
Server file of above code is in /routes/uploadFiles.js and /routes/upload.js.
Client file is in /views/HomePage.ejs and /views/upload.ejs
Update in error :
Please refer this image because the comment box is not responding properly .
dev-to-uploads.s3.amazonaws.com/i/...
You can mail me if you want to text me in detail.
Email : krishnay.75676@gmail.com
Thanks alot for your effort.
THANK YOU BUDDY FOR MAKING THIS VALUEABLE DOCUMENT
Glad it helps
Thank you for this post sir.. Now I can upload my files to my mongo-atlas cluster successfully. Its very kind of you on my reach to this. I can now see the uploaded collections binary data on the collections (upl.files , upl.chunks) in mongo-atlas.
But I can't find a way to download a pdf file that i uploaded sir...Can u please help me on how to download a uploaded pdf file to my database sir. :{ Help sir.
Hi Syed, you can do something like this to download a pdf or any other file for this matter
I have created a route "image/:filename" which then downloads the image which is not limited to a image and can handle any other file types also, but for this demo I was only downloading images.
Thank you so much for your help sir. What I have to do if i need to download the file that the user uploads from his end, and I have to download that from the database collection but not from the user side itself. if i'm not wrong, Like in google classroom app., we upload our assignments as .pdf to our professor and those attachments reaches him/her in their database(provided by that app.) and they download them on their end..... Is there any way to implement like this, sir. Any help would be appreciated.
Hey, thanks for this blog. I have implemented this but the file is not being able to download it says couldn't download.
Absolutely Marvelous. Thank you so much sir, I've been really scratching my head with this. My question is, how do you add other parameters? So the image has a name?
Image does have a name, when we where setting up the GridFS as per Multer I used crypto and randomBytes to create a filename for the file, you can have your own strategy for naming.
following code was written if you missed it:
Yes, thanks but I mean when the user uploads the image, how can they add other fields? The image will need a unique filename, but can there be a title and description?
Yes sure why not, but then that needs to be in other collection and you can reference the image objectid in the collection where you are storing the meta information for the images
Yes I see, so you have to save the file location and file name and add that as a string to the other collection. Thanks for your help
File is being saved in its own collection as per gridfs, you can get an objectid for the image uploaded you can reference this objectid
Hello! Thank you for this helpful post! I have an issue I need some help with though.
I was able to successfully upload the images into MongoDB. However, I am not having much success retrieving the images and sending it to the frontend. The error I am currently getting is
TypeError: Cannot read property 'find' of undefined..this is happening at my ' gfs.find().toArray((err, files) => ' line under my 'app.get('/profilePicture' )' request. ..`
const express = require('express')
const path = require('path')
const crypto = require('crypto')//to generate file name
const mongoose = require('mongoose')
const multer = require('multer')
const GridFsStorage = require('multer-gridfs-storage')
const Grid = require('gridfs-stream')
const app = express()
//CORS Middleware
app.use(function (req, res, next) {
//Enabling CORS
res.header('Access-Control-Allow-Origin', '*');
res.header('Access-Control-Allow-Methods', 'GET,HEAD,OPTIONS,POST,PUT');
res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept, x-client-key, x-client-token, x-client-secret, Authorization');
next();
});
const uri = "mongodb+srv://fakeaccount:password@xxxxx.mongodb.net/x_app?retryWrites=lkngdndfg"
let conn = mongoose.connection
var gfs
conn.once('open', () => {
//initialize our stream
gfs = Grid(conn.db, mongoose.mongo)
gfs.collection('employee')
})
let storage = new GridFsStorage({
url: uri,
file: (req, file) => {
return new Promise(
(resolve, reject) => {
const fileInfo = {
filename: file.originalname,
bucketName: "imageUpload"
}
resolve(fileInfo)
})
const upload = multer({ storage })
app.post("/uploadImg",upload.single("profilePic"),(req,res)=>{
res.json({file:req.file})
})
app.get('/proficPicture', (req, res) => {
Amazing post! I referred to this to implement one of my projects.
Eventually, I put together a reusable Mongoose Schema which further simplifies Mongoose and MongoDB GridFS setup. It is available as an NPM module called GridFile. It also supports the Mongoose schema association and query population.
Here is an example of how to use GridFile to upload/download files to/from MongoDB GirdFS.
Hi,
thanks' for the blog post - its good to have a working example, however I want to take it a little further as I need to add file uploads to my own project and so must refactor your code to fit my app structure. Somewhere along the way I am missing some synchronisation. Upload triggers the file create and immediately shows the home page. If I press refresh the missing image is shown.
I think the error will be in my files.js /upload route - probably the next() is in the wrong place or the fact that I have an app.use nested inside my app.post...
I opened a Stack Overflow question and then saw I could perhaps ask here too :-)
stackoverflow.com/questions/628608...
My ‘fork’ of your repo is here: github.com/stevegroom/redogridfsst...
regards
Steve
why cant we add data more than 16 MB
It is limited from MongoDb end. There are some cases listed down by mongodb where you can use this.
Although this is true for MongoDB itself, GridFS was designed to solve this problem. From Mongo's documentation directly it states: "In MongoDB, use GridFS for storing files larger than 16 MB." Reference: docs.mongodb.com/manual/core/gridfs/
I am using react as a front end and I want to upload profile image with other information such as name, email etc. How to do this stuff, Please help me