*How to Implement *
File Upload Process:
The user will upload the file via /upload endpoint and API immediately returns a response: "File uploaded successfully
The file details (path, job_applicant_id) are added to a queue for processing.
A worker thread picks up the file and uploads it to S3 Bucket.
THe Path is stored in Database as well with the path name
Fetching the File:
1.When a user requests the file via and API(fetch/job_applicant_id/filepath) by the path name
2.The API generates a signed URL for temporary access and returns it
IMPLEMENTATION PART:
1. File Upload API
Uses multer to handle file uploads.
Adds the file path to a queue for processing.
Responds immediately without waiting for the file to upload.
const express = require('express');
const multer = require('multer');
const threadController = require('./threadController');
const app = express();
const upload = multer({ dest: 'uploads/' });
app.post('/upload', upload.single('file'), (req, res) => {
const { job_applicant_id } = req.body;
const filePath = req.file.path;
const fileName = req.file.originalname;
threadController.addToQueue(job_applicant_id, filePath, fileName);
res.json({ message: 'File uploaded successfully' });
});
app.listen(3000, () => {
console.log('Server running on port 3000');
});
2. Worker Thread for Uploading to S3
Picks up files from the queue.
Uploads them to S3 under job_applicant_id.
const { isMainThread, parentPort, workerData } = require('worker_threads');
const AWS = require('aws-sdk');
const fs = require('fs');
AWS.config.update({
accessKeyId: 'AWS_ACCESS_KEY',
secretAccessKey: 'AWS_SECRET_KEY',
region: 'us-east-1'
});
const s3 = new AWS.S3();
if (!isMainThread) {
const { job_applicant_id, filePath, fileName } = workerData;
const fileStream = fs.createReadStream(filePath);
const params = {
Bucket: 'my-file-bucket',
Key: `${job_applicant_id}/${fileName}`,
Body: fileStream,
ContentType: 'application/pdf'
};
s3.upload(params).promise()
.then(data => console.log(`File uploaded: ${data.Location}`))
.catch(err => console.error(`Upload failed: ${err.message}`));
}
3. Fetch File API
Retrieves a temporary signed URL to access the file.
app.get('/get-file/:job_applicant_id/:file_name', (req, res) => {
` const { job_applicant_id, file_name } = req.params;
const params = {
Bucket: 'my-file-bucket',
Key: `${job_applicant_id}/${file_name}`,
Expires: 60 * 5 // Link valid for 5 minutes
};
const url = s3.getSignedUrl('getObject', params);
res.json({ url });
});`
Top comments (0)