DEV Community

Cover image for Parsing CSV Files in Node.js with fs.createReadStream() and csv-parser

Parsing CSV Files in Node.js with fs.createReadStream() and csv-parser

Isa Levine on September 02, 2019

Cover image credit: Hunter x Hunter manga by Yoshihiro Togashi, meme-ified by yours truly. <3 Ah, good ol' CSV files: spreadsheet/database data...
Read full post
Collapse
 
lejepra profile image
Lejepra •

Hi Isa!

Great code!
Thank you for sharing that!

I am using this code, but I'm stuck...
I did the imports and everything, and I push the csv processed data into an array. After the csv processing I with to consume the data from the array, but the array is empty. Please see my code below:

fs.createReadStream(csvFilePath, 'utf8')
  .pipe(csv())
  .on('data', async row => {
    const { title, type, value, category }: CsvTransactions = row;
    console.log(row);
    transactionsArray.push({ title, type, value, category });
    console.log('transactionsArray inside loop: ', transactionsArray);
  })
  .on('end', () => {
    console.log('Csv file successfully processed!');
    return transactionsArray;
  });
await console.log('transactionsArray outside loop: ', transactionsArray);
Enter fullscreen mode Exit fullscreen mode

The console.logs show me that the csv processing is going fine, but the last console.log that should show me the final array with the comment 'transactionsArray outside loop: ' is printed before the processing occurs:
csvFilePath: E:\tmp\transactions.csv
transactionsArray outside loop: []
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' }
transactionsArray inside loop: [
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' }
]
{
title: 'Cellphone',
type: 'outcome',
value: '50',
category: 'Others'
}
transactionsArray inside loop: [
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' },
{
title: 'Cellphone',
type: 'outcome',
value: '50',
category: 'Others'
}
]
{ title: 'Ice cream', type: 'outcome', value: '3', category: 'Food' }
transactionsArray inside loop: [
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' },
{
title: 'Cellphone',
type: 'outcome',
value: '50',
category: 'Others'
},
{ title: 'Ice cream', type: 'outcome', value: '3', category: 'Food' }
]
Csv file successfully processed!

I understand that fs.createReadStream is an async operation, but I am struggling on how to make the code wait for that to finish, before going on with the execution.
Please help!

Thank you!

Collapse
 
isalevine profile image
Isa Levine •

Hi Lejepra! I didn't actually use async / await in my own example code, so I'm basically guessing here.

I'd try moving the async to before the fs.createReadStream call, and seeing if that forces the await call to let the CSV-parsing finish.

Let me know if you're still working on this, and what you find out! :)

Collapse
 
rsjhon90 profile image
Jhony Rodrigues •

Hi. I came across this situation in an academic project and we had to use the traditional Promise() constructor.
csv-parser does not support promises natively. Same for FS stream and console.log methods
Your code would be more or less like this, trying as hard as possible to make it look like it.

function loadTransactions(): Promise<CsvTransactions[]> {
  const transactions: Promise<CsvTransactions[]> = new Promise((resolve, reject) => {
      const transactionsArray: CsvTransactions[] = []

      fs.createReadStream(csvFilePath, 'utf8')
        .pipe(csv())
        .on('data', row => {
          const { title, type, value, category }: CsvTransactions = row;
          console.log(row);
          transactionsArray.push({ title, type, value, category });
          console.log('transactionsArray inside loop: ', transactionsArray);
        })
        .on('end', () => {
          console.log('Csv file successfully processed!');
          resolve(transactionsArray)
        })
        .on('error', (err) => {
          reject(err)
        })
    }
  )

  console.log('transactionsArray outside loop: ', transactions);

  return transactions;
}
Enter fullscreen mode Exit fullscreen mode

I'll take the opportunity and thank the author for the post that clarified other things for me.
Thank you!

Collapse
 
rohit114_1 profile image
Rohit Kumar • • Edited

I have fixed this issue by using Promices,
that issue because the method in createReadStream is executed asynchronously
function createReadStream(){
var csvData = [];
return new Promise(resolve => {
fs.createReadStream('myfile.csv')
.pipe(csv())
.on('data', (data) => csvData.push(data))
.on('end', () => {
resolve(csvData)
});
})
}

const finalData = await createReadStream()

Collapse
 
bolugbogi profile image
Bolugbogi •

I tried the code above but it is telling me

TypeError: csv is not a function
at Object. (C:\Users\user\Desktop\Kepler Project\index.js:19:9)
at Module._compile (node:internal/modules/cjs/loader:1126:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
at Module.load (node:internal/modules/cjs/loader:1004:32)
at Function.Module._load (node:internal/modules/cjs/loader:839:12)
at Function.executeUserEntryPoint as runMain
at node:internal/main/run_main_module:17:47

Collapse
 
lclc68lclc profile image
Willina Clark •

I'm running this. Is there something weird with Dates and CSV, for some reason when I try to console.log my statements, Date reads as undefined and the others read fine.

Collapse
 
isalevine profile image
Isa Levine • • Edited

Hi Willina! As-is, the dates should just be processed as strings, so if the other string is printing fine, the date should too...

Just to test, I created a new directory, did a local install of csv-parser, and here's the code I'm running (copied verbatim):

// app.js

const csv = require('csv-parser');
const fs = require('fs');

const filepath = "./example_data.csv"

fs.createReadStream(filepath)
  .on('error', () => {
    // handle error
  })

  .pipe(csv())
  .on('data', (row) => {
    let str = `${row["BUYER NAME"]} bought ${row["CANDY PURCHASED"]} pieces of candy on ${row["PURCHASE DATE"]} and paid $${row["CASH PAID"]}.`;
    console.log(str)
  })

  .on('end', () => {
    // handle end of CSV
  })



// example_data.csv

PURCHASE DATE,CANDY PURCHASED,CASH PAID,BUYER NAME
2016-04-03,1000,10000.11,Charlie Kelly
2017-11-14,1000,12000.22,Frank Reynolds
2018-01-20,2000,40000.33,Frank Reynolds
2018-03-20,2000,40000.44,Mac
2019-01-02,2000,50000.55,Sweet Dee
2019-01-02,1500,13500.66,Dennis Reynolds



// console

Isas-MacBook-Pro:node-csv-parser isalevine$ node app.js 
Charlie Kelly bought 1000 pieces of candy on 2016-04-03 and paid $10000.11.
Frank Reynolds bought 1000 pieces of candy on 2017-11-14 and paid $12000.22.
Frank Reynolds bought 2000 pieces of candy on 2018-01-20 and paid $40000.33.
Mac bought 2000 pieces of candy on 2018-03-20 and paid $40000.44.
Sweet Dee bought 2000 pieces of candy on 2019-01-02 and paid $50000.55.
Dennis Reynolds bought 1500 pieces of candy on 2019-01-02 and paid $13500.66.

Any discrepancies between the code above and what you're running? Feel free to reply with the code itself and I can take a look :)

Collapse
 
lclc68lclc profile image
Willina Clark • • Edited

I'm sorry, I was running this with my own data, everything shows up except the date which is formatted like this: 2017:12:16 12:45. The header shows up, but when I try to access the rows, I get undefined. Could the formatting be the issue?

Thread Thread
 
isalevine profile image
Isa Levine •

Hi Willina, I'm so sorry I missed your reply!! I hope you were able to get unstuck--if not, I'd be happy to take a look at the data you were using, and see if we can find what's resulting in undefined! :)

Collapse
 
housseyn_cheriet profile image
Housseyn •

Try this package:
npmjs.com/package/select-csv

Collapse
 
marcoazar18 profile image
marcoazar18 •

Hello, If I use this code in a helper function, how can I return the array to the main function without doing console.log? Thank you for helping