DEV Community

Discussion on: Parsing CSV Files in Node.js with fs.createReadStream() and csv-parser

View post
Collapse
 
lejepra profile image
Lejepra •

Hi Isa!

Great code!
Thank you for sharing that!

I am using this code, but I'm stuck...
I did the imports and everything, and I push the csv processed data into an array. After the csv processing I with to consume the data from the array, but the array is empty. Please see my code below:

fs.createReadStream(csvFilePath, 'utf8')
  .pipe(csv())
  .on('data', async row => {
    const { title, type, value, category }: CsvTransactions = row;
    console.log(row);
    transactionsArray.push({ title, type, value, category });
    console.log('transactionsArray inside loop: ', transactionsArray);
  })
  .on('end', () => {
    console.log('Csv file successfully processed!');
    return transactionsArray;
  });
await console.log('transactionsArray outside loop: ', transactionsArray);
Enter fullscreen mode Exit fullscreen mode

The console.logs show me that the csv processing is going fine, but the last console.log that should show me the final array with the comment 'transactionsArray outside loop: ' is printed before the processing occurs:
csvFilePath: E:\tmp\transactions.csv
transactionsArray outside loop: []
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' }
transactionsArray inside loop: [
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' }
]
{
title: 'Cellphone',
type: 'outcome',
value: '50',
category: 'Others'
}
transactionsArray inside loop: [
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' },
{
title: 'Cellphone',
type: 'outcome',
value: '50',
category: 'Others'
}
]
{ title: 'Ice cream', type: 'outcome', value: '3', category: 'Food' }
transactionsArray inside loop: [
{ title: 'Loan', type: 'income', value: '1500', category: 'Others' },
{
title: 'Cellphone',
type: 'outcome',
value: '50',
category: 'Others'
},
{ title: 'Ice cream', type: 'outcome', value: '3', category: 'Food' }
]
Csv file successfully processed!

I understand that fs.createReadStream is an async operation, but I am struggling on how to make the code wait for that to finish, before going on with the execution.
Please help!

Thank you!

Collapse
 
isalevine profile image
Isa Levine •

Hi Lejepra! I didn't actually use async / await in my own example code, so I'm basically guessing here.

I'd try moving the async to before the fs.createReadStream call, and seeing if that forces the await call to let the CSV-parsing finish.

Let me know if you're still working on this, and what you find out! :)

Collapse
 
rsjhon90 profile image
Jhony Rodrigues •

Hi. I came across this situation in an academic project and we had to use the traditional Promise() constructor.
csv-parser does not support promises natively. Same for FS stream and console.log methods
Your code would be more or less like this, trying as hard as possible to make it look like it.

function loadTransactions(): Promise<CsvTransactions[]> {
  const transactions: Promise<CsvTransactions[]> = new Promise((resolve, reject) => {
      const transactionsArray: CsvTransactions[] = []

      fs.createReadStream(csvFilePath, 'utf8')
        .pipe(csv())
        .on('data', row => {
          const { title, type, value, category }: CsvTransactions = row;
          console.log(row);
          transactionsArray.push({ title, type, value, category });
          console.log('transactionsArray inside loop: ', transactionsArray);
        })
        .on('end', () => {
          console.log('Csv file successfully processed!');
          resolve(transactionsArray)
        })
        .on('error', (err) => {
          reject(err)
        })
    }
  )

  console.log('transactionsArray outside loop: ', transactions);

  return transactions;
}
Enter fullscreen mode Exit fullscreen mode

I'll take the opportunity and thank the author for the post that clarified other things for me.
Thank you!

Collapse
 
rohit114_1 profile image
Rohit Kumar • • Edited

I have fixed this issue by using Promices,
that issue because the method in createReadStream is executed asynchronously
function createReadStream(){
var csvData = [];
return new Promise(resolve => {
fs.createReadStream('myfile.csv')
.pipe(csv())
.on('data', (data) => csvData.push(data))
.on('end', () => {
resolve(csvData)
});
})
}

const finalData = await createReadStream()