DEV Community

Discussion on: Rate limiting API calls - sometimes a Bottleneck is a good thing

 
rcoundon profile image
Ross Coundon

Then I'm guessing there's something weird in the way you're building the URLs or making the requests when there are multiple API calls.

To clean things up, if I was you I'd change the code to use map() on the array of cvUrl returning a promise for each call.
The await promise.all() on the result of that map, then do your parsing.

Put console.logs in each iteration to determine exactly what you're sending and wrap in try/catch to see if you can find any more information about what's actually going wrong with the connection.

Thread Thread
 
melitus profile image
Aroh Sunday

This is where I tried it with Promise.all but got the same error. Is this code below look like what you suggested above?

const toReadInParalel = async arrayOfIds => {
  let cvUrl = await getUploadedCVUrl(arrayOfIds);
  let parsedResult = [];
  let save;
  let start = +new Date();
  let parserError;
  await Promise.all(
    cvUrl.map(async url => {
      const file = url.fileurl;
      const filename = url.filename;
      const jobpositionIdFromArray = url.jobPositionId;
      console.log({ file, filename, jobpositionIdFromArray });
      if (!file && !filename) return;

      let singleResult = await throttleApiCall(filename, file);
      console.log({ singleResult });
      const { data } = singleResult;
      if (data.Error) {
        parserError = true;
      }
      if (data) {
        let newparsed = new ParsersCollection(data);
        newparsed.jobPositionId = jobpositionIdFromArray;
        save = await newparsed.save();
        parsedResult.push(save);
      }
    }),
  );
Thread Thread
 
melitus profile image
Aroh Sunday

I need your assistance to get this resolve. Thanks

Thread Thread
 
melitus profile image
Aroh Sunday • Edited

@ross Coundon, I would appreciate your assistance from the wealth of your experience in the field interacting with several third part API on how I can make a concurrent request to the server without experience the "socket hangout issue". In the first interaction of the loop shown above, I will get results from the third-party API but on the second iteration, there will be a delay for a response from the third party and hence the error message below

Trace: { Error: socket hang up
    at createHangUpError (_http_client.js:323:15)
    at TLSSocket.socketOnEnd (_http_client.js:426:23)
    at TLSSocket.emit (events.js:194:15)
    at TLSSocket.EventEmitter.emit (domain.js:441:20)
    at endReadableNT (_stream_readable.js:1125:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  code: 'ECONNRESET',
Thread Thread
 
rcoundon profile image
Ross Coundon

Hi - I'm not sure what to suggest, are you able to share what the 3rd party API is? Do they provide any documentation/information on acceptable usage, time between requests, number of concurrent requests etc?

Thread Thread
 
melitus profile image
Aroh Sunday

They do not have that spell out on their API documentation. I have sent a mail to them to inquire about the acceptable usage, the time between requests, number of concurrent requests