Reject Corrupt UTF-8 Before Reviewing an Order File

A file preview can look readable even after its decoder has replaced invalid bytes. If the damaged sequence was inside a product identifier or order note, reviewing only the resulting string can conceal the fact that the original bytes were not valid UTF-8.

For a Shopify 3PL import tool, decide the encoding contract before parsing rows. This example accepts a complete, nonempty UTF-8 byte array within an explicit size limit. It uses TextDecoder's fatal mode to reject malformed sequences instead of replacing them.

Validate bytes before interpreting columns

'use strict';
const assert = require('node:assert/strict');

function decodeOrderFile(bytes, maxBytes = 1_000_000) {
  if (!(bytes instanceof Uint8Array)) throw new Error('Byte array required');
  if (!Number.isSafeInteger(maxBytes) || maxBytes < 1) {
    throw new Error('Positive safe integer size limit required');
  }
  if (bytes.byteLength === 0 || bytes.byteLength > maxBytes) {
    throw new Error('File size outside configured limit');
  }
  return new TextDecoder('utf-8', { fatal: true }).decode(bytes);
}

const encode = text => new TextEncoder().encode(text);
const sample = 'sku,note\nSKU-1,蓝色 📦';
assert.equal(decodeOrderFile(encode(sample)), sample);
assert.equal(decodeOrderFile(Buffer.from('sku\nSKU-1')), 'sku\nSKU-1');
const bom = Uint8Array.from([0xef, 0xbb, 0xbf, 0x41]);
assert.equal(decodeOrderFile(bom), 'A');
assert.equal(decodeOrderFile(encode('abc'), 3), 'abc');
assert.throws(() => decodeOrderFile(encode('abcd'), 3));
assert.throws(() => decodeOrderFile(new Uint8Array()));
assert.throws(() => decodeOrderFile('sku')); 
assert.throws(() => decodeOrderFile(encode('A'), 0));
for (const bad of [[0xff], [0xc3], [0xe2, 0x82],
  [0xc0, 0xaf], [0xed, 0xa0, 0x80], [0xf4, 0x90, 0x80, 0x80]]) {
  assert.throws(() => decodeOrderFile(Uint8Array.from(bad)), TypeError);
}
assert.equal(new TextDecoder().decode(Uint8Array.from([0x41, 0xff, 0x42])),
  'A\ufffdB');
console.log('Strict UTF-8 file checks passed');

Save as strict-utf8.cjs and run node strict-utf8.cjs with Node.js 24. The checks cover non-ASCII text, emoji, Buffer input, a leading UTF-8 byte-order mark, size boundaries, invalid types, truncated sequences, overlong encoding, encoded surrogates and values beyond Unicode's range. The one-million-byte default is an illustrative configuration, not a claim about any platform's upload limit.

The decoder's default handling removes a leading UTF-8 byte-order mark in this example. The decode() method treats this input as one complete buffer, so an unfinished trailing sequence is rejected. A streaming decoder requires a separate chunk and final-flush policy.

Keep encoding validation separate from an import decision

A drop shipping file and a shopify fulfillment file can both be valid UTF-8 while using incompatible schemas. Successful decoding says nothing about headers, delimiters, required fields or whether an order may be released. Those need their own checks after this boundary.

For China fulfillment data, do not silently try another encoding after a failed UTF-8 check. Ask the file owner to confirm the actual export contract and provide a corrected file through the agreed channel. A China 3PL adapter should retain an identifiable file reference and a useful error without copying private customer rows into a public log.

If “International Fulfillment for Folligen” names a hypothetical integration brief, it still needs an explicit encoding and schema. I work with YX Fulfillment; its Shopify handover guide gives related operational context. These local tests verify decoding behavior; they do not demonstrate a live store connection or a deployed warehouse import.

Warehouse workstations and equipment.