Local Compiler Error that adds comma to my code, fails

So I ran using this:

readTransformWrite(inputStream, outputStream, transform) {
		return new Promise((fulfill, reject) => {
			const chunks = [];
			inputStream.on('readable', () => {
				try {
					let data;
					// eslint-disable-next-line no-cond-assign
					while (data = inputStream.read()) {
						chunks.push(data.toString('utf8'));
					}
				} catch (error) {
                    console.log(error);
					reject(error);
				}
			});

			inputStream.on('end', () => {
				try {
					const content = chunks.join();
                    const transformed = transform(content);
                    if (transformed != content) {
                        console.log("- %s", content);
                        console.log("+ %s", transformed);
                    }
					outputStream.end(transformed);
				} catch (error) {
                    console.log(error);
					reject(error);
				}
			});

			inputStream.on('error', reject);
			outputStream.on('error', reject);
			outputStream.on('finish', fulfill);
		});
	}

And I discovered that content aka what was fed into the preprocessor, contained the comma!

something within the inputStream / chunks operations. I didn’t catch any errors. I’ll see if I can flag on chunks containing commas

1 Like

ah, it could be:

const content = chunks.join();

for example:

var x = ['x', ''];
x.join(); // => "x,"
x.join(''); // => "x"

there might be more going on beyond that bit though :man_shrugging:

Interesting, so the comma appears in chunks but does not appear in data, nor in data.toString('utf8'). Thus the comma is created during the chunks.push(...) operation

So when I run this:

inputStream.on('readable', () => {
				try {
					let data;
                    // eslint-disable-next-line no-cond-assign
					while (data = inputStream.read()) {
                        chunks.push(data.toString('utf8'));
                        console.log("data: \n%s", data);
                        console.log("data.toString('utf8'): \n%s", data.toString('utf8'));
                        console.log("chunks: \n%s", chunks);
                    }
                    console.log("\n\n\n");
				} catch (error) {
                    console.log("error is %s", error);
					reject(error);
				}
			});

The file gets split on the line before the one that ultimately receives the comma.

The comma appears in chunks between the two file halves when the second half of the file is added using chunks.push(...) for the second half.

Edit: so I assume that the console.log function probably is doing some kind of array join behind the scenes in order to output a string version of the array? Not sure if it’s a characteristic of .push or .join

BOOM. You’re right.

const content = chunks.join();

should be changed to:

const content = chunks.join("");

And then everything works as expected!

4 Likes

:tada::clap::pray:

really big thanks for helping to run that down. i’ll see about getting a bugfix merged and released ASAP.

1 Like

I’m a little surprised that this hasn’t come up for other folks before though (or even myself, since I have had even longer files recently). So it feels like there may be some weird condition where this doesn’t happen, but regardless it seems like joining explicitly with the "" seems like the mitigation.

2 Likes

Wow, what great detective work @justicefreed_amper. Thanks!

3 Likes

I’m a little surprised that this hasn’t come up for other folks

well if i understand the underlying issue, you'll only get more than one chunk when you try to preprocess a file larger than 16kb and helpfully ['x'].join(); // => 'x' so it seems most are working with smaller files. :sweat_smile:

Is that your way of kindly telling me to split my code up more? :joy:

Yeah, that makes sense in general, though I’m honestly surprised it worked as long as it did for me - the comma must have been lodged in a block comment haha.

1 Like