Photon udp sends "�" characters at the start and end of a file

Troubleshooting
1

I’m trying to write a program to read a file off of a photon board and send it to a node.js server on a computer, which then saves it to a file.

Here’s my photon code:

#include <SdFat.h>

#include "SdFat.h"
#include <string.h>

#define SD_CS_PIN SS

UDP Udp;
SdFat SD;
SdFile myFile;

IPAddress nodeServer(172,20,10,2);
int port = 33333;

char filename[20] = "sensor0000.TXT";

int counter;
char data[100];

void setup() {
    
    if (!SD.begin(SD_CS_PIN)) {
        Serial.println("initialization failed!");
    }
  
    for (int i = 1; i < 10000; i++) {
    
        filename[6] = '0' + i/1000;
        filename[7] = '0' + (i/100) % 10;
        filename[8] = '0' + (i/10) % 10;
        filename[9] = '0' + i % 10;
        
        //Particle.publish("trying filename", filename);
    
        if (! SD.exists(filename)) {
            i = i - 1;
            
            filename[6] = '0' + i/1000;
            filename[7] = '0' + (i/100) % 10;
            filename[8] = '0' + (i/10) % 10 ;
            filename[9] = '0' + i % 10;
            
            Particle.publish("using filename", filename);
            
            break;
        }
    }

    if (!myFile.open(filename, O_READ)) {
        return;
    }
	
	Udp.begin(port);
}


void loop() {
    counter = 0;
    while ((data[counter] = myFile.read()) >= 0) {
        Particle.process();
        counter++;
        if (counter >= 100) {
            break;
        }
    }
    
    Particle.publish("sending packet", data);
    Udp.sendPacket(data, sizeof(data), nodeServer, port);
}

Node.js code:

var PORT = 33333;
var HOST = '172.20.10.2';

const fs = require('fs');
const dgram = require('dgram');
const server = dgram.createSocket('udp4');

const stream = fs.createWriteStream('./test.txt');

server.on('listening', function() {
    var address = server.address();
    console.log('UDP Server listening on ' + address.address + ':' + address.port);
});

server.on('message', function(message, remote) {
    stream.write(message);
});

server.bind(PORT, HOST);

This works for most of the file, but for some reason the start and end of the file (first and last 20 lines or so) are completely transformed into a long string of � characters. The file also doesn’t appear to end, when I tested it with a smaller file it kept sending packets of � after all of the file data I wrote had been sent.

The only issue I could think of would be that the start of the file has words while the rest is just numbers, but some of the numbers near the beginning and end are corrupted also and the end doesn’t have words or characters besides numbers, so that doesn’t really work.

Thanks in advance for any help!

2

so I changed the code a little bit and either made it better or worse, i’m not sure right now.

here’s the new photon code:

#include <SdFat.h>

#include "SdFat.h"
#include <string.h>

#define SD_CS_PIN SS

UDP Udp;
SdFat SD;
SdFile myFile;

IPAddress nodeServer(172,20,10,2);
int port = 33333;

char filename[20] = "sensor0000.TXT";

int counter;
char data[500];

void setup() {
    
    if (!SD.begin(SD_CS_PIN)) {
        Serial.println("initialization failed!");
    }
  
    for (int i = 1; i < 10000; i++) {
    
        filename[6] = '0' + i/1000;
        filename[7] = '0' + (i/100) % 10;
        filename[8] = '0' + (i/10) % 10;
        filename[9] = '0' + i % 10;
        
        //Particle.publish("trying filename", filename);
    
        if (! SD.exists(filename)) {
            i = i - 1;
            
            filename[6] = '0' + i/1000;
            filename[7] = '0' + (i/100) % 10;
            filename[8] = '0' + (i/10) % 10 ;
            filename[9] = '0' + i % 10;
            
            Particle.publish("using filename", filename);
            
            break;
        }
    }

    if (!myFile.open(filename, O_READ)) {
        return;
    }
	
	Udp.begin(port);
}

int readFromFile() {
    counter = 0;
    while ((data[counter] = myFile.read()) >= 0) {
        Particle.process();
        counter++;
        if (counter >= 100) {
            break;
        }
        if (myFile.peek() == -1) {
            return 0;
        }
    }
    
    Particle.publish("sending packet", data);
    Udp.sendPacket(data, sizeof(data), nodeServer, port);
    return 1;
}

void loop() {
    if (readFromFile() == 1) {
        Particle.publish("still reading");
    }
}

it just sends it in packets of 500 instead of 100 characters and it’s supposed to end better.
here’s the weird bit: instead of just the beginning and end, now there are the weird characters throughout the entire file:

Screenshot_2019-07-17_13-59-08
Screenshot_2019-07-17_13-59-081920×1047 226 KB

Here’s the even weirder part:
if I copy/paste part of the file into something in chrome (discord or google docs) it changes back to the correct numbers.

I don’t even know what’s going on here because I just tried to copy paste into this post edit box to prove my point and it showed up as numbers on one side and broken on the other side.

Screenshot_2019-07-17_14-04-27
Screenshot_2019-07-17_14-04-271920×1047 303 KB

Help appreciated! I genuinely have no clue what’s happening here.

3

@bean, is the data you read from the file binary or ascii? Also, using sizeof(data) will return the size of the array (500) and not the size of the data in the array. This explains why you get the longer output. Instead, you should use your byte count value counter when calling
sendPacket(), more specifically counter-1 since you post increment it after the file read.

1 Like
4

Just some style notes

  • You have multiple includes for the same library - that immediately suggests that’s just some code hastily clobbered together without much thought about what makes sense and what not (often a red flag for seasoned programmers)
  • to construct a filename with leading zeros, try snprintf(filename, sizeof(filename), "sensor%04d", i); instead of doing it with multiple divisions and modulo operations
  • always initialise a buffer when you get the feeling it’s dirty (which may be the reason for your “funny” characters) - memset(data, 0x00, sizeof(data));

BTW, there are multiple encoding schemes around. Copy/pasting content from an application using one scheme into another application using another can cause unexpected results.
Some applications replace “unexpected” characters with blank or omit them entirely, others replace them with a specific substitute character like �

1 Like
5

The data is ascii, thanks for the tip to use counter instead of sizeof().

6

you’re absolutely right about my style although it’s less that I wrote this in a hurry and more that all of my code is generally messy.

It worked after I initialized the buffer, thanks for the tip.

1 Like