Issue streaming bytes via TCP

Troubleshooting
1

Hi everyone,

I am trying to send bytes from my computer to the core (RGB encoding bytes) and store them in the flash memory (in an array) to use them later, to control LED strips.

However, it seems that most of the time only the first byte is going through (sometime, the stream works but rarely), then the stream is blocked and then the execution go through the last part of the code printing “Done” in the serial port although I think the thread should not reach this part since some bytes are missing.

I really don’t get it, and I am beginning to wonder if it is possible to stream accurately bytes without losses.

Moreover, when the transfer is completed, my Spark Core loose the connection, and start blinking fast red, then fast blue. I put Particle.process() along my code to prevent the Spark core to disconnect from the cloud but still the problem persist.

Thanks for your help, I am really confused with the TCP connection!

void loop()
{
  if (client.connected() && !done ) {
     while (i< 428){  
         if(client.connected() && client.available()){
         BlueStack[i] = client.read();
         Serial.println(BlueStack[i]);
         //delay(10);
         }
          else { 
         client = server.available();
         Particle.process();
          }
           if(client.connected() && client.available()){
         RedStack[i] = client.read();
         Serial.println(RedStack[i]);
         //delay(10);
           }
          else { 
         client = server.available();
         Particle.process();
          }
        
          if(client.connected() && client.available()){
         GreenStack[i] = client.read();
         Serial.println(GreenStack[i]);
         //delay(10);
          }
          else { 
         client = server.available();
         Particle.process();
          }
         
         i++;
       /* LedStack[j] = client.read();
        LedCount[j] = client.read();
        j++ */
        Particle.process();
     }
     


    Serial.println("C'est fini");
    for (int h = 0; h<428; h++){
        Serial.print(RedStack[h]);
        Serial.print(GreenStack[h]);
        Serial.println(BlueStack[h]);
    }
    done = true;
   
  } else {
    // if no client is yet connected, check for a new connection
    client = server.available();
    Particle.process();
  }
}
2

Hi again,

Continuing on the same topic, I am now trying to send bytes and store them in the flash memory to use them later, to control LED strips. However, as it appear on this screenshot, some bytes are lost during the transfer (serie of 255). I thought the TCP protocol prevent this from happening (contrary to UDP) so I don’t understand what is happening… Moreover, when the transfer is completed, my Spark Core loose the connection, and start blinking fast red, then fast blue. Following your previous advice, I put a lot of Particle.process() along my code to prevent that from happening but it doesn’t seem to work.

Screen Shot 2015-11-25 at 22.58.43.png2766×1656 330 KB

Here is my code:

void loop()
{
  if (client.connected() && !done ) {
     while (i< 428){  
         if(client.connected()){
         BlueStack[i] = client.read();
         Serial.println(BlueStack[i]);
         //delay(10);
         }
          else { 
         client = server.available();
         Particle.process();
          }
           if(client.connected()){
         RedStack[i] = client.read();
         Serial.println(RedStack[i]);
         //delay(10);
           }
          else { 
         client = server.available();
         Particle.process();
          }
        
          if(client.connected()){
         GreenStack[i] = client.read();
         Serial.println(GreenStack[i]);
         //delay(10);
          }
          else { 
         client = server.available();
         Particle.process();
          }
         
         i++;
       /* LedStack[j] = client.read();
        LedCount[j] = client.read();
        j++ */
        Particle.process();
     }
     


    Serial.println("C'est fini");
    done = true;
   
  } else {
    // if no client is yet connected, check for a new connection
    client = server.available();
    
  }
}
3

Could you also provide the rest of your code - especially the declarations of your vars?

Could you try this approach

const int maxCount = 428;
const uint32_t TIMEOUT = 1000;

void loop()
{
  if (client.connected()) 
  {
     if (done) return;

     uint32_t msTimeout;
     // I assume "i" will be reset somewhere else?
     for(; i < maxCount; i++)
     {  
        msTimeout = millis();
        while(!(client.available() >= 3) && (millis() - msTimeout <= TIMEOUT))
           Particle.process(); // wait for at least 3 byte to read (max 1sec);

        if(millis() - msTimeout > TIMEOUT)
        {
           Serial.printlnf("Timeout occured at %d", i);
           break;
        }
        BlueStack[i]  = client.read();
        RedStack[i]   = client.read();
        GreenStack[i] = client.read();

        // why BRG? RGB would be normal and BGR somehow understandable ;-)
        // alternative for one "int rgb[]"
        // rgb[i] = client.read() << 08 || client.read() << 16 || client.read() << 00; // for BRG send order
        // rgb[i] = client.read() || client.read() << 8 || client.read() << 16; // for RGB
        // execution order right to left

        Serial.printlnf("%3d %3d %3d", BlueStack[i], RedStack[i], GreenStack[i]);
        //Serial.printlnf("%#08X", rgb[i]); // HEX notation as  0x00RRGGBB

        Particle.process();
     }
     Serial.println("C'est fini");
     done = (i >= maxCount);  // only done when all data received
  } 
  else
  {
    // if no client is yet connected, check for a new connection
    client = server.available();
  }
}
3 Likes
4

Hi ScruffR,

First I would like to thank you for your help, I really appreciate!
Here is the beginning of my code:

int i = 0;
int j = 0;
byte RedStack[450];
byte GreenStack[450];
byte BlueStack[450];

byte LedStack[450];
byte LedCount[120];

byte currentbyte;
bool done = false;


Adafruit_NeoPixel strip = Adafruit_NeoPixel(PIXEL_COUNT, PIXEL_PIN, PIXEL_TYPE);

TCPServer server = TCPServer(4000);
TCPClient client;
byte r;

void setup()
{
    
  strip.begin();
   // start listening for clients
  strip.setPixelColor(140, strip.Color(0,0,100));
  strip.show();
  // start listening for clients
  server.begin();

  // Make sure your Serial Terminal app is closed before powering your device
  Serial.begin(9600);
  // Now open your Serial Terminal, and hit any key to continue!
  while(!Serial.available()) Particle.process();

  Serial.println(WiFi.localIP());
  Serial.println(WiFi.subnetMask());
  Serial.println(WiFi.gatewayIP());
  Serial.println(WiFi.SSID());
}

LED stack and LED count will be two other arrays I’ll also fill using the same streaming pipe architecture.

I tried your approach and the stream blocked again at some point (interestingly enough it seems that it always timeout at index 42). I change the code :

while(!(client.available() >= 3) && (millis() - msTimeout <= TIMEOUT))

in

while(!(client.available() >= 1) && (millis() - msTimeout <= TIMEOUT))

and I got better results, but still in most of my trials, at some point a byte is lost and that put an offset in the index messing with the whole array. I join an Excel spreadsheet with my results, the first 5 columns being with !(client.available() >=1) and the four next with !(client.available() >=3).

Excel with tests

Here is part of my code on client side that send the bytes, 3 by 3.

    Iterator<Color> itr = s.ColorStack.iterator();	
    		byte[] color_send = new byte[3]; 		
    		for(int i = 0; i < s.ColorStack.size(); i++){
			Color color = itr.next();
			color_send[0] = (byte)Math.floor(color.getBlue()/10);
			System.out.print(color_send[0] + " ");
			color_send[1] = (byte)Math.floor(color.getRed()/10);
			System.out.print(" " + color_send[1]+ " ");
			color_send[2] = (byte)Math.floor(color.getGreen()/10);
			System.out.println(color_send[2]);
			 for(int j = 0; j<3; j++){
				outToServer.writeByte(color_send[j]);
							}

It seems that bytes are lost in the process but the TCP protocol should prevent this. And it is quite confusing that the stream seems to always block at index 42.

5

I also noticed weird behaviour with the counter of the for loop:

With i declared as a global scope variable, using the syntax you wrote, i jump from 453 to 267!

Screen Shot 2015-11-28 at 19.58.19.png1370×1052 131 KB

If I declare i as a local variable for the loop, then i seems to increment normally but when it reach a certain value (between 400 and 550, the loop going to 618) the core starts blinking red and restart.

Screen Shot 2015-11-28 at 20.00.49.png1374×1060 127 KB 

void loop()
{
  if (client.connected()) 
  {
     if (done){
         Serial.println("done!");
     }

     uint32_t msTimeout;
     for(; i < maxCount; i++)
     {  
        msTimeout = millis();
        while(!(client.available() >= 1) && (millis() - msTimeout <= TIMEOUT))
           Particle.process(); // wait for at least 3 byte to read (max 1sec);

        if(millis() - msTimeout > TIMEOUT)
        {
           Serial.printlnf("Timeout occured at %d", i);
           client = server.available();
           break;
        }
        BlueStack[i]  = client.read();
        RedStack[i]   = client.read();
        GreenStack[i] = client.read();

        Serial.printlnf("%3d %3d %3d %3d", BlueStack[i], RedStack[i], GreenStack[i], i);
        //Serial.printlnf("%#08X", rgb[i]); // HEX notation as  0x00RRGGBB

        Particle.process();
     }
     Serial.println("C'est fini");
     done = true;  // only done when all data received
  } 
  else
  {
    // if no client is yet connected, check for a new connection
    client = server.available();
  }
}
6

I do see some issues here.

First, again you are not actually showing your whole code (where is the declaration for maxCount, PIXEL_COUNT, PIXEL_PIN, PIXEL_TYPE and where will done be set to false again once it became true?).
With a maxCount of 428 and even more your arrays declared as [450] you should never get an index >= 428 and even less >= 450. If you do, you are doomed to get corrupted variables (e.g counters) and eventually a red SOS.
I had written this comment in my code (since I didn’t see your whole code and any indication of you resetting i).

     // I assume "i" will be reset somewhere else?
     for(; i < maxCount; i++)
     ...

If you don’t (as said, I can’t see an indication of it in your other code either - I hate having to guess what happens where and how), you’d need to do this

     for(i = 0; i < maxCount; i++)
     ...

But then you will always restart from scratch when reentering loop().
Another way would be after the for()

     if (i >= maxCount) // finally we got all data
     {
       i = 0;
       Serial.println("C'est fini");
       done = true;
     }
     else 
       Serial.printlnf("only %d received yet", i + 1);

The next thing, you should not change this >= 3, since the following code relies on three bytes to be available to be read. If they aren’t you will cause displacement of your data.
Rather increase the TIMEOUT but moreover check why this happens. It might be due to your transmitting code overwhelming the buffer.
TCP only ensures that that packets get delivered to the WiFi module and get acknowledged there (which it obviously does) but maybe it even acknowledges them despite the possible risk of the receive buffer wrapping round and overwriting unread data.
Try adding a short delay in your transmitting code between each (or even only every 40th) color to give the device some time to empty the queue.

BTW: You have byte LedCount[120]; but write this strip.setPixelColor(140, strip.Color(0,0,100)). Have you got 120, 140, 428 or 450 LEDs on your strip now?
Exactly things like this are a good reason to use constants (like const int maxCount) and only use these all over to ensure that they are consistent all over, in case you’d decide to change things and to make clear if array dimensions and indexes are related or not.
Also naming indexes explicitly (rather than using global i and j) helps avoiding ambiguity and misunderstanding.
(dummy variables like i, j, … should only be used in limited scopes - e.g. for(int i; i<maxCount ;i++) is acceptable)

1 Like
7

I tried it today with a 20ms delay on the client side between each byte and it works :slight_smile:
Thank you so much ScruffR!

To be curious do you know how large is the buffer on the Core?

1 Like