Data loss while sending large set of data over WiFi

Troubleshooting
1

Hi folks!

I am trying out a scenario wherein I capture huge amount of data using ADC (about 20,000 unsigned short values), and send it across to my pre-defined server over WiFi. I am trying to send all the data in a single for loop, using client.print(), where client is my TCP connection to the server, which internally translates to sock_send() over WiFi, in the firmware. I see that sometimes a part of data does not reach the server at all, and sometimes things just work fine. It seems very unpredictable. I am not really sure where to look at.

Couple of question on this.

  1. If the “WiFi buffer” is full, and the program tries to pump in more data, how does the firmware handle such a situation?
  2. If the send() fails for whatever reason, does the firmware try to resend the data? Or should this be handled at the application itself?

Please let me know of any pointers on this front. It’ll be really helpful.
Thanks.

2

The firmware will wait for data to be sent before sending any more data so it’s not possible to go too fast. If send() fails then it most likely will always fail if it’s due to a networking problem. If it’s local to the system, perhaps memory allocation failures might be the cause.

How do you know the data is missing?

3

@mdma Sorry for replying a bit late, but here are the details.

I have been trying to send 20K data samples using Client.print(), to a predefined server (or about 100Kbytes after print() converts them to a string, 4 digit number each and a delimiter, per sample). I have tried multiple approaches

  1. Tried sending all the data in a single for loop. Works well most of the times. Sometimes I only see ~19K samples reaching the server, and the number varies.
  2. Tried grouping the data in packets of 1K bytes, and do a single client.write() - Same observation where all the data does not reach my server side script. Varied the size from 500

Correct me if I am wrong, but I believe the status returned by client.write() isn’t handled properly. write() uses size_t as the return type, which is unsigned, but returns -1 if status() returns a 0. Why size_t and why not an int?
size_t TCPClient::write(const uint8_t *buffer, size_t size)
{
return status() ? socket_send(_sock, buffer, size) : -1;
}

Also, what is behavior of the firmware if the write fails, for whatever reason? I do not see any retry being handled here.

Owing to this, I changed my code to look at the return value from write() and if it is -1, then try re-sending the data. I also started to log the number of retires that was happening. I have seen the retry value to range from 0 to 10K counts, and as I said earlier, it varies (guess depending on the network load, etc…). After these changes, I see that the server sometimes receives more than 20K samlples sometimes, and works well most of the times.

I then added delay(100) in between multiple sends, and the data consistency has significantly improved, but I don’t think using delays() is the right thing to do. I also have made sure to use SYSTEM_THREAD(ENABLED). I still am looking at making it robust to be able to deliver all the samples, all the time.

Any idea what could be causing the data loss?

1 Like
4

The negative return code has indeterminate behavior with respect to what if anything was actually sent. Particularly without access to wiced sdk I don’t think you can assume anything, so it would be best to tear down the connection and re-establish it.

The broadcom community boards indicate that other people have had issues in this space. It’s not entirely clear what the solution is.

So, chunking your sends into smaller pieces and incorporating a delay may be necessary to throttle large sends to avoid an error.

1 Like