Post binary data to A Serverside php cgi

General
1

Hi, I have a sparkcore with a connected NRF24L01 tranceiver. The NRF recieves data in 32 byte chunks, the data is an image. I would like to send the image to a server. I could use ftp but my server is configured to use SFTP (secure FTP) which is SSH in disguise which the spark cannot do.

So I thought no probs, I’ll create a php script to receive the data, issue is that I need to send the image 32 bytes at a time in a loop because the spark doesn’t have the memory to buffer the whole image then send it.

So ideally, I’d open a tcpclient connection to my php script, then ‘tcpclent.print’ in some header data, then loop, ‘tcpclent.printing’ in my 32 byte image chinks, then ‘tcpclent.print’ some footer data, then close the connection.

I have sent data url encoded using tcpclient before but I’m guessing for this app I need to use multipart/form-data. So in order to figure out what to print in my tcpclient ‘stream’, I found this page : Under the hood multipartFormData

Problem is I have no idea how to decipher this into valid tcpclient print statements. Can anyone help please? I have included some code which I have used to send simple data (myVar, int 14) but obviousely this task is a little more complex.

if(client.connect(server, 80))
    {
        connected = true;
        Serial.println("connected");
        client.println("GET /somescript.php?myVar=14 HTTP/1.0");
        client.println("Host: " server);
        client.println("Content-Length: 0");
        client.println();
        client.stop();
2

Ok, so this is what I have so far but it doesn’t work for some reason

#define server "xxx.com"
byte buffer[32];

void setup() 
{
    Serial.begin(9600);
    for(int u=0; u<32; u++)
        buffer[u] = (byte)u;
}

void loop() {
    delay(5000);
    if(client.connect(server, 80))
    {
        Serial.println(F("connected"));
        client.println("Host: " server);
        client.println(F("POST /yyy.php HTTP/1.0"));
        client.println(F("Accept: */*"));
        client.println(F("Content-Length: 1057"));
        client.println(F("Expect: 100-continue"));
        client.println(F("Content-Type: multipart/form-data; boundary=-----------------------c27a2baab8fe94bd"));
        client.println(F("--------------------------c27a2baab8fe94bd"));
        client.println(F("Content-Disposition: form-data; name='foo'"));
        client.println(F("bar"));
        client.println(F("--------------------------c27a2baab8fe94bd"));
        client.println(F("Content-Disposition: form-data; name='img'; filename='IMAGE00.JPG'"));
        client.println(F("Content-Type: image/jpeg"));
        for(int u=0; u<10; u++)
            client.write(buffer, 32);
        client.println(F("--------------------------c27a2baab8fe94bd--"));
        //client.stop();
        Serial.println("sent");
    } 
    else 
    {
        Serial.println("connection failed");
        delay(200);
    }
}
3

Ok, so it appears that the above code is resulting in the cyan flash of death as detailed Here
The core will run, print ‘connected’ to the serial then nothing. I believe that the client.connect is failing and hanging all network connection for potentially 60 seconds. This means that an over the air flash is impossible and I have to download the firmware binary and flash via usb. I think this is a TCPClient library issue, any comments welcome.

4

Where do you dump the data returned by the web host to the TCPClient?

If you don’t wait for the return data and then explicitly discard it, things are going to go badly since the TI part is going to run out of packet buffers while waiting to timeout the connection.

5

Could you uuencode/base64-encode the data and send in smaller chunks using client.println(...);? You’d have to reassemble on the PHP side, but that wouldn’t be too terrible.

2 Likes
6

Hi bko, I’m probably being dumb but I’m not retrieving data, I’m just trying to assemble a POST request and send it to a php script for processing. I’m not sure I understand which lines of code are supposed to be returning a value to me or what I’m supposed to do with that data. So far it’s not working, I’m having real issues trying to figure out what to print to the client in order to send the image, to achieve what I did in the code above I simply ran the python script indicated on the page about form data(linked in first post) and cloned the response into a sequence of client.println calls. I’m obviousely not getting something and to a seasoned coder like yourself that’s probably painfully obviouse but it’s a steep learning curve for me as I have limited knowledge of the tcpclient class and of http request format.

7

Hi wgbartly, I’m not sure how that would solve the issues I’m having, I don’t mind assembling it php side but this is a time critical opperation, I need to send an image a few K in size quickly to the server, it’s my understanding that using println would send one byte per packet, is that correct? that is why I want to send the entire 32 byte buffer at once, after all, there will be like 1000 of them :frowning:

8

bko, do you mean in my loop? do I need to wait for the write to complete before commencing the next write else I’ll overflow the out buffers by really quickly writing many time without waiting for each write/send to complete? I read in the docs about write :

“Returns: byte: write() returns the number of bytes written. It is not necessary to read this value.”

9

I had it in my head that maybe a client.println(...) might force it to send the current TCP buffer. I have know idea how or when I made it up, but it definitely doesn’t exist in the documentation! Ignore my post.

Maybe @bko is talking about checking for responses from the server in your loop and flushing them.

10

no probs wgbartley, I really appretiate the response, I’m kind of a loner sat in my man cave hacking out code and reading so much documentation my head hurts, but I’m getting there, every once in a while I fail to achive the current challenge but when that happens, this community is awsome, any and all suggestions are greatfully received :slight_smile:

11

There are a bunch of things that “non-optimal” here.

  • TCPClient.print() and println() send multiple packets leading to lots of packet buffers being used on the TI chip. If the cloud connection (or other WiFi connection) comes along and can’t get a packet buffer, it blocks and you get flashing cyan until it can get one.
  • The server is sending you bytes back and the TI chip is storing them, waiting for you to pick them up. Eventually this also can cause the cloud connection to run out of packet buffers too. And flashing cyan happens here as well.

In both of these cases, the TI chip will eventually timeout and things will work again, but if you are in a loop having these problems you can never get ahead of the game and win.

The solutions are to use TCPClient.write(buffer, buflength); which sends only one packet and to use two while loops after you send: the first one waits for TCPClient.available() to be non-zero meaning there is data from the server, and the second one reads that data using TCPClient.read() until TCPClient.available() is zero again. You can also add timeout to these while loops so you don’t get stuck forever.

I keep hoping to get some time to throw together a “good” example but if you look at the code from @mtnscott and @Hootie81 who have followed this path, you will see that you can get reliable TCPClient connections.

12

Wow, that’s a brilliant respone :slight_smile: I will beaver away now and see if I can conjour up some code to do just what you have suggested as you are bang on target with what I am actually experiencing. Do you mind verifying I have done it correctly if I post my code up when I’m done please?

1 Like
13

Hi bko, I have been trying to implement your suggestions and the results are as follows,
think I have managed to vanquish the tcp lockup by using your suggestion to use write instead of print but I had massive difficulty producing the header string to feed to write because there is no printf in arduino c so I simply removed the Content-Length header line because I could not combine an int into the string.

I read a lot of posts and found your response to @markopraakli here and used that method to feed my very long string into write. Aside from my mock server saying the request is a bad HTTP/0.9 request type(‘Host:’)400, I think I am on my way. Please could you look over this code and tell me if you think I am reading the response correctly so as to properly empty and free up the TI buffers. I know my test data buffer is just test data and is not very long but hey ho. I also am aware this request is bad and fails but I’m guessing no body can help with that so I’ll have to figure it out myself. I also don’t know how to do a CRLF in a string, I tried CRLF and \n, does anyone know please (ps: solved this, its \r\n)?

TCPClient client;
byte server[] = { 192, 168, 1, 64 }; // Google

int tcpTimeOut = 8000;
int uu = 0;
byte buffer[32];

void setup() {
   Serial.begin(9600);
   for(int u=0; u<32; u++)
     buffer[u] = (byte)u;
}

void loop() {
    delay(5000);
    if(client.connect(server, 8080))
    {    
        Serial.print(F("connected : "));
        Serial.println(uu, INT);
        String str = "Host: 192.168.1.64\nPOST /ttx.php HTTP/1.0\nAccept: */*\nContent-Type: multipart/form-data; boundary=------------------------c27a2baab8fe94bd\n--------------------------c27a2baab8fe94bd\nContent-Disposition: form-data; name='img'; filename='IMAGE00.JPG'\nContent-Type: image/jpeg\n";
   
      client.write((uint8_t*)str.c_str(), str.length());
      unsigned long m = millis();
      for(int u=0; u<3; u++)
        client.write(buffer, 32);
                           
      Serial.print("end buffer : ");
      Serial.println(millis() - m, INT);
      client.write(F("--------------------------c27a2baab8fe94bd--\n"));

       while(client.available())
           while(client.read() != -1)
        
        delay(400);
        client.stop();
        client.flush();
        Serial.println("sent");
    } 
    else 
    {
        Serial.println("connection failed : ");
        
        delay(400);
        client.stop();
        client.flush();
    }
    uu++;
}
1 Like
14

the CRLF would be \r\n

1 Like
15

Hi @rickpbush

Yes I think you are going to get much better results now. I have just two points.

  • The Arduino String class is very convenient but can bring memory problems since certain operations use memory dynamically which can lead to a problem called heap fragmentation. On a PC/Mac/Linux, the OS can recover but on Spark you have to take steps to avoid the problem. So you can print numbers easily in the String class using just something like str + String(aNumber) + " other string here"; but you might run into heap problems eventually. I would try it and see. If you can move to C char arrays instead of Arduino Strings and keep your memory entirely static, you will have full control.

  • I like to wait for the server to respond at all before looking for the returned data. Like this:

while(client.available() == 0) {}  //loop waiting for first data

while(client.available() != 0) {
  client.read();  //dump return data
}

If you look around at the forum, you can find code that does with a timeout, so you cannot get stuck in these loops forever.

I am out of time to reply right now, but keep going! You are going to get there!

16

Cheers @Hootie81, I discovered that after much forum trawling :slight_smile:

17

Hi again, as a point of interest I have got this sort of working. It now uses static char arrays and timeouts.

It now hits the server but unfortunately the php script doesnt receive the file or the file name etc, ie, as far as the php is concerned it cant see any post vars.

In any case, I’m posting my code as it is helpful to keep a running log of my work as it is sort of based around issues that many people seem to have with tcpclient. I have yet to figure out how I’m going to calculate Content-Length so until I do it is static, also, the binary data is generated sample data. Thanks to @bko, without you I’d not even have got this far :slight_smile:

TCPClient client;

#define server "xxx.com"

int uu = 0;
byte buffer[32];

void setup() {
 Serial.begin(9600);
 for(int u=0; u<32; u++)
    buffer[u] = (byte)u;
}

void loop() {
   delay(5000);
   if(client.connect(server, 80))
    {
        Serial.print(F("connected : "));
		Serial.println(uu, INT);
		
		char str[212] = "POST /ttx.php HTTP/1.1\r\nConnection: keep-alive\r\nHost: www.xxx.com\r\nAccept: */*\r\nContent-Length: 268\r\nExpect: 100-continue\r\nContent-Type: multipart/form-data; boundary=c27a2baab8fe94bd\r\n\r\n";
		client.write((uint8_t*)str, 212);
		blockWaiting(500);
		
		char str2[124] = "--c27a2baab8fe94bd\r\nContent-Disposition: form-data; name='img'; filename='IMAGE00.JPG'\r\nContent-Type: image/jpeg\r\n\r\n";
		client.write((uint8_t*)str2, 124);
		blockWaiting(500);
			              
		for(int u=0; u<3; u++)
		{
			client.write(buffer, 32);
			blockWaiting(500);
		}
			               
		Serial.print("end buffer : ");

		client.write(F("\r\n--c27a2baab8fe94bd--\r\n\r\n"));
        
        blockWaiting(500);
        delay(400);
        client.stop();
        client.flush();
        Serial.println("sent");
    } 
    else 
    {
        Serial.println("connection failed : ");
        delay(400);
        client.stop();
        client.flush();
    }
    uu++;
}

void blockWaiting(unsigned long timeOut)
{
    unsigned long startWait = millis();
    while(client.available() == 0) 
    {
        if(millis() - startWait > timeOut) 
        return;
    }

    while(client.available() != 0)
        client.read();  //dump return data
}
18

try printing the return data instead of dumping it… there may be a important clue your missing by throwing it away. just a Serial.print(client.read()); should do it

1 Like
19

Cool idea, I’ll give that a go :slight_smile:

20

Well, day three of trying to get this to work :stuck_out_tongue:

I have now got an output from the server but it makes absolutely no sense. I’m looking at the server response codes which all seem to be in thr range of 1xx - 5xx as seen here

My output is this :smile:

connected : 0
After Header
72
84
84
80
47
49
46
49
32
49
48
48
32
67
111
110
116
105
110
117
101
13
10
13
10
Data - timed out
Data - timed out
Data - timed out
Finish - timed out
sent

To obtain this data I modified my waiting function like so :smile:

void blockWaiting(unsigned long timeOut, String desc)
{
    unsigned long startWait = millis();
    while(client.available() == 0) 
    {
         if(millis() - startWait > timeOut) 
        {
            Serial.print(desc);
            Serial.println(" - timed out");
            return;
        }
    }

    Serial.println(desc);

    while(client.available() != 0)
        Serial.println(client.read());  //dump return data
}

I also had an idea, to use the F macro instead of char arrays, that should avoid heap fragmentation right ? by keeping the ‘string’ in progmem. Like so :smile:

	client.write(F("POST /ttx.php HTTP/1.1\r\nConnection: keep-alive\r\nHost: www.xxx.com\r    \nAccept: */*\r\nContent-Length: 268\r\nExpect: 100-continue\r\nContent-Type: multipart/form-data; boundary=c27a2baab8fe94bd\r\n\r\n--c27a2baab8fe94bd\r\nContent-Disposition: form-data; name=\"img\"; filename=\"IMAGE00.JPG\"\r\nContent-Type: image/jpeg\r\n\r\n"));

I appretiate that sending a large blob of binary data is probably not something many people want to do and it’s not something the core is going to be very good at concidering its limited resources but I really want to get this to work even if it turns out it wont satisfy my end requirements and I have to go another route. I’ve now been working on this for two days, hopefully today will see it finished lol. Cheers for the help so far, if anyone has anything to add on this any help is really greatfully appretiated, the HTTP specs I’ve read make little sense to me unfoortunately, as far as I can see, I’m doing it correctly.