TCPclient GET request with --data-urlencode

Hi,

I am trying to send the following Curl GET request to an InfluxDB on my server.

curl -GET 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode "q=SELECT LAST(value) FROM cpu_load_short WHERE region='us-west'"

I found out that pretty=true formats the returned json data in a pretty way. I am not sure if this is usefull with a Photon.
And --data-urlencode encodes special character like spaces but it confuses me. I am not sure how to format this GET request for my Photon and would appreciate your help.

w
/query?pretty=true’ --data-urlencode “db=mydb” --data-urlencode “q=SELECT LAST(value) FROM cpu_load_short WHERE region=‘us-west’”

should

What I did so far is that I have manually encoded the query and wrote a http header:

/query?&db%3Dmydb&q%3DSELECT%20LAST(value)%20FROM%20cpu_load_short%20WHERE%20region%3D%27us-west%27

I have removed pretty=true and manually encoded spaces and =. I am not sure if the ? after query needs to encoded too?

int readInflux()
{
    client.connect(INFLUXDB_HOST, INFLUXDB_PORT);
    if(client.connected())
    {
        Particle.publish("connected and ready to query");
    }
    
    client.println("GET /query?&db%3Dmydb&q%3DSELECT%20LAST(value)%20FROM%20cpu_load_short%20WHERE%20region%3D%27us-west%27 HTTP/1.1");
client.println("Host: " + String(INFLUXDB_HOST));
client.println("User-Agent: Photon/1.0");
client.println("Connection: close");
client.println("Content-Type: application/x-www-form-urlencoded");
client.println();
client.flush();

int n = client.available();  
char y[n];


for( int i =0; i < n; i ++ )
{
    y[i] = client.read();
} 
String str(y);
Particle.publish(str);
client.flush();
client.stop();
}

The code seems to work but Particle.publish(str); publishes only 2 characters like hu, Xq, 8r,… So I am definitely doing something wrong. I would be very grateful if you could help me!

Edit: After some more try and error I found out that the variable n never gets any value from client.available();
So the error is maybe in the header. I don’t know why there are always two characters getting published :confused:

I think at least one problem is that client.avaiable() works differently than you are using it. It returns the number of characters available to read from the TCP connection at that exact instant in time. In fact, if there is any delay at all in the response from your server, n will be zero, because there is no response yet. Depending on how you write the data out from your server code, it could be any subset of the actual response.

Depending on how the data comes back from the server, you need to loop until some other condition is met. If the server sends a \n (LF, newline) after the response string, you could check until client.read() returns that. If the server closes the connection at the end of the response, you could check for while(client.connected()). There are lots of possibilities.

1 Like

Thank you very much for your reply @rickkas7
I have made some progress and it works but not always :confused:
I have used Curl to send the GET request to my server and monitored the traffic with Wireshark. So I found out how the request needs to be formatted:

Curl request:

curl -GET “http://mydyndnsaddress:8086/query” --data-urlencode “db=sensors” --data-urlencode “q=SELECT LAST(kWh) FROM sensors WHERE Location=‘street’”

curl -GET “http://mydyndnsaddress:8086/query” --data-urlencode “db=mydb” --data-urlencode “q=SELECT LAST(value) FROM cpu_load_short WHERE region=‘us-west’”

Wireshark:

Frame 15: 227 bytes on wire (1816 bits), 227 bytes captured (1816 bits) on interface 0
Ethernet II, Src: IntelCor_ae:db:c8 (00:24:d7:ae:db:c8), Dst: Elitegro_77:e4:7a (b8:ae:ed:77:e4:7a)
Internet Protocol Version 4, Src: 10.0.0.17, Dst: 10.0.0.14
Transmission Control Protocol, Src Port: 1532 (1532), Dst Port: 8086 (8086), Seq: 1, Ack: 1, Len: 173
Hypertext Transfer Protocol
    GET /query?db=sensors&q=SELECT%20LAST%28kWh%29%20FROM%20sensors%20WHERE%20Location%3D%27street%27 HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET /query?db=sensors&q=SELECT%20LAST%28kWh%29%20FROM%20sensors%20WHERE%20Location%3D%27street%27 HTTP/1.1\r\n]
        Request Method: GET
        Request URI: /query?db=sensors&q=SELECT%20LAST%28kWh%29%20FROM%20sensors%20WHERE%20Location%3D%27street%27
        Request Version: HTTP/1.1
    Host: 10.0.0.14:8086\r\n
    User-Agent: curl/7.49.0\r\n
    Accept: */*\r\n
    \r\n
    [Full request URI: http://10.0.0.14:8086/query?db=sensors&q=SELECT%20LAST%28kWh%29%20FROM%20sensors%20WHERE%20Location%3D%27street%27]
    [HTTP request 1/1]
    [Response in frame: 17]

Server response:

{"results":[{"series":[{"name":"sensors","columns":["time","last"],"values":[["1970-01-01T00:00:00Z",0.13]]}]}]}

I have also modified the code to read the incomming chars and wrote a complete new sketch just for reading the database. This sketch runs on a Particle Core:

#define INFLUXDB_HOST   "my dyn dns address"
#define INFLUXDB_PORT   8086
#define INFLUXDB_DB     "sensors"

#define DEBUG_SERIAL true
#define SKIP_CHARS 0
#define BUFFER_SIZE_MAX 1024

TCPClient client;

unsigned long previousMillis = 0;
unsigned long previousMillis2 = 0;
unsigned long interval = 15000;

char buffer[BUFFER_SIZE_MAX];

void readInflux()
{
    int i = 0;
    int j = 0;
    int k = 0;
    bool printOnce = false;

    int i = 0;
int j = 0;
int k = 0;
bool printOnce = false;

if (client.connect(INFLUXDB_HOST, INFLUXDB_PORT)) {
    client.print("GET ");
    client.print("/query?db=sensors&q=SELECT%20LAST%28kWh%29%20FROM%20sensors%20WHERE%20Location%3D%27Bachgasse%27");
    client.println(" HTTP/1.1");
    client.print("HOST: ");
    client.println(INFLUXDB_HOST);
    client.println("Connection: close");
    client.println();
    client.flush();
} else {
    if(DEBUG_SERIAL) Serial.println("\r\n\r\nConnection Failed!");
    client.flush();
    client.stop();
    return;
   }

  // wait 5 seconds or less for the host to respond
  uint32_t startTime = millis();
  while(!client.available() && (millis() - startTime) < 5000);

  if(DEBUG_SERIAL) Serial.println("\r\n\r\nREADING HOST DATA......");
  uint32_t lastRead = millis();
  // If the host doesn't close it's connection, we'll timeout in 10 seconds.
  while (client.connected() && (millis() - lastRead) < 10000) {
    while (client.available()) {
      char c = client.read();
      
      if(DEBUG_SERIAL) {
        Serial.print(c);
        if(i++ > 100) {
          delay(5);
          i = 0;
        }
      }
      
      if(c == -1) {
        Serial.print("\r\n\r\nERROR......\r\n\r\n");
        client.flush();
        client.stop();
      }
      if(j++ >= SKIP_CHARS) { // don't buffer the first X bytes to save memory
        if(DEBUG_SERIAL && !printOnce) {
          Serial.print("\r\n\r\nSAVING......\r\n\r\n");
          printOnce = true;
        }
        buffer[k++] = c; // save character to buffer
        Serial.print(c);
        delayMicroseconds(100);
      	if(k >= BUFFER_SIZE_MAX) { // if we reach the end of our buffer, just bail.
          Serial.print("\r\n\r\nOUT OF BUFFER SPACE......\r\n\r\n");
          client.flush();
      	  client.stop();
      	}
      }
      // as long as we're reading data, reset the lastRead time.
      lastRead = millis();
    } // END while(client.available())
  } // END while(client.connected())
  
  client.flush();
  client.stop();
  
  if(DEBUG_SERIAL) {
    Serial.print("\r\nCHARACTERS RECEIVED: ");
    Serial.println(SKIP_CHARS + k);
  }
}


void setup() {
    Serial.begin(19200);
   
    //kwh = readInflux();
    
}

void loop(){
   
    unsigned long currentMillis = millis();
    if(currentMillis - previousMillis >= interval) {
        
        Serial.println("reading Influx");    
        readInflux();
        previousMillis = currentMillis;
            
        
    }else{
        Serial.println("Device not ready");
        delay(1000);
    }
}

Serial output:

Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
reading Influx


READING HOST DATA......
H

SAVING......

HTTTTPP//11..11  220000  OOKK


CCoonntteenntt--TTyyppee::  aapppplliiccaattiioonn//jjssoonn


RReeqquueesstt--IIdd::  bb55ccbb55557777--2255bbff--1111ee66--9933aa55--000000000000000000000000


XX--IInnfflluuxxddbb--VVeerrssiioonn::  00..1122..11


DDaattee::  SSuunn,,  2299  MMaayy  22001166  1177::0066::4400  GGMMTT


CCoonntteenntt--LLeennggtthh::  111122


CCoonnnneeccttiioonn::  cclloossee





{{""rreessuullttss""::[[{{""sseerriieess""::[[{{""nnaammee""::""sseennssoorrss"",,""ccoolluummnnss""::[[""ttiimmee"",,""llaasstt""]],,""vvaalluueess""::[[[[""11997700--0011--0011TT0000::0000::0000ZZ"",,00..1166]]]]}}]]}}]]}}
CHARACTERS RECEIVED: 318
Device not ready
Device not ready
Device not ready
Device not ready

Serial output when it is not working:

Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
Device not ready
reading Influx


READING HOST DATA......

CHARACTERS RECEIVED: 0
Device not ready
Device not ready
Device not ready
Device not ready
Device not ready

The code I am using to read the characters is from @technobly: https://github.com/technobly/SparkCore-FacebookLikesAlert/blob/master/application.cpp

It seems to read the characters so fast that it is reading them multiple times. Is there a way to handle this?

Edit: I have not changed anything but it stopped working: CHARACTERS RECEIVED: 0
Edit2: I have found the Error :smile: Now I only need to solve the problem of multiple characters.
Edit3: I have forgotten to set DEBUG_SERIAL false :unamused: