Feeding data from photon to elasticsearch

Troubleshooting
1

Hi
I am working on a project where i am reading all the data from a DHT sensor and i want to push the results to an Elasticsearch instance to visualise the temperature/humidity along the day.

To do my testing, i want to use a local instance of ELK, looking forward to move it to AWS at some point.

What is the best option i have to do it? I am trying to build a webhook to push it to ELK, but apparently the integrations dont work with any local url (as i said, my elasticsearch is a local instance)

Should i do the other way around? It is, write a mini python script that reads the events fom particle API and ingests it it elasticsearch, or is there a better option to do it?

Second question question is… i am realtively new to IOT world and particle, so i dont know if there is any platform better than elasticsearch to monitor that data

Thanks in advance

2

If it’s all local and doesn’t require secure connection (HTTPS) then you could use TCPClient or HTTPClient all locally.
If you need HTTPS you may still want to go via the Particle webhooks but need to setup your local elasticsearch with a public IP & Port.

1 Like
3

Thanks

Im trying httpclient, exactly this example – https://github.com/nmattisson/HttpClient

However it doesnt work at all (response code is -1). This is the output of the serial monitor

Serial monitor opened successfully:

Application> Start of Loop.
Application> Response status: -1
Application> HTTP Response Body:

Any good tutorial on how to use httpclient please?

Thanks

4

It’s usually little use to refer to an off-the-shelve example as even the tiniest difference between that and your code may be causing the trouble.

It’s always best to actually show your code rather than leaving the differences in the dark.

BTW, using www.timeapi.org/utc/now in a browser also renders an error, so no surprise there.

1 Like
5

Literally I copied the example as a brand new application. It doesn’t do anything else

I have also used tcp library and I have the same result, so I fear the problem might be related to connectivity but I can see the device attached to the WiFi and it pushes events to the cloud correctly

Is there any way to debug and see what is exactly the problem? Or even if you have an example application I can test it would be helpful

Thanks

6

If you literally copied the example code as is, you also copied the target address and hence this statement still stands

and

as said, trying the target address in your browser would be the first debugging step to take.

7

I tried other sites and had the same result. Even hitting my local elasticsearch I have the same problem

8

In that case this statement becomes void

while my previous statement still stands

9

Ok. Lets start again :slight_smile:

Thats my TCPClient code

TCPClient client;
byte server = { 74, 125, 224, 72 }; // Google
void setup()
{
// Make sure your Serial Terminal app is closed before powering your device
Serial.begin(9600);
// Wait for a USB serial connection for up to 30 seconds
waitFor(Serial.isConnected, 30000);

Serial.println("connecting...");

if (client.connect(server, 80))
{
Serial.println("connected");
client.println("GET /search?q=unicorn HTTP/1.0");
client.println("Host: www.google.com");
client.println("Content-Length: 0");
client.println();
}
else
{
Serial.println("connection failed");
}
}

void loop()
{
if (client.available())
{
char c = client.read();
Serial.print(c);
}

if (!client.connected())
{
Serial.println();
Serial.println("disconnecting.");
client.stop();
for(;;);
}
}

And this is the HTTPClient code

#include <HttpClient.h>

/**

  • Declaring the variables.
    */
    unsigned int nextTime = 0; // Next time to contact the server
    HttpClient http;

// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers = {
// { "Content-Type", "application/json" },
// { "Accept" , "application/json" },
{ "Accept" , "/"},
{ NULL, NULL } // NOTE: Always terminate headers will NULL
};

http_request_t request;
http_response_t response;

void setup() {
Serial.begin(9600);
}

void loop() {
if (nextTime > millis()) {
return;
}

Serial.println();
Serial.println("Application>\tStart of Loop.");
// Request path and body can be set at runtime or at setup.
request.hostname = "www.google.com";
request.port = 80;
request.path = "/search?q=unicorn";

// The library also supports sending a body with your request:
//request.body = "{\"key\":\"value\"}";

// Get request
http.get(request, response, headers);
Serial.print("Application>\tResponse status: ");
Serial.println(response.status);

Serial.print("Application>\tHTTP Response Body: ");
Serial.println(response.body);

nextTime = millis() + 10000;

}

10

I finally had some time to check that

It's not that the request doesn't work, but your target server is temporarily blacklisting your device for repeated requests.
When I tried your HTTPClient code I got proper 200 responses a few times round but after that I also kept getting -1 (even after some minutes pause).
To get a new 200 response I just had to alter the search terms and then got a response again.

However, a successful request against google.com/search?q=whatever will produce a bigger response.body than you can catch in one go. So you need to keep reading the response in multiple chunks - something HTTPClient doesn't seem to support.

And as it seems @nmattisson hasn't addressed open issues since February 2017 :pensive:

However, once you avoid getting your device blacklisted this TCPClient approach should work

TCPClient client;
IPAddress server( 172, 217, 16, 142 ); // google.com
void setup()
{
  Serial.begin(9600);
}

void loop() {
  static uint32_t msRepeat  = 0;
  uint32_t msTimeout = 0;
  char req[1024];
  
  if (msRepeat && millis() - msRepeat < 30000) return;
  
  if (client.connect(server, 80))
  {
    int len;
    Serial.printlnf("connected");
    Serial.println("-----------------------");
    snprintf(req
            , sizeof(req)
            , "GET /search?q=%s%d HTTP/1.0\r\n"
              "Connection: close\r\n"
              "Host: www.google.com\r\n"
              "Content-Length: 0\r\n"
              "Accept: */*\r\n"
              "\r\n"
            , "mySearchTerm"
            , millis() % 100 // append a changing two digit number to avoid blacklisting ;-)
            );
    Serial.print(req);
    Serial.println("-----------------------");
    client.print(req);
  }
  else
    Serial.println("connection failed");

  msTimeout = millis();
  while((client.available() || client.connected()) && millis() - msTimeout < 60000)
  {
    Serial.write(client.read());
    if (millis() - msRepeat > 100) { // cloud keeping every 100ms
      Particle.process();
      msRepeat = millis();;
    }
  }

  Serial.println("\r\ndisconnecting.");
  client.stop();

  msRepeat = millis();
}