[SOLVED] How to set user agent for http get

[Michele]

This code used to work but no longer does.

 request.hostname = "newsapi.org" ;
   request.port = 80;
   request.path = "/v1/articles?source=reuters&sortBy=latest&apiKey=keyRedacted" 
    
    // The library also supports sending a body with your request:
    //request.body = "{\"key\":\"value\"}";

     // Get request
        Serial.println (" Connecting to news ...") ;
        http.get (request, response, headers) ;

Now I get this error:

{"status":"error","code":"userAgentMissing","message":"Please set your User-Agent header to identify your application. Anonymous requests are not allowed."}s":"error","code":"userAgentMissing","message":"Please set your User-Agent header to identify your app

Fair enough, but how do I do that? This post https://community.particle.io/t/http-get-response-code-1/25657/12 seems to be the same problem but adding

   http_header_t headers[] = {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    {"User-Agent","ParticleHttpClient"},
    { "Accept" , "*/*"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

directly before setting request.hostname didn't work (code compiles and runs but still fails to get the data).

[rickkas7]

It is possible that the server you are connecting to requires a valid string in the User-Agent field, not just that it's present. See the User-Agent specification.

Another thing to test is to direct the request to a service like RequestBin so you can see exactly the headers that are being transmitted to make sure that's working correctly.

[Michele]

Good suggestions. First I tried pasting the entire URL into Firefox and it worked OK there. I got the user agent string Firefox was using and put that in my Photon code:

{"User-Agent","Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"},

Still returns -1. Anyway, this site: https://techflarestudio.com/how-to-use-news-api-with-examples/ claims you can put anything in the user agent field for newsapi.org.

So I tried it in RequestBin using "RequestBin" for the user agent. That did work - it returned status 200.

I then tried moving the user agent string to the init section of my code:

// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers[] =
  {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    { "Accept" , "*/*"},
    {"User-Agent","Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
  } ;

but that didn't work either. I'm mystified.

[rickkas7]

When you used RequestBin, did the User-Agent show up correctly in the request headers? The library code looks like it should be adding it, but that is a good thing to make sure of.

[Michele]

Good question. This is what it said (and the event has a green checkmark next to it):

steps.code{1}
$response{3}
status:200
headers{0}
body{8}
aboutPipedream is the fastest way to connects APIs. Build and run workflows with code-level control when you need it — and no code when you don't.
event_id:
2kZN7cnbmQrjrKVWvAuerFsTj8m
workflow_id:
p_ZJCVGlB
owner_id:
o_bLINRPJ
deployment_id:
d_n3sK8ja6
timestamp:
2024-08-12T18:22:52.580Z
inspect:
https://pipedream.com/@/p_ZJCVGlB
quickstart:
https://pipedream.com/quickstart/

Where does it show the headers?

[Michele]

For the record, here is a complete minimal example of how to use User-Agent in an http request. It's tested and working on the Photon,as of 09/03/24. I'm marking this as Solved.

// This #include statement was automatically added by the Particle IDE.
#include <HttpClient.h>

// Include Particle Device OS APIs
#include "Particle.h"
#include "HttpClient.h"

// Let Device OS manage the connection to the Particle Cloud
SYSTEM_MODE(AUTOMATIC);

// Show system, cloud connectivity, and application logs over USB
// View logs with CLI using 'particle serial monitor --follow'
SerialLogHandler logHandler(LOG_LEVEL_INFO);

HttpClient http;
 
http_header_t headers[] =
  {
    { "Content-Type", "application/json" },
    { "User-Agent", "ParticlePhoton" },
    { NULL, NULL } // Terminate headers with NULL
  };
 
http_request_t request;
http_response_t response;


// setup() runs once, when the device is first turned on
void setup()
  {
  // Put initialization like pinMode and begin functions here

    Serial.begin(9600);
    request.hostname = "newsapi.org";
    request.port = 80;
    request.path = "/v2/top-headlines?country=us&apiKey=myKey"; // Replace with your API key
  } // end setup

// loop() runs over and over again, as quickly as it can execute.
void loop()
  {
    http.get(request, response, headers);
    Serial.println(response.status);
    Serial.println(response.body);

    // Stop the loop after one request
    while (true);
  } // end loop