I made a tutorial to augment the [Particle Google Cloud Platform Tutorial] (https://docs.particle.io/tutorials/topics/google-cloud-platform/). Much of the beginning material is the same, but it continues on where the original tutorial ends with creating a node script to run on your computer.
My tutorial creates a node script and deploys it to the Google App Engine where it automatically takes the Google Pub/Sub events and routes them into the Google Datastore NoSQL database. No code runs on your computer; it’s handled entirely within the cloud.
This is similar to my [Google Firebase Tutorial] (https://github.com/rickkas7/firebase_tutorial) but using Google Cloud directly gives you a lot more flexibility. You can transform the data using Javascript code before putting it in the database, for example.
Thank you for the tutorial @rickkas7. Please forgive me as I am a newbie to Google Cloud. I have followed your tutorial through until setting up the App Engine. Do I need to create an actual app in Google Cloud? I had trouble cloning hello-world. I am also having difficulty with understanding where your 1-app directory should be placed with the app.yaml file and then deploying it with the google cloud sdk on my mac. Any help would be much appreciated! Thanks
I’m not sure why you couldn’t clone the hello world, example. The exact error might help, but you don’t really need to do that, anyway.
There are two parts to the app engine example. The first part downloads and runs it on your own computer. This greatly speeds up the process if you’re editing and debugging the app.js code for your own purposes because it takes a second to run locally and a couple minutes to deploy remotely.
You can put the files anywhere, but presumably they’d be somewhere in your Documents directory. The main thing is that you just need to use the cd command to position yourself in that directory in order to either run the example locally or deploy it.
Thank you @rickkas7, I got the script to deploy and run. However, the data is not storing. In the read logs, it seems like pub/sub is reading the data from particle cloud but is not writing to datastore. Any suggestions? Thanks
issued from your computer on your 1-app directory will read the logs for your deployed server. It should show whether the problem is in your app subscription, or in the database write.
@rickkas7 upon further examination of the read logs, the error seems to occur here:
“Failed at the particle-datastore@0.0.1 start script ‘node app.js’. This is most likely a problem with the particle-datastore package, not with npm itself. Tell the author that this fails on your system: node app.js.”
That means there’s a bug in app.js. There might be an error above that somewhere in the log, though it can be kind of hard to find. That’s why I mentioned being able to run the app.js on your machine is helpful. A problem in app.js will show up immediately with a full stack trace when you run locally.
Hi, I believe I had a different configuration in declaring the variables. I added message.data to var obj in app.js. Probably not the cleanest but works for my dev project.
@rickkas7 I’ve followed your tutorial to get the Node.js script running within App Engine in the short term but, it appears to have stopped working only after 15 hours…
A bit of background, I have an Electron reading DHT11 Sensor data every 5min and publishing to particle, looking at the Integration logs in Particle console I can see that the messages have been received by particle and sent on to the google cloud platform. (as of writing last message sent 22:37:40)
When I check the Google Datastore the last message stored was at 15:49:40 but, checking the App Engine logs in the CLI, show the last message processed by the script was at 15:54:43.
I am using the exact same app.js as you, but I have uncomment some of the other things so I can store them in the database.
What about the Particle Logs in the console? Are you still seeing messages generated there? Also, you could use the technique for adding a temporary Google command line listener on the Google pub/sub topic to make sure it’s making it from Particle to Google pub/sub. There’s also a link for viewing the logs for the app engine instance, but in retrospect I should have added more logging to that.
Thanks for the quick reply and good shout about creating a Google command line listener, didn’t think of that!
The problem with Particle console logs, is that they only appear when the Electron is connected and then seem to disappear again once the connection is disconnected.
I have just restarted the App Engine version and all of my data is now within the Datastore…
Therefore does the data get cached in Particle and once the script is up and running again, it just subscribes and retrieves all of the ‘lost’ data?
The Particle cloud does not retain publishes, but the Google cloud does, as far as I know. So that seems to indicate to me that something happened to your app instance. I’d definitely check the logs for that. I had my test instance running for a month without problems, but it was mostly idle.
I’ve gone through the logs and there isn’t much to go on, it looks like it was working and then just stopped, there is no difference in the logs from before it stopped, to when it stopped and after I restarted the app.
Do you know how much data gcloud will cache and for how long it will retain the data for?
Edit/ Stopped again today at 11:38:40, I’ll post up the logs tonight when I am back at home.
This was an excellent tutorial Rick, and hugely helpful! With all of the detail you provided, I've got something successfully up and running that I would have been otherwise totally unable to tackle – thanks!
I just got it running, but I had to make two small changes right at the end. Both were made in the app.yaml file:
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: The environment variable(s) 'GCLOUD_PROJECT' are not allowed.
After reviewing the authentication documentation, it looks like the project ID can either be hard-coded, or it will be detected automatically. I simply commented out the GCLOUD_PROJECT line in app.yaml, and the app deployed without an error.
Hopefully this helps anyone else trying to follow this tutorial. The App Engine Flexible Environment is still in Beta, so there may be other changes that come later on down the road.
Thank You @rickkas7 for the tutorial. I have been trying to finish this integration for a few weeks, ran into issues. This is big help. I finally am able to finish the integration Also thanks to @jameshallam for pointing out the two issues.
Thanks @rickkas7 for sharing this amazing tutorial. Really glad to watch this tutorials. This community has lots
of information and it is useful for us.
Thanks for all the work you put into these.