Connecting the Electron 3G over TCP to Your own Server with third party SIM to send SD card data


#7

You can use the SD Fat library to use SD cards with the Photon and Electron. Just started testing this myself and it works perfectly.


#8

Do I understand correctly that you are effectively dual-porting the SD card between two processors? If so I would appreciate more details.

I am in the process of converting an existing system from Photon to Electron or E series. The application captures a 0.5 MByte file twice daily. The intent of using the Electron is to transmit metadata following each recording session without the need for the user to set up wifi credentials,. I then need to be able to recover the full data set once the device is returned to the lab. This is not feasible given the data rates charged by Particle/Telstra in Australia, There are third party suppliers that are feasible - this is an option with the Electron but not the E series.

I am exploring the option of using something like an ESP8266 or ESP32 to transmit the data over wifi once the device is back in the lab. Your solution of dual-porting an SD card sounds interesting. When it is time to extract the full files I could have the Electron trigger a web server on the ESP32 then get out of the way, allowing the ESP32 to serve files from the SD card. Sort of the reverse of your application.

I would appreciate your thoughts and experience.


#9

Hi @OziGreybeard.
I should have given more details about that. Having 2 masters on 1 slave is actually proving to be unreliable. What we did was to disable the SPI interface of the sinknode. Sometimes, we just don’t read any data and the Electron reports an error at sdInitErrorHalt(). So we have since reverted to sending the data over the serial port. In one forum, I have been pointed to this document. (Our sinknode is an AVR microcontroller). The solution they mention is to free all the SPI lines of one master to be GPIO I/Os. I am giving this a shot this week because it would greatly simplify the task. I will post my luck as soon as it gets through.


#10

From what I understand of your project and cost constraints, you might consider LoRaWAN for this use case. It operates in the ISM bands and should offer better range (900 MHz versus 2.4 GHz) and power consumption profile (assuming Class A will work for you). Fairly straightforward to setup a private network. https://www.thethingsnetwork.org/


#11

Thanks for the suggestion. Lorawan is not suited for the daily summary reports as there is no viable lore an net in Adelaide. Patients come from a wide area. It’s data rate is too slow for downloading the files in the lab.


#12

LoRaWAN is not suitable for this use case. The total separation of the wireless nodes in this case is typically very small. Usually all nodes are within 10-20m of each other for Automatic Weather Stations and may be as much as 100m, depending on the transmission power and circumstances. All transmissions from the sensor nodes are typically downlink to the sink node, which has the one uplink to the server. You could have an uplink per sensor node but that brings in a power challenge, and power minimization is on top of our list right now.
I was happy to listen to a presentation on LoRaWAN in Cape Town in September, 2017 at the IEEE AFRICON conference. It’s possible application is in environment monitoring but the needs and requirements and implementation would be different.


#13

Dear mbyamukama,

Can you share the TCP listener you wrote? I’m trying to do a similar project but I’m facing an issue regarding how to create the web server.

Thank you,


#14

here you go. The executable file is larger with the Microsoft C# compiler than with Mono on Linux. If you are using a linux server, better to compile using mono on the server. the file size is smaller and stream write operations are just a little bit faster, but to no useful consequence. The listener receives data and writes to a text file. Another process sends the data to a relational database.


#15

@OziGreybeard
We did manage to implement the 2-master-1 slave design by using a normally closed quad single-pole single-throw analog switch (DG201BDJE3 - http://www.vishay.com/docs/70037/dg201b.pdf)
In essence, this simply makes it a multi-tasked 1-master 1-slave. The Electron sets all the IN (switch control) lines high from one of its digital pins when it is powered up and causes the switch to isolate the 4 SPI lines from the other master.
We struggled with the on-resistance though. For a single supply, you have the SPI signals running through 90Ω before reaching the SD card. We think a better similar design is to use the ADG794 (http://www.analog.com/media/en/technical-documentation/data-sheets/ADG794.pdf) which instead has 4 2:1 multiplexers.

That said, we did not implement any of these designs. What we have now is the Electron in Deep Sleep, instead of being powered off. The sink node wakes it up though the D1 pin and then sends the data over the serial port. The Electron writes the data to the SD card itself and resumes the sleep state. It uploads the data after a given number of reports. Our opportunity cost is that we now have a 14.6mA system instead of a 13mA system. The extra 1.6mA from the Electron is little extra juice for significantly more uncomplicated design.


#16

@mbyamukama, if you are waking the Electron via D1 then it can’t be in deep sleep. You would need to wake via the WKP pin for that. Deep sleep would give you even lower power consumption.


#17

@peekay123.
Sorry, you’re right. We simply have it in sleep mode–since we need to preserve most of the RAM contents anyway, such as the report count, when we wake up. The line doing that looks like this:

System.sleep(D1,RISING,3000);

#18

There is backup RAM which alliws to keep several KB of variables retained even after a reset.


#19

@ScruffR, Thanks. Just read about it in the post by @rickkas7 here. Easy syntax as well.


#20

Thanks. With imminent (July) release of the Xenon and Boron boards I am considering a different track myself. I have found a 3rd party provider with reasonable data allowances for my needs.

I am investigating using a Xenon to collect data at 250 Hz and transmit it to Boron as a recorder, analyser and gateway. This not only isolates the acquisition from timing glitches induced by recording to sd card (specifically flushing the buffer) and possibly the 3G/LTE link but would allow the sensor to be attached to the device with a BLE or Mesh link to the gateway/display.

Thanks for your update


#21

Hi @mbyamukama
Do use ever have issues with your SD card failing to initialize or communicate via SPI?
@jaza_tom and I have been having some SD issues and wonder what strategies you employ to keep your read/writes working smoothly over long periods of time. We’ve tried different SD cards, re-formatting, etc. Usually a reset fixes things.
Thanks.


SD Card Crashes periodically in embedded PCB with Electron
#22

@SCM

  1. What else is connected to your particle device?
  2. What SD card module are you using?
  3. What SD card brand are you using?

In my application, the only issues we have identified with SD cards are:

  1. Low VCC on the card- the standard says 2.7-3.6V but we often got a few issues before reaching 2.7V especially for large writes
  2. Fake SD cards- Some cards from china claim to be “San Disk” and yet they weren’t. Often, they do work in your PC but not with the microcontroller.
  3. Grounding - Damage to PCB tracks or loose jumper wires can cause poor grounding. We have at times found a small voltage between the SD GND and the system GND because of this.
  4. Load on any of the SPI pins - Do you have another slave connected to the SPI? Some loads may have a low impedance and drain some current (like the 2nd master I was talking about in this thread), leaving insufficient current for the SD R/W operation.
  5. We have also at times corrupted the data just by trying to remove the SD card when it is powered!

My advice:
We chose the module in question because of our interest in low power operation. If your application can use 5mA extra, I’d advise you to change the module to this one like this below:
Untitled

We bought some from here
The AMS117 ensures 3.3V always. You can power the module from a 5V supply. You can get 5V from a tiny boost converter module like this one hooked up directly to your battery. This module features a SN74LVC125A bus buffer gate for level shifting (In case you are using some other microcontroller). We never had any issues with those, except the extra power.


#23

Thanks @mbyamukama
Answering your questions:

  1. Lots of things, but to the SPI pins that our SD card holder (integrate into a custom PCB), the only other thing is a LCD display.
  2. Answered above.
  3. SanDisk (purchased on Amazon, seemingly authentic).

Re: the issues you mentioned:

  1. We rely on the Electron’s 3.3V supply (powered by an isolated 5V supply)
  2. I’ve also read about this. Was in Shenzhen recently and very tempted to buy a suitcase full of cheap SD cards, but resisted. Still buying off Amazon primarily.
  3. Haven’t come across any grounding issues yet
  4. I suspect this is a likely candidate.
  5. We are pretty careful about this - only removing the SD card when powered down or in DFU mode (to ensure no R/W cycles happening)

@jaza_tom Any other thoughts?


#24

I suspect it is most likely a result of voltage supply issues during read/write operations.

Since we are powering the SD card off the same 3.3V supply that is used by the modem, there may be unacceptably low voltage dips on that supply during modem transmission events which may coincide with read/write operations on the SD card.

The voltage dips may also be caused by differential “ground” voltages between Electron and SD card (basically same as power supply dip issue, except could be caused by other ground currents flowing in the ground plane between Electron and SD card).

Finally, I guess it may be possible the the logic “high” level of the Electron is not quite as high as it should be during modem transmission, which could result in corrupted command sequences being sent to the SD controller.

I think powering the SD card / SPI pullup resistors with a dedicated 3.3V DC/DC converter is probably the best solution because it will eliminate the potential voltage supply issues discussed above, and will also allow our software to power cycle the SD card (assuming we pick a DC/DC converter that has an enable pin that can be controlled by the Electron).

We could also look at using a tristate buffer or multiplexer to share SPI between LCD and SD, or re-map pinout to use two dedicated separate SPI peripherals concurrently.


#25

How often are you writing to the SD card? How much data would you miss out on if you halted the write process until after the upload? Can you buffer during that period? In my application, there’s about 25 seconds to connect to the cellular network and another 10 to upload ~33KB of data. We forego the data in those ~35 seconds (which may be anything between 0 and 10 reports) but in environment monitoring that is very little data.


#26

I like the idea of creating a timing buffer between known transmissions and sd writes. I’m going to look into this I think.

Normally we publish right after SD card writes to avoid the possibility of transmission-induced voltage dips, but there are occasional writes to SD during/shortly after a transmission. It would be useful if there was a way to know if a transmission was taking place before writing (i.e. the non-app transmission events that occur behind the scenes as part of the Particle ecosystem - keepAlives, handshakes, etc).

I think the most challenging part of implementing some sort of write block during transmission events is the startup sequence… there are several spots where SD writes during the startup sequence are desirable. Probably easiest way to test if transmission voltage dips are our culprit would be to delay connecting to the cloud until after all SD writing operations that are part of the startup sequence have been completed.