Hi all, having a very strange and difficult to reproduce issue…
We have a bunch of boron- based systems running in a dattalogging/sending application. for efficiency, we have another mcu duty cycling the boron with the enable pin. this other mcu has it’s own 3.3v regulator. We are running the same software that we were last week without incident. This week we started running tests on the batteries and we charged and discharged them a lot. The behavior I noticed was that sometimes(not often) when the battery was charged up to about 4.2v, the master mcu would raise the enable pin to 3.3v and the boron would not boot. That is to say no led would turn on at all and my code would not run. I verified that the enable pin was up with the scope, 3.30v on the dot.
I saw this behavior twice yesterday when we had all the batteries charged up, at first I thought I’d droped a screwdriver somwhere and shorted something… Then I saw it on another board. I haven’t been able to replicate it today… I tried basically just re-powering the master mcu like ~50 times… always behaved as expected…
Next I thought that it was a threshold problem with the enable mosfet, because of the voltage increase, we’re getting closer to pinch off. So I hooked psu’s to the enable pin and the li+ pin, I found that the enable threshold was somewhere in the 1.5v range… right where it should be. I tried setting it up like my setup, with 4.2 volts at the li+ and 3.3 at the en but it worked just fine. hmmm…
we’re running particle v1.1 currently in automatic mode.
any help would be appreciated, I’m stumped on this one…
We’re still having this issue. I’m wondering if it could be a problem with the pmic communication… when we see this issue, the enable pin is 3.3, and the regulator is 3.3 but the mcu does not boot and the led doesen’t blink. About 2 weeks ago we added the following lines of code in our setup() routine:
//shamelessley coppied from Rftop on the forums, ensures that the pmic reboots every time the boron restarts and doesen't power down after a no sun scenario, also ups the charge rate limit current.
pmic.begin();
//detachInterrupt(LOW_BAT_UC); //this is necessary if we use boron firmware above v1.1, apparently they screwed up in later versions...
pmic.setInputVoltageLimit(5080); // for 6V Solar Panels
pmic.setChargeVoltage(4208); // Set Li-Po charge termination voltage to 4.21V, // this is a bit high.. ?
pmic.enableDPDM();
pmic.enableBuck(); // enableBuck required when using EN Pin Shutdown.
pmic.enableCharging();
pmic.setChargeCurrent(1,1,1,1,1,1); //no limit on charge current, solar voltage shouldn't drop below 5.08v(set above), so the system is in voltage control mode
we added this a few days before we started seeing this bug, so it would seem to correspond but I’m not sure how that is possible. I guess I don’t really know what some of this means. enableBuck and enabledpdm are especially baffling to me. I assume that enablebuck sets the register in the pmic to enabl the buck converter even while the mcu is off. I couldn’t find too much info on exactly what either of these does.
anyways what I don’t understand about this is that the pmic and the 3.3v regulator seem to be working fine when enabled. I guess what I’m wondering is what the 1st thing the boron does, even before it flashes leds? Is there an i2c conversation with the pmic that could hang and explain my behavior?
I'm still looking for an explanation here.. the latest update on this is that the above pmic code does not seem to be an issue. after testing for almost a week we found that some of the boards without the pmic code would also bug. It seems to just be some kind of latch state in the boot code.
a few days ago we put devices in the field so I had to come up with a workaround fast. The way that I get these devices out of this state is by pulling the battery out and pluging it back in. In the field we can't do that obviously so I use a FET (that i drove all over to find) as a low side switch from another pin on the master mcu. If the boron doesn't respond after a while, I pull power from it for 30 seconds. It's a workaround that 'should' work based on my knowledge of the issue but we didn't actually have time to test it so I'm not sure it works. I'd like to actually figure out whats happening here. I found one other forum member who seems to have the same issue, I'm including it mainly to show that I'm not insane:
That is also my understanding and what I experienced on the early firmwares. This solution is also what I would do, if forced to launch a product with Boron today.
I am still looking for a "one chip solution" that does not need to be programmed, to pull the enable pin for 30s after signs of inactivity.
thanks thrmtn, are you saying that in later versions this is not an issue? I thought that 1.1 was a safe bet in terms of stability but I may be wrong, I’ve heard some horror stories from later versions. I guess I can start some tests on later firmwares.
I only tried it on the early versions almost a year ago, like 0.9.0.
While my Boron still running 0.9.0 is partially offline every week despite good signal, my Argon has been running my AC at home for months on 1.1.0 without trouble. For "safe bet" considerations, I always include the company track record.
If anybody is interested… we did (sort of) track this down…
we communicate from the master to the boron using i2c(pulled to 3.3 w 5k res). when we find one of the boards in the bugstate we cut the i2c line, and it brings it powers on again. Somehow the sda or scl lines are keeping it off. When you measure the voltages of the gpio’s in the bugged state, you see that they’re like .6 below the battery. I guess what’s happening is the sda/scl lines are acting as current sink, from V_li to 3.3v, this explains why it’s more likely to happen in a high battery charge state, that voltage difference is higher.
Currently our workaround is to use the DEEP_SLEEP mode waking up on pin8 instead of the EN pin. This uses a lot more power however, either 700uA or 1100uA (depending on battery level). This really isn’t great for a chip thats supposed to be off. I haven’t really looked into it much but I feel like they could be using bigger resistors somewhere in the pmic/regulator circuit…
Another update on this if anybody is interested/having similar issues…
We still find that the boron doesn’t return from sleep sometimes, so this is still an issue. We’ve tried switching it on/off with an external lowside fet but it didn’t have any effect. That was with a low side switch, which was basically redundant to the EN pin. Our next attempt will be a high side switch, this should not allow current through the li+ pin. It will also not allow us to use the charging circuit. We know that pulling power entirely does get it unstuck, we’re just having difficulty actually doing that electrically. This is because, I think the current if flowing from v_li to SCl/SDA, not to gnd, so when we un grounded the circuit, it didn’t do much.
I think this is some kind of hardware issue with the charging/power circuit, because we can sleep it from the en pin or from software and we get the same result.
Has anybody had a similar issue??? Would moving to SPI help or be basically the same?
i2c devices (slaves) sometimes leak current from their supply to inactive i2c lines. In the past I have cut the power to the i2c devices while sleeping, either by supplying them from pins on the master directly, or with a dedicated supply/switch for peripherals.
Depending on the type of master, I have also tri-state it’s i2c lines before sleeping and initialised i2c again when waking. Sometimes grounding pins is said to results in lower sleep current, but not a great idea with leaking slaves, or other unforeseen failure situations.
Yeah, this is kind of what I’ve been thinking. On Friday I figured out how to increase the internal pull ups on the master mcu… Turns out the default was only 1.5k, I upped that to 12k… I set that board running on Saturday, haven’t seen it bug yet so I’m keeping my fingers crossed…
I’ve been trying to set the master sda and scl pins to normal input pins (high imp) while they’re not being used. Having limited luck with this though, i think the boron doesn’t understand the floating voltages… more to come on that I guess.
We’re also seeing this issue with our newly created TinySupervisor feather, initially designed by jacky4566 and adapted to be an external wdt and low power device.
What we are seeing is that sometimes the Boron does not want to start up when applying power to the enable pin. Seems that the PMIC is in a dodgy state, Either because of parasitic power it’s leeching from somewhere or something else
So measuring the connected pins we can see nearly 35uA per pin connected going too the Boron from our feather (SDA, SCL, EN and even 3v3 had a little). So something is definitely leaking.
Have you made any further progress finding a solution?
Have you tried switching the GND rather than VCC? sometimes even when the the unit is disabled it is “backfed” from the SPI / I2C lines. If you remove the ground, rather than VCC, it removes the ability to for it sink current through the I/O lines.
Not sure if it is the problem or not, I haven’t tried it on Particle device but it definitely helps/works on other devices.
johnny, I’m not sure what you’re tinysupervisor is but if it uses i2c and the enable pin, it should b succeptable to this bug state, you’re right in that the current seems to be flowing through the pmic/buck converter…
Marshall- yes, this is what I tried at first. I figured that since the EN pin used a pmosfet, I’d try an nmos to gnd. I did this and attempted to field a bunch of test boards that all failed within a week… We were able to replicate this in the lab, even with the ground pulled, the board bugs. only when I pull the i2c connector, it can come out of the bugstate, it seems that current flows from the +li down through the pmic and is sunk by the i2c lines. I still don’t fully understand the path its taking…
also, when selecting a mosfet, be aware that the boron can sink like 200ma momentarily during tx. Make sure that the fet has a very low on resistance, otherwise you’ll find that it will never connect to the cloud… I was in a pinch and had to drive all over eastern mass to find the right fet the day before our last field trial. It’s dificult to find a logic level fet that would meet the bill in hobby shops. There’s a place called you-do-it electronics in needham if you’re ever desperate and can’t wait for digi.
We recently fielded some more test boards with a pmosfet at the LI+ pin. This seems to work, we had 6 boards running for a week, so we think it’s ok?? its very difficult to tell/prove… The problem with this is that you can no longer use the boron for solar charging, we used an external charger from adafruit. This takes away a lot of the value of the boron. A better solution is to use an i2c isolator, which I’ve included in my next batch of boards. I’m testing with an ISO1540 right now which seems to be going well, I don’t think it’s possible for current to get through this while it’s powered down, it should b high impedence.
Chip, I think that’s referring to “with a pmosfet at the LI+ pin”, since the Boron (and it’s charging circuit) would no longer be connected to the Li-Po during Shutdown with this approach.
that’s right Rftop- well sort of… This depends a bit on the pmos you’re using, I’m using one with a reverse diode so it’s pretty safe, in fact I think you could charge the battery but it would be very ineficient. what would happen is the boron charger would try to dump charge to the battery, thus raising the li+ voltage trying to force current into the battery. The pmos is obviously a unidirectional device, if it were a jfet you could do this without an issue, but a jfet would be much less efficient… the dangerous thing is if you bring the drain above the source voltage momentarily then I guess just acts like a diode and it goes into avalanche mode which is bad. I don’t really know what that means but I vaguely remember professor Mcneil telling me not to do it. Anyways, on most power mosfets you’ll see that there’s a little protection diode from d-s or s-d, that’s a diode w a smaller voltage drop than the one on the mosfet, to ensure that never happens. All that being said- now that I think about it I guess you can charge with this setup, but it will be much less efficient because you’re applying yet another diode drop.