I’ve finally isolated what’s causing my otherwise perfectly operational I2C stream to fail. Specifically, it’s when I’m actively performing a LittleFS write() update that the I2C data stream gets corrupted.
My I2C streaming conversion runs in a 40Hz thread, and after 1 second of data accumulation, I package up a small data struct (440 bytes) for a LittleFS write() update in my mainline loop. The I2C read corruption occurs during the active write() operation, and returns to normal once the write() operation has been completed.
Running DevOS v6.3.3 on a Photon2.
Any insight appreciated…
I'd guess it's a timing issue when LittleFS writes out blocks to the flash when it interrupts between I2C writes.
What I'd try is to create a mutex that surrounds your entire I2C operation. But also obtain the same mutex when writing to a file. This will prevent the two operations from occurring at the same time without affecting threading in the rest of the system.
Come to think of it, this same operation was running previously in an earlier v5.8.0 implementation, so I’m going to give it a go with that earlier OS.
Back when I first characterized the LitlleFS write() operation it took well over 25ms to persist to flash, so mutex’ing between I2C and LittleFS is a non-starter as it would block my 40Hz threading that attempts to sample in real time.
I finally had a chance to verify this, and after downgrading my Photon 2 to v.5.8.0, the I2C corruption has been mitigated. I’m not sure where in the process of up-versioned Dev OS code this breaks specifically, but it’s definitely an issue in v6.3.3.
As to the issue specifically, it’s as described in my first post. The I2C stream corruption occurs during the LittleFS operations that include:
The 480 byte update times can be seen here (1Hz) with some logging posts
(showing the millis() clock):
*Start update: 176417
*End update: 176582
*Start update: 177417
*End update: 177587
*Start update: 178427
*End update: 178598*
It’s during the approximate 165ms LittleFS update that the I2C data is corrupted. The I2C thread runs at (40Hz / 25ms) and has at least 6 conversions/updates during this open/write/close time.
p.s. the logger content is spooled off to a FIFO that’s serviced for display in loop()
Hi @jimini, thanks for the debugging here! We’re seeing something potentially similar with I2C corruption & a fuel gauge on a customer project. We are in the process of root causing, we’ll let you know where we get with it / if we have any questions for you.
@jimini when you say "I2C stream corruption", do you mean the values are corrupted or is it just the timing of reads changes?
I created a sample app to reproduce the issue with the internal FuelGauge::getSoC() as the I2C device, and the values are never corrupted, but the timing of reads can be delayed a bit from the nominal "every 25ms" while the file is being written. It is delayed in chunks though, so the longest is usually only around 45ms between I2C reads.
Sorry for my lag (on vacation).
The data does get clocked in corruptly.
I might have some data logs captured somewhere if that's of any benefit?
For now I've rolled back to v5.8.0.
Yes, any logs would be helpful. You can DM them to me if you'd like to.
Also more info on what I2C reads you are doing... is a long transaction, how many bytes, what bus speed, anything special like a double start?
Are you using the default thread priority for your I2C thread? These don't produce much different results for me.
How are you running it at 40Hz? delay(25) or shorter delay(1) and millis() timer check, or? These do produce different results in terms of how much priority your thread gets.
I tried writing a bunch of data to a file in loop(), then dumping everything the I2C FIFO has received in the last 500ms / 1s / 2s every time through loop(). I used a custom I2C buffer as well, but my reads are really short... only sets up the address, then reads two bytes, which is 5 bytes total. No data corruption. The file writing is timed to take 200ms, and I can see it delaying the timing of the I2C reads. I am using a mutex when saving and accessing data from the FIFO though, not when writing to the file. Just so I don't read partially written data to the FIFO. Not sure if you are doing something like that?