Solved! Long Longs are (timb was) Wrong Wrong?

Has anyone had trouble with 64-bit variables on the Core?

I’m working on a library for the MS5607 Barometric Pressure Sensor and getting weird results. Here’s my code:

offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;
sensP = calData[0] * 65536 + (calData[2] * deltaT) / 128;
P = (rawP * sensP / 2097152 - offP) / 32768;

P, offP and sensP are int64_t types. rawP is a uint32_t and calData[] is an array of the type uint16_t.

Here’s a flowchart from the datasheet:

It’s returning completely invalid results. I’ve done the math by hand and know what it should be reporting (around 105000).

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xA4
44527
40588
27507
25360
32823
27567
Temp C: 2035
Temp F: 6863
Pressure: -32072

The first six numbers are the six entries in the calData[] array (C1 through C6 in the flowchart). The temperature results are correct (20.35C/68.63F), but to me it looks like the pressure variable is overflowing, since it should never return a negative value…

The compiler might be multiplying 32bit x 64bit first which could be larger than 64bit (explaining the overflow).

Try this:

int64_t temp = sensP / 2097152; // widdle that 64bit number down a bit first
temp = rawP * temp; // then bloat it back up
P = (temp - offP) / 32768;

What is the type of deltaT? If it’s not a long long, your expression is being computed as an int (32-bit, which you’re overflowing) before being stored into a long long.

No, a 32-bit x 64-bit can’t overflow, because the result is 64-bit.

Say what?

64bit x 2 = 65bit, which overflows back into 64bit space.

However since this is signed, it would be 7FFF FFFF FFFF FFFF x 2 which would not overflow to 65 bits, but multiply is by 3 and it will (i.e. a 2 bit number)

64bit x 32bit is definitely going to overflow.

32bit x 32bit is just going to be 64bit, but not overflow.

Gah, you’re right, and I need about a gallon more coffee …

1 Like

No problem… I pour this shtuff in my :eyes: daily!! :coffee: so I’m surprised I can even read anymore…

Still, the original question stands: if deltaT is not a 64-bit datatype, everything’s being computed as 32-bit, which will overflow.

No, wait, it’s going to overflow anyway:

offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;

The first two terms are 32-bit:

calData[1] * 131072

The result of this is going to be 32-bit, which will overflow (caldata[1] is 40588, if I’m interpreting his data correctly, and that will overflow).

EDIT: To force the expected computation, we can go old-school:

offP = (long long)calData[1] * 131072 + ((long long)calData[3] * deltaT) / 64;

Er, to make it more readable, perhaps we should use:

offP = calData[1] * 131072LL + ((long long)calData[3] * deltaT) / 64;

Here’s the whole sketch. @BDub’s suggestion didn’t work. (Note, the only way I can read the data is by casting int64_t P as an int32_t with the Serial1.println command, as it doesn’t support long longs.)

#define MS5607_ADDR 0x76
#define RESET_REG 0x1E
#define PROM_BASE_REG 0xA0


uint16_t calData[6] = {0, 0, 0, 0, 0, 0};

uint32_t rawT;
uint32_t rawP;

int32_t deltaT;
int32_t Tc;
int32_t Tf;

int64_t offP;
int64_t sensP;
int64_t P;

void setup() {
    Wire.begin();
    Serial1.begin(115200);
    Serial1.println("MS5607 Library by @TimothyBrown");
    Serial1.println("Version: 20130323 - Build: 0xA7");
    sensorStart();
    getData();
    Serial1.print("Temp C: ");
    Serial1.println(Tc);
    Serial1.print("Temp F: ");
    Serial1.println(Tf);
    Serial1.print("Pressure: ");
    Serial1.println((int)P);
}

void loop() {

}

void sensorStart() {
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(RESET_REG);
    Wire.endTransmission();
    delay(5);
    uint8_t calReg = PROM_BASE_REG;
    for(uint8_t i = 0; i < 6; i++) {
        calReg = calReg + 0x02;
        Wire.beginTransmission(MS5607_ADDR);
        Wire.write(calReg);
        Wire.endTransmission();
        delay(1);
        Wire.requestFrom(MS5607_ADDR, 2);
        while(Wire.available()) {
            calData[i] = Wire.read();
            calData[i] = calData[i] << 8 | Wire.read();
            
        }
    Serial1.println(calData[i]);
    }
}

void getData() {
    
    /* Read Raw Temperature Data */
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x58);
    Wire.endTransmission();
    delay(10);
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x00);
    Wire.endTransmission();
    delay(1);
    Wire.requestFrom(MS5607_ADDR, 3);
    while(Wire.available()) {
        rawT = Wire.read();
        rawT = rawT << 8 | Wire.read();
        rawT = rawT << 8 | Wire.read();
    }
    
    /* Read Raw Pressure Data */
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x48);
    Wire.endTransmission();
    delay(10);
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x00);
    Wire.endTransmission();
    delay(1);
    Wire.requestFrom(MS5607_ADDR, 3);
    while(Wire.available()) {
        rawT = Wire.read();
        rawT = rawP << 8 | Wire.read();
        rawT = rawP << 8 | Wire.read();
    }
    
    /* Process Temperature */
    deltaT = rawT - calData[4] * 256;
    Tc = 2000 + deltaT * calData[5] / 8388608;
    Tf = Tc * 1.8 + 3200;
    
    /* Process Pressure */
    offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;
    sensP = calData[0] * 65536 + (calData[2] * deltaT) / 128;
    //P = (rawP * sensP / 2097152 - offP) / 32768;
    int64_t x = sensP / 2097152; // widdle that 64bit number down a bit first
    x = rawP * x; // then bloat it back up
    P = (x - offP) / 32768;
    
} 

Thinking about what @Raldus said, can I just cast the 16 and 32-Bit variables as 64-Bit ones when doing math on them?

Yes, calData[1] * 131072 will overflow if calData[1] is 16K or larger.

EDIT: I’m not sure, but I think an easier way is to simply declare calData as an unsigned long long. The values in calData appear to be only used in computations, and do not seem to be sent back to any hardware registers and the like.

char buf[100];
long long x = 42;

sprintf(buf, "x = %lld", x);
Serial.println(buf);

or, in 64-bit hex:

char buf[100];
long long x = 42;

sprintf(buf, "x = 0x%016llx", x);
Serial.println(buf);

Unfortunately that screws up the temperature calculations:

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xA8
44527
40588
27507
25360
32823
27567
Temp C: -25613
Temp F: -42903
Pressure: -8796093082952

The same thing happens if I make deltaT 64-Bit. I’m not used to dealing with 64-Bit math in this fashion, so this is breaking my brain.

Here we go, something helpful!

float MS561101BA::getPressure(uint8_t OSR) {
  // see datasheet page 7 for formulas
  
  int32_t dT = getDeltaTemp(OSR);
  if(dT == NULL) {
    return NULL;
  }
  
  uint32_t rawPress = rawPressure(OSR);
  if(rawPress == NULL) {
    return NULL;
  }
  
  int64_t off  = ((uint32_t)_C[1] <<16) + (((int64_t)dT * _C[3]) >> 7);
  int64_t sens = ((uint32_t)_C[0] <<15) + (((int64_t)dT * _C[2]) >> 8);
  return ((( (rawPress * sens ) >> 21) - off) >> 15) / 100.0;
}

I’m studying the rest of this Arduino library I found, and it seems they store all their variables in a similar method to the way I’m doing it. I’m doing some bit twiddling now and making a bit of progress.

Behold, progress!

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xB1

Temp C: 2034
Temp F: 6861
Pressure: 70330

offP = ((uint32_t)calData[1] * 131072) + ((calData[3] * (int64_t)deltaT) / 64);
sensP = ((uint32_t)calData[0] * 65536) + ((calData[2] * (int64_t)deltaT) / 128);
P = ((((rawP * sensP) / 2097152) - offP) / 32768);

Ah,

int64_t off  = ((uint32_t)_C[1] <<16) + (((int64_t)dT * _C[3]) >> 7);

This is a bit different than your code :frowning: :

offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;

This is the same as:

offP = calData[1] << 17 + (calData[3] * deltaT) >> 6;

Sorry. :frowning:

Edit - OK, I’m really confused. If you’ve got things working, I’ll just leave you alone. :slight_smile:

Yeah, I know. I’m following a newer algorithm for the offset calculations then they used in their code.

Serial1.print("Pressure: 0x");
Serial1.print((int32_t)((P>>32)&0xFFFFFFFF),HEX);
Serial1.println((int32_t)(P&0xFFFFFFFF),HEX);

maybe start here and see what you really have?

Yeah, I’m not using that method to print it anymore. I used @Raldus’ helpful CPP snippet.

Serial1.print("Pressure: ");
char buf[100];
sprintf(buf, "%lld", P);
Serial1.println(buf);
Serial1.print("Pressure: 0x");
Serial1.print((int32_t)((P>>32)&0xFFFFFFFF),HEX);
Serial1.println((int32_t)(P&0xFFFFFFFF),HEX);

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xB1
Temp C: 2035
Temp F: 6863
Pressure: 70329
Pressure: 0x0112B9

0x0112B9 in dec => 70,329 

So it matches up. That would be 703.29mb of pressure. I’m trying to find the altitude of the nearby airport so I can convert it to a value at sea-level and compare it to the current weather map.

FYI: sprintf() consumes 20KB of FLASH :wink: In case you end up running out of space…