 # Solved! Long Longs are (timb was) Wrong Wrong?

Has anyone had trouble with 64-bit variables on the Core?

I’m working on a library for the MS5607 Barometric Pressure Sensor and getting weird results. Here’s my code:

``````offP = calData * 131072 + (calData * deltaT) / 64;
sensP = calData * 65536 + (calData * deltaT) / 128;
P = (rawP * sensP / 2097152 - offP) / 32768;
``````

`P`, `offP` and `sensP` are `int64_t` types. `rawP` is a `uint32_t` and `calData[]` is an array of the type `uint16_t`.

Here’s a flowchart from the datasheet: It’s returning completely invalid results. I’ve done the math by hand and know what it should be reporting (around 105000).

``````MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xA4
44527
40588
27507
25360
32823
27567
Temp C: 2035
Temp F: 6863
Pressure: -32072
``````

The first six numbers are the six entries in the `calData[]` array (C1 through C6 in the flowchart). The temperature results are correct (20.35C/68.63F), but to me it looks like the pressure variable is overflowing, since it should never return a negative value…

The compiler might be multiplying 32bit x 64bit first which could be larger than 64bit (explaining the overflow).

Try this:

``````int64_t temp = sensP / 2097152; // widdle that 64bit number down a bit first
temp = rawP * temp; // then bloat it back up
P = (temp - offP) / 32768;
``````

What is the type of `deltaT`? If it’s not a `long long`, your expression is being computed as an `int` (32-bit, which you’re overflowing) before being stored into a `long long`.

No, a 32-bit x 64-bit can’t overflow, because the result is 64-bit.

Say what?

64bit x 2 = 65bit, which overflows back into 64bit space.

However since this is signed, it would be 7FFF FFFF FFFF FFFF x 2 which would not overflow to 65 bits, but multiply is by 3 and it will (i.e. a 2 bit number)

64bit x 32bit is definitely going to overflow.

32bit x 32bit is just going to be 64bit, but not overflow.

Gah, you’re right, and I need about a gallon more coffee …

1 Like

No problem… I pour this shtuff in my daily!! so I’m surprised I can even read anymore…

Still, the original question stands: if `deltaT` is not a 64-bit datatype, everything’s being computed as 32-bit, which will overflow.

No, wait, it’s going to overflow anyway:

``````offP = calData * 131072 + (calData * deltaT) / 64;
``````

The first two terms are 32-bit:

``````calData * 131072
``````

The result of this is going to be 32-bit, which will overflow (`caldata` is 40588, if I’m interpreting his data correctly, and that will overflow).

EDIT: To force the expected computation, we can go old-school:

``````offP = (long long)calData * 131072 + ((long long)calData * deltaT) / 64;
``````

Er, to make it more readable, perhaps we should use:

``````offP = calData * 131072LL + ((long long)calData * deltaT) / 64;
``````

Here’s the whole sketch. @BDub’s suggestion didn’t work. (Note, the only way I can read the data is by casting `int64_t P` as an `int32_t` with the `Serial1.println` command, as it doesn’t support long longs.)

``````#define MS5607_ADDR 0x76
#define RESET_REG 0x1E
#define PROM_BASE_REG 0xA0

uint16_t calData = {0, 0, 0, 0, 0, 0};

uint32_t rawT;
uint32_t rawP;

int32_t deltaT;
int32_t Tc;
int32_t Tf;

int64_t offP;
int64_t sensP;
int64_t P;

void setup() {
Wire.begin();
Serial1.begin(115200);
Serial1.println("MS5607 Library by @TimothyBrown");
Serial1.println("Version: 20130323 - Build: 0xA7");
sensorStart();
getData();
Serial1.print("Temp C: ");
Serial1.println(Tc);
Serial1.print("Temp F: ");
Serial1.println(Tf);
Serial1.print("Pressure: ");
Serial1.println((int)P);
}

void loop() {

}

void sensorStart() {
Wire.write(RESET_REG);
Wire.endTransmission();
delay(5);
uint8_t calReg = PROM_BASE_REG;
for(uint8_t i = 0; i < 6; i++) {
calReg = calReg + 0x02;
Wire.write(calReg);
Wire.endTransmission();
delay(1);
while(Wire.available()) {
calData[i] = calData[i] << 8 | Wire.read();

}
Serial1.println(calData[i]);
}
}

void getData() {

/* Read Raw Temperature Data */
Wire.write(0x58);
Wire.endTransmission();
delay(10);
Wire.write(0x00);
Wire.endTransmission();
delay(1);
while(Wire.available()) {
rawT = rawT << 8 | Wire.read();
rawT = rawT << 8 | Wire.read();
}

/* Read Raw Pressure Data */
Wire.write(0x48);
Wire.endTransmission();
delay(10);
Wire.write(0x00);
Wire.endTransmission();
delay(1);
while(Wire.available()) {
rawT = rawP << 8 | Wire.read();
rawT = rawP << 8 | Wire.read();
}

/* Process Temperature */
deltaT = rawT - calData * 256;
Tc = 2000 + deltaT * calData / 8388608;
Tf = Tc * 1.8 + 3200;

/* Process Pressure */
offP = calData * 131072 + (calData * deltaT) / 64;
sensP = calData * 65536 + (calData * deltaT) / 128;
//P = (rawP * sensP / 2097152 - offP) / 32768;
int64_t x = sensP / 2097152; // widdle that 64bit number down a bit first
x = rawP * x; // then bloat it back up
P = (x - offP) / 32768;

} ``````

Thinking about what @Raldus said, can I just cast the 16 and 32-Bit variables as 64-Bit ones when doing math on them?

Yes, `calData * 131072` will overflow if `calData` is 16K or larger.

EDIT: I’m not sure, but I think an easier way is to simply declare `calData` as an `unsigned long long`. The values in `calData` appear to be only used in computations, and do not seem to be sent back to any hardware registers and the like.

``````char buf;
long long x = 42;

sprintf(buf, "x = %lld", x);
Serial.println(buf);
``````

or, in 64-bit hex:

``````char buf;
long long x = 42;

sprintf(buf, "x = 0x%016llx", x);
Serial.println(buf);``````

Unfortunately that screws up the temperature calculations:

``````MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xA8
44527
40588
27507
25360
32823
27567
Temp C: -25613
Temp F: -42903
Pressure: -8796093082952
``````

The same thing happens if I make `deltaT` 64-Bit. I’m not used to dealing with 64-Bit math in this fashion, so this is breaking my brain.

``````float MS561101BA::getPressure(uint8_t OSR) {
// see datasheet page 7 for formulas

int32_t dT = getDeltaTemp(OSR);
if(dT == NULL) {
return NULL;
}

uint32_t rawPress = rawPressure(OSR);
if(rawPress == NULL) {
return NULL;
}

int64_t off  = ((uint32_t)_C <<16) + (((int64_t)dT * _C) >> 7);
int64_t sens = ((uint32_t)_C <<15) + (((int64_t)dT * _C) >> 8);
return ((( (rawPress * sens ) >> 21) - off) >> 15) / 100.0;
}
``````

I’m studying the rest of this Arduino library I found, and it seems they store all their variables in a similar method to the way I’m doing it. I’m doing some bit twiddling now and making a bit of progress.

Behold, progress!

``````MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xB1

Temp C: 2034
Temp F: 6861
Pressure: 70330
``````

``````offP = ((uint32_t)calData * 131072) + ((calData * (int64_t)deltaT) / 64);
sensP = ((uint32_t)calData * 65536) + ((calData * (int64_t)deltaT) / 128);
P = ((((rawP * sensP) / 2097152) - offP) / 32768);``````

Ah,

``````int64_t off  = ((uint32_t)_C <<16) + (((int64_t)dT * _C) >> 7);
``````

This is a bit different than your code :

``````offP = calData * 131072 + (calData * deltaT) / 64;
``````

This is the same as:

``````offP = calData << 17 + (calData * deltaT) >> 6;
``````

Sorry. Edit - OK, I’m really confused. If you’ve got things working, I’ll just leave you alone. Yeah, I know. I’m following a newer algorithm for the offset calculations then they used in their code.

``````Serial1.print("Pressure: 0x");
Serial1.print((int32_t)((P>>32)&0xFFFFFFFF),HEX);
Serial1.println((int32_t)(P&0xFFFFFFFF),HEX);
``````

maybe start here and see what you really have?

Yeah, I’m not using that method to print it anymore. I used @Raldus’ helpful CPP snippet.

``````Serial1.print("Pressure: ");
char buf;
sprintf(buf, "%lld", P);
Serial1.println(buf);
Serial1.print("Pressure: 0x");
Serial1.print((int32_t)((P>>32)&0xFFFFFFFF),HEX);
Serial1.println((int32_t)(P&0xFFFFFFFF),HEX);
``````

``````MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xB1
Temp C: 2035
Temp F: 6863
Pressure: 70329
Pressure: 0x0112B9
``````

``````0x0112B9 in dec => 70,329
``````

So it matches up. That would be 703.29mb of pressure. I’m trying to find the altitude of the nearby airport so I can convert it to a value at sea-level and compare it to the current weather map.

FYI: sprintf() consumes 20KB of FLASH In case you end up running out of space…