Solved! Long Longs are (timb was) Wrong Wrong?

timb · March 27, 2014, 1:09am

Has anyone had trouble with 64-bit variables on the Core?

I’m working on a library for the MS5607 Barometric Pressure Sensor and getting weird results. Here’s my code:

offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;
sensP = calData[0] * 65536 + (calData[2] * deltaT) / 128;
P = (rawP * sensP / 2097152 - offP) / 32768;

P, offP and sensP are int64_t types. rawP is a uint32_t and calData[] is an array of the type uint16_t.

Here’s a flowchart from the datasheet:

It’s returning completely invalid results. I’ve done the math by hand and know what it should be reporting (around 105000).

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xA4
44527
40588
27507
25360
32823
27567
Temp C: 2035
Temp F: 6863
Pressure: -32072

The first six numbers are the six entries in the calData[] array (C1 through C6 in the flowchart). The temperature results are correct (20.35C/68.63F), but to me it looks like the pressure variable is overflowing, since it should never return a negative value…

BDub · March 27, 2014, 1:50am

The compiler might be multiplying 32bit x 64bit first which could be larger than 64bit (explaining the overflow).

Try this:

int64_t temp = sensP / 2097152; // widdle that 64bit number down a bit first
temp = rawP * temp; // then bloat it back up
P = (temp - offP) / 32768;

Raldus · March 27, 2014, 1:52am

What is the type of deltaT? If it’s not a long long, your expression is being computed as an int (32-bit, which you’re overflowing) before being stored into a long long.

Raldus · March 27, 2014, 1:54am

No, a 32-bit x 64-bit can't overflow, because the result is 64-bit.

BDub · March 27, 2014, 2:03am

Say what?

64bit x 2 = 65bit, which overflows back into 64bit space.

However since this is signed, it would be 7FFF FFFF FFFF FFFF x 2 which would not overflow to 65 bits, but multiply is by 3 and it will (i.e. a 2 bit number)

64bit x 32bit is definitely going to overflow.

32bit x 32bit is just going to be 64bit, but not overflow.

Raldus · March 27, 2014, 2:05am

Gah, you're right, and I need about a gallon more coffee ...

BDub · March 27, 2014, 2:07am

No problem… I pour this shtuff in my daily!! so I’m surprised I can even read anymore…

Raldus · March 27, 2014, 2:20am

Still, the original question stands: if deltaT is not a 64-bit datatype, everything’s being computed as 32-bit, which will overflow.

No, wait, it’s going to overflow anyway:

offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;

The first two terms are 32-bit:

calData[1] * 131072

The result of this is going to be 32-bit, which will overflow (caldata[1] is 40588, if I’m interpreting his data correctly, and that will overflow).

EDIT: To force the expected computation, we can go old-school:

offP = (long long)calData[1] * 131072 + ((long long)calData[3] * deltaT) / 64;

Er, to make it more readable, perhaps we should use:

offP = calData[1] * 131072LL + ((long long)calData[3] * deltaT) / 64;

timb · March 27, 2014, 2:25am

Here’s the whole sketch. @BDub’s suggestion didn’t work. (Note, the only way I can read the data is by casting int64_t P as an int32_t with the Serial1.println command, as it doesn’t support long longs.)

#define MS5607_ADDR 0x76
#define RESET_REG 0x1E
#define PROM_BASE_REG 0xA0


uint16_t calData[6] = {0, 0, 0, 0, 0, 0};

uint32_t rawT;
uint32_t rawP;

int32_t deltaT;
int32_t Tc;
int32_t Tf;

int64_t offP;
int64_t sensP;
int64_t P;

void setup() {
    Wire.begin();
    Serial1.begin(115200);
    Serial1.println("MS5607 Library by @TimothyBrown");
    Serial1.println("Version: 20130323 - Build: 0xA7");
    sensorStart();
    getData();
    Serial1.print("Temp C: ");
    Serial1.println(Tc);
    Serial1.print("Temp F: ");
    Serial1.println(Tf);
    Serial1.print("Pressure: ");
    Serial1.println((int)P);
}

void loop() {

}

void sensorStart() {
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(RESET_REG);
    Wire.endTransmission();
    delay(5);
    uint8_t calReg = PROM_BASE_REG;
    for(uint8_t i = 0; i < 6; i++) {
        calReg = calReg + 0x02;
        Wire.beginTransmission(MS5607_ADDR);
        Wire.write(calReg);
        Wire.endTransmission();
        delay(1);
        Wire.requestFrom(MS5607_ADDR, 2);
        while(Wire.available()) {
            calData[i] = Wire.read();
            calData[i] = calData[i] << 8 | Wire.read();
            
        }
    Serial1.println(calData[i]);
    }
}

void getData() {
    
    /* Read Raw Temperature Data */
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x58);
    Wire.endTransmission();
    delay(10);
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x00);
    Wire.endTransmission();
    delay(1);
    Wire.requestFrom(MS5607_ADDR, 3);
    while(Wire.available()) {
        rawT = Wire.read();
        rawT = rawT << 8 | Wire.read();
        rawT = rawT << 8 | Wire.read();
    }
    
    /* Read Raw Pressure Data */
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x48);
    Wire.endTransmission();
    delay(10);
    Wire.beginTransmission(MS5607_ADDR);
    Wire.write(0x00);
    Wire.endTransmission();
    delay(1);
    Wire.requestFrom(MS5607_ADDR, 3);
    while(Wire.available()) {
        rawT = Wire.read();
        rawT = rawP << 8 | Wire.read();
        rawT = rawP << 8 | Wire.read();
    }
    
    /* Process Temperature */
    deltaT = rawT - calData[4] * 256;
    Tc = 2000 + deltaT * calData[5] / 8388608;
    Tf = Tc * 1.8 + 3200;
    
    /* Process Pressure */
    offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;
    sensP = calData[0] * 65536 + (calData[2] * deltaT) / 128;
    //P = (rawP * sensP / 2097152 - offP) / 32768;
    int64_t x = sensP / 2097152; // widdle that 64bit number down a bit first
    x = rawP * x; // then bloat it back up
    P = (x - offP) / 32768;
    
}

timb · March 27, 2014, 2:26am

Thinking about what @Raldus said, can I just cast the 16 and 32-Bit variables as 64-Bit ones when doing math on them?

Raldus · March 27, 2014, 2:27am

Yes, calData[1] * 131072 will overflow if calData[1] is 16K or larger.

EDIT: I’m not sure, but I think an easier way is to simply declare calData as an unsigned long long. The values in calData appear to be only used in computations, and do not seem to be sent back to any hardware registers and the like.

Raldus · March 27, 2014, 2:33am

char buf[100];
long long x = 42;

sprintf(buf, "x = %lld", x);
Serial.println(buf);

or, in 64-bit hex:

char buf[100];
long long x = 42;

sprintf(buf, "x = 0x%016llx", x);
Serial.println(buf);

timb · March 27, 2014, 2:42am

Unfortunately that screws up the temperature calculations:

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xA8
44527
40588
27507
25360
32823
27567
Temp C: -25613
Temp F: -42903
Pressure: -8796093082952

The same thing happens if I make deltaT 64-Bit. I'm not used to dealing with 64-Bit math in this fashion, so this is breaking my brain.

timb · March 27, 2014, 3:00am

Here we go, something helpful!

float MS561101BA::getPressure(uint8_t OSR) {
  // see datasheet page 7 for formulas
  
  int32_t dT = getDeltaTemp(OSR);
  if(dT == NULL) {
    return NULL;
  }
  
  uint32_t rawPress = rawPressure(OSR);
  if(rawPress == NULL) {
    return NULL;
  }
  
  int64_t off  = ((uint32_t)_C[1] <<16) + (((int64_t)dT * _C[3]) >> 7);
  int64_t sens = ((uint32_t)_C[0] <<15) + (((int64_t)dT * _C[2]) >> 8);
  return ((( (rawPress * sens ) >> 21) - off) >> 15) / 100.0;
}

I’m studying the rest of this Arduino library I found, and it seems they store all their variables in a similar method to the way I’m doing it. I’m doing some bit twiddling now and making a bit of progress.

timb · March 27, 2014, 3:18am

Behold, progress!

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xB1

Temp C: 2034
Temp F: 6861
Pressure: 70330

offP = ((uint32_t)calData[1] * 131072) + ((calData[3] * (int64_t)deltaT) / 64);
sensP = ((uint32_t)calData[0] * 65536) + ((calData[2] * (int64_t)deltaT) / 128);
P = ((((rawP * sensP) / 2097152) - offP) / 32768);

Raldus · March 27, 2014, 3:20am

Ah,

int64_t off  = ((uint32_t)_C[1] <<16) + (((int64_t)dT * _C[3]) >> 7);

This is a bit different than your code :

offP = calData[1] * 131072 + (calData[3] * deltaT) / 64;

This is the same as:

offP = calData[1] << 17 + (calData[3] * deltaT) >> 6;

Sorry.

Edit - OK, I’m really confused. If you’ve got things working, I’ll just leave you alone.

timb · March 27, 2014, 3:24am

Yeah, I know. I’m following a newer algorithm for the offset calculations then they used in their code.

BDub · March 27, 2014, 3:35am

Serial1.print("Pressure: 0x");
Serial1.print((int32_t)((P>>32)&0xFFFFFFFF),HEX);
Serial1.println((int32_t)(P&0xFFFFFFFF),HEX);

maybe start here and see what you really have?

timb · March 27, 2014, 3:45am

Yeah, I’m not using that method to print it anymore. I used @Raldus’ helpful CPP snippet.

Serial1.print("Pressure: ");
char buf[100];
sprintf(buf, "%lld", P);
Serial1.println(buf);
Serial1.print("Pressure: 0x");
Serial1.print((int32_t)((P>>32)&0xFFFFFFFF),HEX);
Serial1.println((int32_t)(P&0xFFFFFFFF),HEX);

MS5607 Library by @TimothyBrown
Version: 20130323 - Build: 0xB1
Temp C: 2035
Temp F: 6863
Pressure: 70329
Pressure: 0x0112B9

0x0112B9 in dec => 70,329

So it matches up. That would be 703.29mb of pressure. I’m trying to find the altitude of the nearby airport so I can convert it to a value at sea-level and compare it to the current weather map.

BDub · March 27, 2014, 3:57am

FYI: sprintf() consumes 20KB of FLASH In case you end up running out of space…

Topic		Replies	Views
Problems when using long long Troubleshooting	20	4069	May 19, 2017
DS18B20 Library ERROR: Bad readings below 0C / 32F Troubleshooting boron	13	864	March 30, 2019
Q: Does the length of an integer datatype impact the precision of calculation? General	9	1599	June 14, 2015
Weird Results for a variable named "T9" Troubleshooting	13	3150	March 21, 2016
Weird/Wrong values for Particle Variables Troubleshooting xenon	3	916	March 13, 2019

Solved! Long Longs are (timb was) Wrong Wrong?

Related Topics