PublishQueueExtRK corrupted file

gusgonnet · April 4, 2025, 10:07pm

Hi, I'm using the PublishQueueExtRK library, and noticed that some times some devices do not publish what needs to be published.

I BELIEVE I have never observed this up until I started using the BINARY publish version where I publish a variant.
Could it be that my code is creating this variant in a way (maybe a wrong way) that impacts the PublishQueueExtRK library?

One observed log:

0000082668 [app.pubq] INFO: queue files invalid magic 0x31372c30 /usr/pubqueue2/00000001.pq
0000082671 [app.pubq] INFO: invalid sizes dataSize=741488428 metaSize=14387 /usr/pubqueue2/00000001.pq
0000082681 [app.pubq] INFO: discarding corrupted file 1

My code uses the library like this:

setup() {
  PublishQueueExt::instance().setup();
  PublishQueueExt::instance().withFileQueueSize(20);
}

loop() {
  PublishQueueExt::instance().loop();
}


fx_that_publishes_text_off_loop(){
        bool success = PublishQueueExt::instance().publish(eventName, data);
}

another_fx_that_publishes_a_variant_off_loop(){
    PublishQueueExt::instance().publish(EVENT_NAME_STATS, v, ContentType::BINARY);
}

Thanks!

rickkas7 · April 5, 2025, 4:50pm

A post was split to a new topic: PublishQueuePosixRK during setup

rickkas7 · April 5, 2025, 5:39pm

How big is the binary data? Do you have a log of this happening? Or the actual binary data? Does it always happen?

I added a new option to the test suite to test variant with binary data and it worked with test (publish offline) and test 6 (publish offline, reset) and the queue was able to be re-read after reset.

The queue file does seem corrupted in your case; it looks like the block trailer is ASCII data instead of the block trailer.

gusgonnet · April 6, 2025, 8:03pm

ok, I'm sort of able to reproduce.
Seems like it happens when the connection to the cloud is messed up with (example: a location is in progress, and this impacts the cloud connection as reported here).

here are the logs:

0000061349 [system] INFO: Cloud connected
0000061871 [app.pubq] INFO: queue files invalid magic 0xee341b1a /usr/pubqueue2/00000114.pq
0000061871 [app.pubq] INFO: invalid sizes dataSize=443900173 metaSize=13339 /usr/pubqueue2/00000114.pq
0000061872 [app.pubq] INFO: discarding corrupted file 114
0000062357 [app] INFO: Publishing events...
0000063857 [app] INFO: Publishing events...
0000065394 [app] INFO: Publishing events...
0000065922 [system] INFO: Cloud: disconnecting
0000065923 [system] INFO: Cloud: disconnected
0000066894 [app] INFO: Publishing events...
0000068394 [app] INFO: Publishing events...
0000069894 [app] INFO: Publishing events...
0000070922 [system] INFO: Cloud: connecting
0000070927 [system] INFO: Cloud socket connected
0000070927 [comm.protocol.handshake] INFO: Establish secure connection
0000070933 [comm.dtls] INFO: session has 0 uses
0000070949 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000070949 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,2,190, next_coap_id=299
0000070951 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=665
0000070951 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000071395 [comm.protocol.handshake] INFO: Skipping HELLO message
0000071396 [app] INFO: Publishing events...
0000071401 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000071401 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000071731 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000071833 [system] INFO: Cloud connected
0000072903 [app] INFO: Publishing events...
0000074403 [app] INFO: Publishing events...
0000075903 [app] INFO: Publishing events...
0000075904 [system] INFO: Cloud: disconnecting
0000075904 [system] INFO: Cloud: disconnected
0000076416 [wiring] ERROR: Failed to send event: -1001
0000077403 [app] INFO: Publishing events...
0000078903 [app] INFO: Publishing events...
0000080403 [app] INFO: Publishing events...
0000080954 [system] INFO: Cloud: connecting
0000080959 [system] INFO: Cloud socket connected
0000080959 [comm.protocol.handshake] INFO: Establish secure connection
0000080965 [comm.dtls] INFO: session has 0 uses
0000080981 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000080981 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,2,200, next_coap_id=2a2
0000080983 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=674
0000080983 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000081421 [comm.protocol.handshake] INFO: Skipping HELLO message
0000081427 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000081428 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000081792 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000081894 [system] INFO: Cloud connected
0000081903 [app] INFO: Publishing events...
0000082441 [app.pubq] INFO: queue files invalid magic 0xee341b1a /usr/pubqueue2/00000126.pq
0000082442 [app.pubq] INFO: invalid sizes dataSize=443900173 metaSize=13339 /usr/pubqueue2/00000126.pq
0000082443 [app.pubq] INFO: discarding corrupted file 126
0000083403 [app] INFO: Publishing events...
0000084903 [app] INFO: Publishing events...
0000085980 [system] INFO: Cloud: disconnecting
0000085982 [system] INFO: Cloud: disconnected
0000085983 [wiring] ERROR: Failed to send event: -1001
0000086403 [app] INFO: Publishing events...
0000087903 [app] INFO: Publishing events...
0000089403 [app] INFO: Publishing events...
0000090903 [app] INFO: Publishing events...
0000091481 [system] INFO: Cloud: connecting
0000091486 [system] INFO: Cloud socket connected
0000091486 [comm.protocol.handshake] INFO: Establish secure connection
0000091492 [comm.dtls] INFO: session has 0 uses
0000091508 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000091508 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,2,209, next_coap_id=2aa
0000091510 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=682
0000091510 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000091947 [comm.protocol.handshake] INFO: Skipping HELLO message
0000091953 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000091955 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000092287 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000092389 [system] INFO: Cloud connected
0000092403 [app] INFO: Publishing events...
0000092973 [app.pubq] INFO: queue files invalid magic 0x32313030 /usr/pubqueue2/00000131.pq
0000092974 [app.pubq] INFO: invalid sizes dataSize=909456435 metaSize=14391 /usr/pubqueue2/00000131.pq
0000092974 [app.pubq] INFO: discarding corrupted file 131
0000093979 [app] INFO: Publishing events...
0000095479 [app] INFO: Publishing events...
0000096510 [system] INFO: Cloud: disconnecting
0000096511 [system] INFO: Cloud: disconnected
0000096512 [wiring] ERROR: Failed to send event: -1001
0000096979 [app] INFO: Publishing events...
0000098479 [app] INFO: Publishing events...
0000099979 [app] INFO: Publishing events...
0000101479 [app] INFO: Publishing events...
0000101512 [system] INFO: Cloud: connecting
0000101574 [system] INFO: Cloud socket connected
0000101576 [comm.protocol.handshake] INFO: Establish secure connection
0000101582 [comm.dtls] INFO: session has 0 uses
0000101596 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000101598 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,2,219, next_coap_id=2b3
0000101599 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=691
0000101600 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000102112 [comm.protocol.handshake] INFO: Skipping HELLO message
0000102118 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000102118 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000102427 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000102530 [system] INFO: Cloud connected
0000102979 [app] INFO: Publishing events...
0000103567 [app.pubq] INFO: queue files invalid magic 0x32313030 /usr/pubqueue2/00000137.pq
0000103567 [app.pubq] INFO: invalid sizes dataSize=909456435 metaSize=14391 /usr/pubqueue2/00000137.pq
0000103568 [app.pubq] INFO: discarding corrupted file 137
0000104479 [app] INFO: Publishing events...
0000105979 [app] INFO: Publishing events...
0000106637 [system] INFO: Cloud: disconnecting
0000106638 [system] INFO: Cloud: disconnected
0000106732 [wiring] ERROR: Failed to send event: -1001
0000106733 [app.pubq] ERROR: published failed immediately, discarding
0000107479 [app] INFO: Publishing events...
0000108979 [app] INFO: Publishing events...
0000110479 [app] INFO: Publishing events...
0000111631 [system] INFO: Cloud: connecting
0000111636 [system] INFO: Cloud socket connected
0000111636 [comm.protocol.handshake] INFO: Establish secure connection
0000111642 [comm.dtls] INFO: session has 0 uses
0000111658 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000111660 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,2,227, next_coap_id=2ba
0000111660 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=698
0000111662 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000111979 [app] INFO: Publishing events...
0000112141 [comm.protocol.handshake] INFO: Skipping HELLO message
0000112147 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000112149 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000112548 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000112650 [system] INFO: Cloud connected
0000113479 [app] INFO: Publishing events...
0000114979 [app] INFO: Publishing events...
0000116479 [app] INFO: Publishing events...
0000117098 [system] INFO: Cloud: disconnecting
0000117099 [system] INFO: Cloud: disconnected
0000117979 [app] INFO: Publishing events...
0000119479 [app] INFO: Publishing events...
0000120979 [app] INFO: Publishing events...
0000122100 [system] INFO: Cloud: connecting
0000122105 [system] INFO: Cloud socket connected
0000122105 [comm.protocol.handshake] INFO: Establish secure connection
0000122111 [comm.dtls] INFO: session has 0 uses
0000122127 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000122129 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,2,236, next_coap_id=2c2
0000122129 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=706
0000122131 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000122479 [app] INFO: Publishing events...
0000122763 [comm.protocol.handshake] INFO: Skipping HELLO message
0000122769 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000122770 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000123547 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000123649 [system] INFO: Cloud connected
0000123979 [app] INFO: Publishing events...
0000125479 [app] INFO: Publishing events...
0000127042 [app] INFO: Publishing events...
0000127728 [system] INFO: Cloud: disconnecting
0000127729 [system] INFO: Cloud: disconnected

and here's the firmware I'm using:


#include "Particle.h"
#include "PublishQueueExtRK.h"

SYSTEM_MODE(AUTOMATIC);
SerialLogHandler logHandler(LOG_LEVEL_INFO);

char configData[2049];

void fx_that_publishes_text_off_loop()
{
  const char *eventName = "test_event";
  const char data[1000] = "test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890";
  PublishQueueExt::instance().publish(eventName, data);
}

void another_fx_that_publishes_a_variant_off_loop()
{
  memset(configData, 0, sizeof(configData));
  JSONBufferWriter writer(configData, sizeof(configData) - 1);

  writer.beginObject();

  writer.name("is").value(12345678);
  writer.name("ls").value(987654);
  writer.name("ll").value(123131);
  writer.name("lp").value(7897897);
  writer.name("f").value(456453645);
  writer.name("g").value(456453645);
  writer.name("h").value(456453645);
  writer.name("i").value(456453645);
  writer.name("j").value(456453645);
  writer.name("k").value(456453645);
  writer.name("lk").value(456453645);
  writer.name("m").value(456453645);
  writer.name("n").value(456453645);
  writer.name("o").value(456453645);
  writer.name("p").value(456453645);
  writer.name("q").value(456453645);
  writer.name("r").value(456453645);
  writer.name("s").value(456453645);
  writer.name("t").value(456453645);
  writer.name("u").value(456453645);

  writer.endObject();

  particle::Variant v = particle::Variant::fromJSON(configData);

  if (v.isNull())
  {
    Log.error("Failed to convert JSON to Variant: %s", configData);
    return;
  }

  PublishQueueExt::instance().publish("variant event", v, ContentType::BINARY);
}

void setup()
{
  waitFor(Serial.isConnected, 5000);
  delay(1000);
  PublishQueueExt::instance().setup();
  PublishQueueExt::instance().withFileQueueSize(20);
}

void loop()
{

  static uint32_t lastLogTime = 0;
  if (millis() - lastLogTime >= 1500)
  {
    lastLogTime = millis();
    Log.info("Publishing events...");
    fx_that_publishes_text_off_loop();
    another_fx_that_publishes_a_variant_off_loop();
  }

  static uint32_t lastLogTime2 = 0;
  if (millis() - lastLogTime2 >= 5000)
  {
    lastLogTime2 = millis();
    if (Particle.connected())
    {
      Particle.disconnect();
    }
    else
    {
      Particle.connect();
    }
  }

  PublishQueueExt::instance().loop();
}

Let me know if you are able to observe this issue on your side.
Thanks!

rickkas7 · April 7, 2025, 7:51am

Thanks for the code.

Do you actually have code like the second example? If you are generating from JSON data, you should use the publish overload that takes a Variant and does not have a content type, because it's inferred as STRUCTURED.

I think using the combination of a VariantMap (structured data, what you get from converting from JSON) with the BINARY content type is not a valid combination. I suspect it's not serialized properly on device when saving to the file system. Even without the publish queue, it would not be handled properly in the cloud because the data would be converted to Base 64 instead of decoded from CBOR to JSON.

gusgonnet · April 7, 2025, 1:18pm

Yes I do.
So to confirm, instead of doing this:

PublishQueueExt::instance().publish("variant event", v, ContentType::BINARY);

I should do this:

PublishQueueExt::instance().publish("variant event", v);

Correct?
Thanks again!

gusgonnet · April 7, 2025, 1:54pm

@rickkas7 Hi,
even without the BINARY type provided, after a couple of minutes I observe errors again:

0000106195 [app] INFO: Publishing events...
0000107644 [system] INFO: Cloud: connecting
0000107648 [system] INFO: Cloud socket connected
0000107649 [comm.protocol.handshake] INFO: Establish secure connection
0000107655 [comm.dtls] INFO: session has 0 uses
0000107671 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=0
0000107672 [comm.dtls] INFO: out_ctr 0,1,0,0,0,0,1,16, next_coap_id=f4
0000107674 [comm.dtls] INFO: restored session from persisted session data. next_msg_id=244
0000107674 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 2
0000107695 [app] INFO: Publishing events...
0000108126 [comm.protocol.handshake] INFO: Skipping HELLO message
0000108133 [comm.protocol] INFO: Checksum has not changed; not sending application DESCRIBE
0000108134 [comm.protocol] INFO: Checksum has not changed; not sending subscriptions
0000108448 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000108550 [system] INFO: Cloud connected
0000109078 [app.pubq] INFO: queue files invalid magic 0x32313030 /usr/pubqueue2/00000851.pq
0000109079 [app.pubq] INFO: invalid sizes dataSize=909456435 metaSize=14391 /usr/pubqueue2/00000851.pq
0000109080 [app.pubq] INFO: discarding corrupted file 851
0000109229 [app] INFO: Publishing events...
0000110729 [app] INFO: Publishing events...
0000112229 [app] INFO: Publishing events...
0000112840 [system] INFO: Cloud: disconnecting
0000112840 [system] INFO: Cloud: disconnected
0000112896 [wiring] ERROR: Failed to send event: -1001
0000112897 [app.pubq] ERROR: published failed immediately, discarding

Firmware:


#include "Particle.h"
#include "PublishQueueExtRK.h"

SYSTEM_MODE(AUTOMATIC);
SerialLogHandler logHandler(LOG_LEVEL_INFO);

char configData[2049];

void fx_that_publishes_text_off_loop()
{
  const char *eventName = "test_event";
  const char data[1000] = "test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890test data 0123456789001234567890012345678900123456789001234567890012345678900123456789001234567890";
  PublishQueueExt::instance().publish(eventName, data);
}

void another_fx_that_publishes_a_variant_off_loop()
{
  memset(configData, 0, sizeof(configData));
  JSONBufferWriter writer(configData, sizeof(configData) - 1);

  writer.beginObject();

  writer.name("is").value(12345678);
  writer.name("ls").value(987654);
  writer.name("ll").value(123131);
  writer.name("lp").value(7897897);
  writer.name("f").value(456453645);
  writer.name("g").value(456453645);
  writer.name("h").value(456453645);
  writer.name("i").value(456453645);
  writer.name("j").value(456453645);
  writer.name("k").value(456453645);
  writer.name("lk").value(456453645);
  writer.name("m").value(456453645);
  writer.name("n").value(456453645);
  writer.name("o").value(456453645);
  writer.name("p").value(456453645);
  writer.name("q").value(456453645);
  writer.name("r").value(456453645);
  writer.name("s").value(456453645);
  writer.name("t").value(456453645);
  writer.name("u").value(456453645);

  writer.endObject();

  particle::Variant v = particle::Variant::fromJSON(configData);

  if (v.isNull())
  {
    Log.error("Failed to convert JSON to Variant: %s", configData);
    return;
  }

  PublishQueueExt::instance().publish("variant event", v);
}

void setup()
{
  waitFor(Serial.isConnected, 5000);
  delay(1000);
  PublishQueueExt::instance().setup();
  PublishQueueExt::instance().withFileQueueSize(20);
}

void loop()
{

  static uint32_t lastLogTime = 0;
  if (millis() - lastLogTime >= 1500)
  {
    lastLogTime = millis();
    Log.info("Publishing events...");
    fx_that_publishes_text_off_loop();
    another_fx_that_publishes_a_variant_off_loop();
  }

  static uint32_t lastLogTime2 = 0;
  if (millis() - lastLogTime2 >= 5000)
  {
    lastLogTime2 = millis();
    if (Particle.connected())
    {
      Particle.disconnect();
    }
    else
    {
      Particle.connect();
    }
  }

  PublishQueueExt::instance().loop();
}

Am I doing something else wrong?
Thanks

rickkas7 · April 7, 2025, 3:01pm

0.0.3 (2025-04-07)

Fixed a bug where if a queued file failed to publish (such as because of a connectivity issue), the file in the queue would be corrupted and would be discarded instead of retransmitted later.

gusgonnet · April 7, 2025, 4:13pm

Thanks for the new version.

I was still able to get one corrupted file, although the occurrence is way less often than before the fix (took ~15 minutes):

0000873928 [wiring] ERROR: open() failed: 28
0000873929 [app.pubq] ERROR: error saving event to fileNum 4136
0000873938 [comm.protocol] INFO: Sending subscriptions
0000874201 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000874640 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000874642 [comm.protocol] INFO: Received TIME response: 1744042106
0000874646 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000874647 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000874649 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000874651 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000874752 [system] INFO: Cloud connected
0000875130 [app] INFO: Publishing events...
0000875241 [wiring] ERROR: open() failed: 28
0000875241 [app.pubq] ERROR: error saving event to fileNum 4137
0000875358 [wiring] ERROR: open() failed: 28
0000875358 [app.pubq] ERROR: error saving event to fileNum 4138
0000875374 [app.pubq] INFO: queue files invalid magic 0x32313030 /usr/pubqueue2/00003189.pq
0000875375 [app.pubq] INFO: invalid sizes dataSize=909456435 metaSize=14391 /usr/pubqueue2/00003189.pq
0000875376 [app.pubq] INFO: discarding corrupted file 3189
0000875427 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x04
0000877092 [app] INFO: Publishing events...
0000877615 [wiring] ERROR: open() failed: 28
0000877615 [app.pubq] ERROR: error saving event to fileNum 4139
0000877735 [wiring] ERROR: open() failed: 28
0000877736 [app.pubq] ERROR: error saving event to fileNum 4140
0000877835 [system] INFO: Cloud: disconnecting
0000877836 [system] INFO: Cloud: disconnected
0000878015 [wiring] ERROR: Failed to send event: -1001

Is there anything else I should be doing different?
Thanks

gusgonnet · April 7, 2025, 5:27pm

I also noticed that if I switch the device off for, say, 5 minutes, when I switch it on I can see these logs.
Unsure why there would be "discarded event"s or why there would be files with zero bytes: "queue files size 0 is too small".
Thanks

0000178718 [app.pubq] INFO: discarded event 6800
0000178746 [app.pubq] INFO: discarded event 6801
0000178775 [app.pubq] INFO: discarded event 6802
0000178802 [app.pubq] INFO: discarded event 6803
0000178840 [app.pubq] INFO: discarded event 6839
0000178874 [app.pubq] INFO: discarded event 6840
0000178909 [app.pubq] INFO: discarded event 6841
0000178942 [app.pubq] INFO: discarded event 6842
0000178976 [app.pubq] INFO: discarded event 6843
0000179008 [app.pubq] INFO: discarded event 6844
0000179009 [system] INFO: Cloud socket connected
0000179009 [comm.protocol.handshake] INFO: Establish secure connection
0000179009 [app] INFO: Publishing events...
0000179014 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=2
0000180509 [app] INFO: Publishing events...
0000180693 [comm.protocol.handshake] INFO: Sending HELLO message
0000181025 [comm.protocol.handshake] INFO: Handshake completed
0000181026 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000181028 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000181491 [comm.protocol] INFO: Sending subscriptions
0000182009 [app] INFO: Publishing events...
0000182062 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000182507 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000182509 [comm.protocol] INFO: Received TIME response: 1744046556
0000182514 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000182515 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000182517 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000182518 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000182619 [system] INFO: Cloud connected
0000183130 [app.pubq] INFO: queue files size 0 is too small /usr/pubqueue2/00006845.pq
0000183130 [app.pubq] INFO: discarding corrupted file 6845
0000183174 [app.pubq] INFO: queue files size 0 is too small /usr/pubqueue2/00006846.pq
0000183175 [app.pubq] INFO: discarding corrupted file 6846
0000183212 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x04
0000183230 [app.pubq] INFO: queue files size 0 is too small /usr/pubqueue2/00006847.pq
0000183234 [app.pubq] INFO: discarding corrupted file 6847
0000183281 [app.pubq] INFO: queue files size 0 is too small /usr/pubqueue2/00006848.pq
0000183281 [app.pubq] INFO: discarding corrupted file 6848
0000183326 [app.pubq] INFO: queue files size 0 is too small /usr/pubqueue2/00006849.pq
0000183326 [app.pubq] INFO: discarding corrupted file 6849
0000183368 [app.pubq] INFO: queue files size 0 is too small /usr/pubqueue2/00006850.pq
0000183369 [app.pubq] INFO: discarding corrupted file 6850

rickkas7 · April 7, 2025, 5:39pm

You're queuing events far faster than you are dequeueing them. There could be a bug in queue cleanup that is causing files to not be deleted properly which is causing the file system to fill up, which is why the files are empty.

gusgonnet · April 7, 2025, 6:29pm

good point, thanks

rickkas7 · April 8, 2025, 9:29am

0.0.4 (2025-04-08)

Fixed a bug where queue cleanup did not work properly and could also leave corrupted files in the queue if you exceeded the file queue size.

gusgonnet · April 8, 2025, 1:58pm

No errors in one hour! I'd say you killed it here.
Thanks a bunch for all the fixes.

system · May 8, 2025, 11:59pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
PublishQueueExt (CloudEvent API) corrupt/not sending events, but Particle.publish does Libraries boron	11	47	May 27, 2025
PublishQueue for typed and extended publish! (PublishQueueExtRK) Libraries	6	72	May 23, 2025
PublishQueue Library bug Firmware	4	608	June 7, 2018
PublishQueuePosixRK Library Issues Edge Systems tracker , tracker-som	0	424	August 17, 2022
PublishQueueAsyncRK and Large Datasets Libraries	52	3272	March 15, 2022

PublishQueueExtRK corrupted file

0.0.3 (2025-04-07)

0.0.4 (2025-04-08)

Related topics