Spark Core just randomly crashed into SOS - requesting some help

I am prototyping and working on functionality using my Spark Core. Right now, I am working on using an I2C 16x2 RGB LCD and publishing events to the Cloud API.

I have had my Core running for over 24 hours and I haven’t made any changes. While I was sitting here researching, I noticed my Core decided to flash cyan and then restart itself. It had some trouble connecting to the Cloud at first (maybe I was being a little impatient), so I waited. It flashed SOS with 11 flashes in between. Looking through the documentation and the forums, it lists #11 as invalid case. What does that even mean?

I couldn’t find any information expanding on that particular issue. It makes it even more perplexing because I haven’t had SOS crashes in the past and I haven’t made any firmware changes all day! It was responding correctly, sending events to the Cloud successfully, and exposing functions that are accessible through the CLI.

What exactly causes SOS #11 crashes? Without a stack trace, I’m completely in the dark.

Thanks!

There are two occurances of InvalidCase I could find
in ´cc3000_spi.c`

/**
 * @brief  The handler for Interrupt that is generated on SPI at the end of DMA
 transfer.
 * @param  None
 * @retval None
 */
void SPI_DMA_IntHandler(void)
{
	unsigned long ucTxFinished, ucRxFinished;
	unsigned short data_to_recv = 0;

	ucTxFinished = DMA_GetFlagStatus(CC3000_SPI_TX_DMA_TCFLAG );
	ucRxFinished = DMA_GetFlagStatus(CC3000_SPI_RX_DMA_TCFLAG );
	switch(sSpiInformation.ulSpiState)
	{
	case eSPI_STATE_READ_IRQ:
	  // Both Done
          if (ucTxFinished && ucRxFinished)
          {
                  /* Clear SPI_DMA Interrupt Pending Flags */
                  DMA_ClearFlag(CC3000_SPI_TX_DMA_TCFLAG | CC3000_SPI_RX_DMA_TCFLAG);

                  sSpiInformation.ulSpiState = eSPI_STATE_READ_PROCEED;

                  uint16_t *pnetlen = (uint16_t *) &sSpiInformation.pRxPacket[READ_OFFSET_TO_LENGTH];
                  data_to_recv = ntohs(*pnetlen);
                  if (data_to_recv)
                     {
                       /* We will read ARRAY_SIZE(spi_readCommand) + data_to_recv. is it odd? */

                       if ((data_to_recv +  arraySize(spi_readCommand)) & 1)
                         {
                           /* Odd so make it even */

                           data_to_recv++;
                         }

                       /* Read the whole payload in at the beginning of the buffer
                        * Will it fit?
                        */
                       SPARK_ASSERT(data_to_recv <= arraySize(wlan_rx_buffer));
                       SpiIO(eRead,sSpiInformation.pRxPacket,data_to_recv, FALSE);
                     }
          }
	  break;

	case eSPI_STATE_READ_PROCEED:
          //
          // All the data was read - finalize handling by switching to the task
          // and calling from task Event Handler
          //
          if (ucRxFinished)
          {
                  /* Clear SPI_DMA Interrupt Pending Flags */
                  DMA_ClearFlag(CC3000_SPI_TX_DMA_TCFLAG | CC3000_SPI_RX_DMA_TCFLAG);

                  SpiPauseSpi();
                  SetState(eSPI_STATE_IDLE, eDeAssert);
                  // Call out to the Unsolicited handler
                  // It will handle the event or leave it there for an outstanding opcode
                  // It it handles it the it Will resume the SPI ISR
                  // It it dose not handles it and there are not outstanding Opcodes the it Will resume the SPI ISR
                  sSpiInformation.SPIRxHandler(sSpiInformation.pRxPacket);

          }
          break;

	case eSPI_STATE_FIRST_WRITE:
	case eSPI_STATE_WRITE_PROCEED:
          if (ucTxFinished)
          {
                  /* Loop until SPI busy */
                  while (SPI_I2S_GetFlagStatus(CC3000_SPI, SPI_I2S_FLAG_BSY ) != RESET)
                  {
                  }

                  /* Clear SPI_DMA Interrupt Pending Flags */
                  DMA_ClearFlag(CC3000_SPI_TX_DMA_TCFLAG | CC3000_SPI_RX_DMA_TCFLAG);

                  if ( sSpiInformation.ulSpiState == eSPI_STATE_FIRST_WRITE)
                  {
                      sSpiInformation.ulSpiState = eSPI_STATE_WRITE_PROCEED;
                  }
                  else
                  {
                      SetState(eSPI_STATE_IDLE, eDeAssert);
                  }
          }
          break;

	default:
	  INVALID_CASE(sSpiInformation.ulSpiState);
	  break;

	}
}

and in evnt_handler.c

//*****************************************************************************
//
//!  hci_event_handler
//!
//!  @param  pRetParams     incoming data buffer
//!  @param  from           from information (in case of data received)
//!  @param  fromlen        from information length (in case of data received)
//!
//!  @return         none
//!
//!  @brief          Parse the incoming events packets and issues corresponding
//!                  event handler from global array of handlers pointers
//
//*****************************************************************************


UINT8 * hci_event_handler(void *pRetParams, UINT8 *from, INT32 *fromlen)
{
	UINT8 *pucReceivedData, ucArgsize;
	UINT16 usLength;
	UINT8 *pucReceivedParams;
	UINT16 usReceivedEventOpcode = 0;
	UINT32 retValue32;
	UINT8 * RecvParams;
	UINT8 *RetParams;

	volatile system_tick_t start = GetSystem1MsTick();

	while (1)
	{
		if (tSLInformation.usEventOrDataReceived == 0)
		{
			KICK_WDT();
			volatile system_tick_t now = GetSystem1MsTick();
			volatile long elapsed = now - start;
			if (elapsed < 0) { // Did we wrap
				elapsed = start + now; // yes now
			}

			if (cc3000__event_timeout_ms && (elapsed >= cc3000__event_timeout_ms))
			{
				ERROR("Timeout now %ld start %ld elapsed %ld cc3000__event_timeout_ms %ld",now,start,elapsed,cc3000__event_timeout_ms);
				ERROR("Timeout waiting on tSLInformation.usRxEventOpcode 0x%04x",tSLInformation.usRxEventOpcode);

				// Timeout Return Error for requested Opcode
				// This sucks because callers should have initialized pucReceivedParams
				switch(tSLInformation.usRxEventOpcode)
				{

				default:
					INVALID_CASE(tSLInformation.usRxEventOpcode);
					break;

				case HCI_CMND_SIMPLE_LINK_START:
				case HCI_CMND_READ_BUFFER_SIZE:
					break;

				case HCI_CMND_WLAN_CONFIGURE_PATCH:
				case HCI_NETAPP_DHCP:
				case HCI_NETAPP_PING_SEND:
				case HCI_NETAPP_PING_STOP:
				case HCI_NETAPP_ARP_FLUSH:
				case HCI_NETAPP_SET_DEBUG_LEVEL:
				case HCI_NETAPP_SET_TIMERS:
				case HCI_EVNT_NVMEM_READ:
				case HCI_EVNT_NVMEM_CREATE_ENTRY:
				case HCI_CMND_NVMEM_WRITE_PATCH:
				case HCI_NETAPP_PING_REPORT:
				case HCI_EVNT_MDNS_ADVERTISE:
				case HCI_EVNT_READ_SP_VERSION:
				case HCI_EVNT_SELECT:
					*(UINT8 *)pRetParams = -1;
					break;
  ...
}

I would imagine it would be the latter as I don’t have anything using SPI.

I see that it has ERROR statements and it is handed tSLInformation.usRxEventOpcode but I don’t know what exactly that particular object is or where ERROR statements are rendered so that I can view them.

It seems that maybe these happen during Particle.publish(), maybe? I’m looking at the @brief but I’m quite sure what exactly that particular function is doing. I don’t have any context and I’m not exactly an expert-level programmer.

I can follow the logic but I don’t know what many of those variables and objects in evnt_handler.c actually do.

The conditional IF statement that evaluates if cc3000__event_timeout_ms && (elapsed >= cc3000__event_timeout_ms) are true and then a switch case that defaults the given tSLInformation.usRxEventOpcode (as it doesn’t match any of the other cases) and then it breaks is what seems to be happening. What exactly is the tSLInformation object? I’m assuming that the usRxEventOpcode is one of the HCI event codes that are received. This looks like it could possibly be a sanity check that is failing or it could be an error handler.

I looked further into HCI and I’m guessing the HCI codes could be the handlers for particular methods.

Is there any light that could be shed on this that could lead me to figuring this out?

That’s a new one for me too. I haven’t see this particular SOS code. Hopefully it’s only something transient that won’t happen again.

@jvanier It has happened a couple of times and the SOS codes are the same–9 for SOS and 11 in between. I haven’t made any changes to the firmware.

Do Cores produce a stack trace? I’m not too familiar with some of the more embedded functions but to troubleshoot this further, I would need something like that.

Hi @eavery

I would try re-flashing your CC3000 with “deep update.” Sometimes the TI part just loses its mind in my experience.

With your core in DFU mode, try this from the CLI:

particle flash --usb cc3000

This will scrub all WiFi credentials as well, so you will need re-setup the Core.

2 Likes

I get the same SOS error #11 randomly during startup. I performed the deep update just a few days ago, prior to noticing this issue.

@jpvonoy Are you using a Core or a Photon?

I am using a Core. The error happens often after I power on the Core or flash firmware. I am using SPI as well as particle.publish(). After displaying this error code, the Core seems to return to breathing cyan without any input from me.

@jpvonoy I experience the exact same. There doesn’t seem to be a lot of information regarding “invalid case”.

It crashes and then it restarts and works fine. I don’t know what causes it but I have removed Spark.publish().