After experimenting with the Tracker Configuration it seems that my device has saved old configurations on the system. This is indicated by the message in the Particle Console:
The setting is probably still stored in the local settings in the flash file system on the device. To reset the local settings, follow the instructions to send a function call to the device.
Unfortunately that didn't solve the issue. After the reset, I see in the serial monitor that the old configurations are still sent to the cloud where the display message is shown again.
I stumbled across the function Tracker::instance().configService.flush(). I put it right before I initialized my configurations but it didn't help either. But maybe I just applied it in the wrong way?