With a friend, I found the source of the error and the panel is now working more or less as expected. there were still some strange ghosting effects (flickering Lines/Dots on the matrix, just for like 0.1 second, but still clearly visible). I set everything up again, but now with a 3.3V to 5V level-shifter between Photon shield and the Panel. With the levels shifted, the Matrix displayed everything without any problem. No ghosting effects or anything!
I used this levelshifter: https://www.adafruit.com/product/395
@peekay123 Now I would like to integrate that levelshifter on your handy shield. Is it possible, that you could provide the KiCAD Project? I would happily post my version on here, when done. I would add the same levelshifter that is also on those breakoutboards that i used (link above).
@RWB I have also read somewhere, that those Adafruit Panels are produced in batches and can be slightly different to eachother. So its possible, that I have a panel in the lower tolerance region. I dont know. What clock are you useing it for?