Exactly! This is what I meant with my reasoning above and with this in mind the results do make sense for me.
@UST, just for testing you could come up with your own handshaking via the use of some digital pins.
@ScruffR, problem is that the flow control needs to be done at the driver level since the buffer pointers are not public vars. One way to do thing is to have the sender (master) send a max of 64 bytes to the receiver (slave) and wait for the slave to process those ( until Serial.available() returns zero) and return an “ack” byte. This “chunked” approach guarantees a non-overflow condition but slows down large message communications. With this approach, you could create a larger “uber”-buffer for the outgoing data since chunck size does not exceed the send buffer. You could service the serial link with a small FSM (finite state machine) in loop(). 
I implemented this via:
void receive()
{
string = "";
count = 0;
while (Serial1.available())
{
count++;
char stuff = Serial1.read();
string += String(stuff);
if(count%20 == 0) delay(20);
}
Serial.println(string);
Serial.print("count = "); Serial.println(count);
}
Using the counting, which is 63 bytes, to delay in between chunks . . . but still I'm receiving the same results.
That makes sense to me as well, IF I wasn't reading from the buffer and letting the data drop. But I'm constantly reading from the Rx Buffer, so if anything there should be inconsistent cuts of data right? The fact that it always cuts off at 63 bytes is what I'm not understanding. Having the Rx Buffer read consistently should be making room for the incoming data.
And if you mean make a handshake as in . . . "Hey I received 20 bytes, set D4 to HIGH Sender says okay I'll send another 20! set D4 LOW " Then I will try to implement this, although I see the same issue occurring IF the buffer isn't having bytes removed.
Since you have set both baud rates to the same the actual action of reading and writing should be near exactly the same - triggered by an hardware interrupt.
But when you do Serial1.read() you don't actually read the bit stream that is sent out by the transmitter at this very moment. The TX/RX action has - in most cases - already taken place and the extra bytes are already lost.
Only when the async TX/RX action is happening while your code does the Serial1.read()ing your code makes space while new bytes come in. But at a high baud rate the RX interrupts might not leave you a lot of time to do your buffer emptying.
No you are not. The Core does do a lot of stuff in the background.
2 Likes
@peekay123, I didn't mean propper flow control, but semi syncing the Serial1.write() action to the actual time where the receiven Core is just about to do the Serial.read()ing hence increasing the likelyness that the async TX/RX will fill the buffer and the user program keeps pulling at least some bytes from the buffer and hence proving my theory right or wrong 
As mentioned above to Paul, I'd rather have the transmitting Core wait in a while() for the receiver to indicate that it is just about to enter the read-loop. Once the receiver has reached this position in code, the transmitter can hammer away and the receiver will pull bytes - always interrupted by the RX interrupt, which should prevent the Serial1.read() to overtake the transmitter (too much)
1 Like
That makes a ton of sense. @peekay123 I just saw that I delayed the receive and not the counter, I will switch this and let you know. @ScruffR if that fails I'll implement your handshake method or some type of delay between the two, because now things make lil more sense. If things are happening at the same baud rate, then the new bytes might just be dropped before .read() even gets a chance to perform ...
@UST, there is a “zero” pin cost for the ack method I outlined above and it essentially does the same thing. 
Exacty!
But the "handshake" should only be a prove of concept and is not a production practice
Now I understand, I can do this with a digital pin or even Serial1 itself ( @peekay123’s “ack” method I believe :P)
I’ll try that out in a few !
@ScruffR, since serial is an asynchronous protocol and both cores have no method for providing synchronicity, the handshake MUST become a production practice for reliable communications. Assuming the “push” and “pull” rates at the application level is a sure-fire method of getting dropped data. If reliability on every message is critical, then handshaking is a must. If not, then the receiver can simply assess if the received data is complete (assuming a fixed message structure) and toss the entire message if not and wait for the next. 
1 Like
@UST, if you don’t want to block loop() waiting for an ACK from the slave, then you do a simple two state machine:
case SENDING:
if (data to send) {
send a chunk of data;
state = WAIT_ACK;
}
break;
case WAIT_ACK;
if (data received && data == ACK)
state = SENDING;
break; //continue with loop()
Hopefully, you get the idea. There could be a timeout in WAIT_ACK so you could send another data chuck and/or keep a timeout error counter. Also, if the data received is not an ACK, that’s an error.
I sort of understand @peekay123 . . . I would implement this using “switch” right and have it in the main loop?
And would both states be on the Send Core, or SENDING state on the send Core and WAIT_ACK State on the receive Core?
@ScruffR I tried implementing your handshake method using this:
void send()
{
while(X != 'Y')
{
X = Serial1.read();
if (X == 'Y')
{
Serial1.write("Today's date is 2015/02/05 10:10:00");
Serial1.write(',');
Serial1.write("1");
Serial1.write(',');
Serial1.write("2");
Serial1.write(',');
Serial1.write("3");
Serial1.write(',');
Serial1.write("data received A.O.K.");
Serial1.write("Do we get to here?");
}
X = 0;
}
}
AND
void receive()
{
string = "";
Serial1.write('Y');
while(Serial1.available())
{
char stuff = Serial1.read();
string += String(stuff);
}
Serial.println(string);
}
But to no avail … it didn’t help, actually just gave me more confusing results … Unless I completely did it wrong.
@ScruffR @bko @peekay123 – Could this whole issue of synchronicity be fixed if I just lower the baud rate of Serial1 on the Sender Core?
I agree that handshake should be incorporated and the STM32 does even support it in hardware and I’ve raised this as a feature request ages ago, but it was never take up.
If you do need to transmit/receive data considerably longer than 63 byte I’d go with @peekay123 's suggestions too.
But as I understand your original post you normally only have 62 byte, you should be safe by just limiting the transmitters update frequency.
If you really need to go that close to the limit you’d also have alternative routes - e.g. building the firmware locally with stretching the SERIAL_BUFFER_SIZE.
As for my other suggestion, I’d rather have thought of something like
Transmitter:
...
while(digitalRead(D7)); // make sure to keep this < 4000ms by the receiver
Serial1.write(yourString);
...
Receiver:
...
char recString[256];
int i = 0;
...
digitalWrite(D7, LOW);
while(!Serial1.available()); // should be well bellow < 1ms
while(Serial1.available())
{
recString[i++] = Serial1.read();
}
recString[i] = '\0';
i = 0;
digitalWrite(D7, HIGH);
...
As said this is not meant for production use but only to see the effect of async/sync data transfer in conjunction with the 64 byte RX buffer limit.
If you want to work without any flow control (hardware pin, send an ACK back, etc.) then you have to manage the rates such that the send never can send faster than the receiver can process that data. You might think that the sender could send at the exact rate the receiver processes data but that never works in the real-world since there is always overhead.
Here the receiver is doing quite a bit of processing including the concatenation of the incoming data and then sending the data back out (at the same rate currently), so the receiver takes at least twice as long as the sender.
So we have two problems: (1) the average rate of the sender is higher than the receiver can handle and (2) if you just put a lump delay after transmitting, the burst rate is also higher than the receiver can handle.
At 115200 baud you are sending one character about every 87us so 32 chars takes under 3ms. Maybe you could try something like this (I picked the points of delay somewhat arbitrarily based on your code):
Serial1.write("Today's date is 2015/02/05 10:10:00");
delay(3);
Serial1.write(',');
Serial1.write("1");
Serial1.write(',');
Serial1.write("2");
Serial1.write(',');
Serial1.write("3");
Serial1.write(',');
Serial1.write("data received A.O.K.");
delay(3);
Serial1.write("Do we get to here?");
delay(3);
I hope this works better than the lumped delay you tried above. If 3 isn’t quite enough, try 4, 5 or 6.
If you want to learn about the science behind queuing, you should read about Little’s Law.
2 Likes
I will take a look into everything you mentioned, thank you @bko
Also this implementation is without using ACK or the Handshake method correct @bko?
1 Like
@UST, the receive FSM is slight different:
case RECEIVING:
if (data to received) {
read the data;
if (!Serial.available()) // this was the last char
if (message is "correct")
state = SEND_ACK;
else
bad message so send a NAK or ignore message and send ACK
}
break;
case SEND_ACK;
send the ACK char
state = RECEIVING;
break; //continue with loop()
Again, I hope you get the idea
That's right, there is no flow control.
If you want or need flow control, then follow @peekay123 's advice!
2 Likes
No success with the delays, thank you though @bko !
I’m going to implement @peekay123’s FSM (2 state) idea and see if that can get me more than 63bytes through the Rxbuffer
1 Like