TCPClient Connect Issues on Electron


#1

I am working with an FTP site using TCPClient. I have found that I get frequent connection issues, which i believe are on my Electron firmware side. I have other programs which contact this FTP site with no issues.
In one of the forum postings there was a reference to TCPClient connections timing out after 5 to 6 seconds. Is this true and if so what has to be done to keep it alive?
I have put some effort into getting delays at appropriate points to make sure everything happens at a sedate pace, it seems to be more reliable that way, but sometime the client does not seem to send anything, for instance I can open a ftp server OK, get the welcome message and then it never responds to the user and password commands, it is though the TCPClient is just not sending the messages. I am working with a remote server so it is difficult to see what is happening on that side in real time but it is rock solid with other applications.
Any help would be appreciated.I get failures maybe 25% of the time and that is more than annoying.


Electron Project Discussion [Hardware]
#2

I’m seeing issues where the Electron hangs and never goes back to sleep sometimes also. I try to send data to Ubidots and then go back to sleep but something happens some time that keeps the Electron from sleeping and it may have something to do with the same problem. Your seeing but I’m not sure.


#3

I have not been able to resolve this problem, tried all sorts of approaches but I am finding that a good percentage of the time the TCPClient does not seem to send short packets of data such as are required to interface to an FTP site. for instance the user logon is just “USER fred”. I am suspecting that the TCPClient waits for more information to build up a packet before it sends or that it delays for some period of time, probably seconds waiting for the extra information. That is just my gut feeling.
What I need is to be able to look through the code for the TCPClient. I have not tried delving into the firmware before but I may be able to spot something. Can someone tell me which particular file I should be looking at on Github? Thanks


#4

This is probably a good place to start:

If you are able to provide a super simple test application that exhibits this bug, it’d be great if you could post a Github Issue. that’ll get this in front of our firmware team, and the important thing for them is that they need to be able to easily replicate the bug. Here’s the Github Issues interface:

Even better, if you are able to find the source of the issue, please highlight that in a github issue, or feel free to make a pull request if you are able to go so far as fixing the bug.

Thanks!


#5

@zach thanks I shall start prowling


#6

@zach and @olegkhr This issue I am experiencing seems to have many of the characteristics raised in Problem with sending short TCP packets on Electron olegkhr is using FTPino I was originally using ParticleFTPClient https://github.com/jychuah/ParticleFtpClient which I have ended up writing my own FTP client to try and get to the bottom of what is happening.
There is definitely an issue with TCPClient sending short packets of data.
I have not been able to solve it. I have sidetracked to see if I could use the AT commands on the ublox modem but I am suspecting that Cellular.command() is not capable of the streaming data that I need. Others have found a solution ie @mountainecho [SOLVED] - General-purpose read from modem serial and Electron: Direct Access to Cellular Module through USART has tried to access Serial3 and talk direct to the ublox but these all involve hacking the firmware and compiling it locally etc.This I was trying to avoid.
It seems that there is enough evidence to raise a firmware bug at this point?
Would it be possible to allow access to Serial3 in exactly the same way as we have access to Serial1 etc? This would allow direct access to the ublox to remove the limitations imposed by Cellular.connect()
Thanks for any help you can give Particle team.


#7

If I have to start compiling the firmware locally what is the recommended best way to do it. I have looked at Local development and gdb debugging with NetBeans, a step by step guide and have not been able to determine how useful that is.
There must be a preferred or recommended approach to debugging the firmware preferably with the JTAG interface to look at code as it executes. I would appreciate any suggestions and reasons why. It looks to me as though I will have to start working at that level to finish this project I have been working on now.


#8

I’ll set up something to test this today. I’ll report back today or tomorrow with what I find.


#9

@rickkas7 thanks heaps, in the meantime I will keep trying to get on top of something at this end


#10

@pNrie I haven’t experienced what you’re describing with TCP, but I can say with certainty that it happens in my UDP connection. I suspect this is not a particle problem. I think it’s a modem issue, like a timeout value the socket value uses to trigger a send before the socket buffer is full. I would check out the ublox documentation. Likely solvable with a modem command.


#11

The other thing I’ve seen at times is that it takes a few seconds for the modem to go to high power mode when you’re first connecting to your server, and if you’re using cellular.command w/o waiting for an ok at every step your commands will fail.


#12

I did some investigation of using the Electron to upload data by FTP. It can be done and it works quite well, though there are some slightly weird stuff going on, which I’ll explain in detail below.

I wrote a test program that uploads a file by FTP server every minute or so. It logs in (USER and PASS), changes the directory (CWD), requests passive mode (PASV), stores the file on the server (STOR), then quits (QUIT). The source is below.

I also have a separate monitor program that monitors the Electron serial, parsing the debug entries, and also has access to the FTP server directories, where it validates that files were uploaded successfully. It checks the file size and data to make sure it was uploaded correctly. That’s written in Java and is not included.

I ran it on 0.5.3 on a 3G Electron (U260).

Actually, it’s 0.5.3 DEBUG_BUILD=y so I also have logging information from inside the system firmware, in particular the messages about the inner workings inside the modem parser. This requires a gcc-arm local build environment but is very handy for this sort of thing.

For example, when I sent the QUIT command, this is what gets logged. Lots of good info here:

    493.225 AT send       6 "QUIT\r\n"
    493.355 AT read  +   15 "\r\n+USOWR: 1,6\r\n"
    493.365 AT read OK    6 "\r\nOK\r\n"
    493.715 AT read  +   17 "\r\n+UUSORD: 1,14\r\n"
 Socket 1: handle 1 has 14 bytes pending
 socketRecv: _cbUSORD
    493.735 AT send      15 "AT+USORD=1,14\r\n"
    493.745 AT read  +   31 "\r\n+USORD: 1,14,\"221 Goodbye.\r\n\""
    493.755 AT read OK    6 "\r\nOK\r\n"
 0000493765:DEBUG: virtual int TCPClient::available() (145):recv(=14)

Anyway, I ran the test for over an hour and attempted 70 connections, and all successfully uploaded the data. 52 worked perfectly. 18 (25 %) has the weird command connection problem described below, but the file was still uploaded correctly.

I also tested without the debug build, and it behaves the same way.

FTP is horrible

Just thought I’d mention FTP is:

  • Insecure. Passwords are sent clear-text.
  • Chatty. It takes many back-and-forth operations to do anything.
  • Requires multiple TCP connections for simple operations.
  • Has no way to tell whether uploaded or downloaded data is truncated.

Of course other protocols will have overhead as well, but sending my 1024 byte payload requires 2988 to 3081 bytes of cellular data. That’s not a great overhead ratio!

Also, because of the weird way that the state transitions interact with the command connection and data connection, you really have to implement it with at least two finite state machines. I used three. Otherwise, you can get into weird timing-related issues when stuff arrives in an order other than what you expected.

And the FTP RFC has ridiculous things like this:

                            1
      +---+   USER    +---+------------->+---+
      | B |---------->| W | 2       ---->| E |
      +---+           +---+------  |  -->+---+
                       | |       | | |
                     3 | | 4,5   | | |
         --------------   -----  | | |
        |                      | | | |
        |                      | | | |
        |                 ---------  |
        |               1|     | |   |
        V                |     | |   |
      +---+   PASS    +---+ 2  |  ------>+---+
      |   |---------->| W |------------->| S |
      +---+           +---+   ---------->+---+
                       | |   | |     |
                     3 | |4,5| |     |
         --------------   --------   |
        |                    | |  |  |
        |                    | |  |  |
        |                 -----------
        |             1,3|   | |  |
        V                |  2| |  |
      +---+   ACCT    +---+--  |   ----->+---+
      |   |---------->| W | 4,5 -------->| F |
      +---+           +---+------------->+---+

The code

Here’s my sample program.

This is a proof-of-concept to make sure FTP works.

I would not recommend using it as a basis for a production FTP client. Actually, I wouldn’t recommend using FTP at all.

[Updated: Fixed bug in continuation line handling as described below.]

#include "Particle.h"

// Adds debugging output over Serial USB
// ALL_LEVEL, TRACE_LEVEL, DEBUG_LEVEL, INFO_LEVEL, WARN_LEVEL, ERROR_LEVEL, PANIC_LEVEL, NO_LOG_LEVEL
SerialDebugOutput debugOutput(9600, ALL_LEVEL);

SYSTEM_THREAD(ENABLED);

const char *FTP_HOST = "server.example.com";
const int FTP_COMMAND_PORT = 21;
const char *FTP_USER = "username";
const char *FTP_PASSWORD = "secret";
const char *FTP_REMOTE_DIR = "reports";
const size_t RESPONSE_SIZE = 512;
const size_t SEND_DATA_SIZE = 1024;

const unsigned long INITIAL_WAIT_MS = 30000;
const unsigned long REPORT_PERIOD_MS = 60000;
const unsigned long COMMAND_TIMEOUT_MS = 30000;
const unsigned long SEND_TIMEOUT_MS = 60000;
const size_t MAX_TCP_WRITE = 512;

// Forward declarations
void cleanupAndWait();
void prepareSendData();
bool processResponse();
bool processCommandStateMachine(int code);
void processDataStateMachine();

typedef enum {
	INITIAL_WAIT_STATE,
	READY_WAIT_STATE,
	COMMAND_STATE,
	NEXT_REPORT_WAIT_STATE
} State;

typedef enum {
	BANNER_WAIT_STATE,
	USER_RESPONSE_WAIT_STATE,
	PASS_RESPONSE_WAIT_STATE,
	CWD_RESPONSE_WAIT_STATE,
	PASV_RESPONSE_WAIT_STATE,
	STOR_RESPONSE_WAIT_STATE,
	QUIT_RESPONSE_WAIT_STATE
} CommandState;

typedef enum {
	DATA_IDLE_STATE,
	DATA_CONNECT_STATE,
	DATA_SEND_WAIT_READY,
	DATA_SEND_STATE,
	DATA_CLEANUP_STATE,
} DataState;

State state = INITIAL_WAIT_STATE;
CommandState commandState;
DataState dataState;
TCPClient commandClient;
TCPClient dataClient;
unsigned long stateTime = 0;
char responseBuf[RESPONSE_SIZE + 1];
size_t responseOffset = 0;
bool firstLine = true;
IPAddress dataAddr;
int dataPort;
bool sendReady = false;
String sendFilename;
char sendBuf[SEND_DATA_SIZE];
size_t sendBufOffset = 0;
size_t sendSize;
unsigned long sendTime = 0;
unsigned long sendCompleteTime = 0;
bool continuation = false;

void setup() {
	Serial.begin(9600);
}

void loop() {
	switch(state) {
	case INITIAL_WAIT_STATE:
		if (millis() - stateTime >= INITIAL_WAIT_MS) {
			state = READY_WAIT_STATE;
		}
		break;

	case READY_WAIT_STATE:
		if (Particle.connected()) {
			Serial.printlnf("** connecting to %s:%d", FTP_HOST, FTP_COMMAND_PORT);

			if (!commandClient.connect(FTP_HOST, FTP_COMMAND_PORT)) {
				// Failed to connected
				Serial.println("failed to connect");
				state = NEXT_REPORT_WAIT_STATE;
				stateTime = millis();
				break;
			}

			// Successfully connected
			state = COMMAND_STATE;
			stateTime = millis();
			responseOffset = 0;
			commandState = BANNER_WAIT_STATE;
		}
		break;

	case COMMAND_STATE:
		// Handle receiving data from the command connection
		if (commandClient.connected()) {
			size_t bytesLeft = RESPONSE_SIZE - responseOffset;
			if (bytesLeft == 0) {
				Serial.println("server returned too large of a response");
				cleanupAndWait();
				break;
			}
			int bytesRead = commandClient.read((uint8_t *) &responseBuf[responseOffset], bytesLeft);
			if (bytesRead > 0) {
				// Additional data has arrived
				responseOffset += bytesRead;

				// Null-terminate buffer for strstr, note that buffer is one byte longer than
				// RESPONSE_SIZE to account for this.
				responseBuf[responseOffset] = 0;

				while(processResponse()) {
					// Handle all of the lines in the response buffer
				}
			}

			if (millis() - stateTime >= COMMAND_TIMEOUT_MS) {
				Serial.println("** timeout in command mode");
				cleanupAndWait();
			}
		}
		else {
			Serial.println("** server disconnected");
			cleanupAndWait();
		}
		break;

	case NEXT_REPORT_WAIT_STATE:
		if (millis() - stateTime >= REPORT_PERIOD_MS) {
			state = READY_WAIT_STATE;
		}
		break;
	}

	processDataStateMachine();
}

void prepareSendData() {

	size_t offset = 0;

	sendFilename = Time.format("%Y%m%dT%H%M%S") + ".txt";
	Serial.printlnf("** sending file: %s", sendFilename.c_str());

	// For testing purposes, we just create some basic information like the time and date, then
	// fill the buffer with sequential ASCII characters for error detection
	sendSize = SEND_DATA_SIZE;

	offset += snprintf(&sendBuf[offset], sendSize - offset, "created: %s\n", Time.format(TIME_FORMAT_DEFAULT).c_str());
	offset += snprintf(&sendBuf[offset], sendSize - offset, "millis: %lu\n", millis());
	offset += snprintf(&sendBuf[offset], sendSize - offset, "sendSize: %u\n", sendSize);

#if Wiring_Cellular
	CellularData cellularData;
	Cellular.getDataUsage(cellularData);
	offset += snprintf(&sendBuf[offset], sendSize - offset, "cellular usage: cid=%d, tx_session=%d, rx_session=%d\n",
			cellularData.cid, cellularData.tx_session, cellularData.rx_session);
#endif

	// Fill rest of the buffer with sequential ASCII data
	char ch = 32;
	for(; offset < (sendSize - 1); offset++) {
		sendBuf[offset] = ch++;
		if (ch >= 127) {
			ch = 32;
		}
	}

	// Make it end with a newline, makes validating easier
	sendBuf[sendSize - 1] = '\n';

	sendBufOffset = 0;
}

void cleanupAndWait() {
	Serial.println("cleanupAndWait");

	if (dataClient.connected()) {
		Serial.println("** closing data connection");
		dataClient.stop();
		dataState = DATA_IDLE_STATE;
	}

	commandClient.stop();
	state = NEXT_REPORT_WAIT_STATE;
	stateTime = millis();

	Serial.println("** ready for next send");
}

bool processResponse() {
	bool checkAgain = false;

	char *endOfLine = strstr(responseBuf, "\015\012");
	if (endOfLine) {
		size_t lineSizeWithEOL = endOfLine - responseBuf + 2;
		*endOfLine = 0;


		if (responseBuf[3] == ' ') {
			continuation = false;

			int code;
			if (sscanf(responseBuf, "%03d", &code) == 1) {
				Serial.printlnf("code %d", code);

				while(processCommandStateMachine(code)) {
					//
				}
			}
			else {
				Serial.printlnf("invalid line %s", responseBuf);
			}
		}
		else
		if (continuation) {
			Serial.printlnf("continuation %s", responseBuf);
		}
		else
		if (responseBuf[3] == '-') {
			Serial.printlnf("continuation start %s", responseBuf);
			continuation = true;
		}
		else {
			Serial.printlnf("unexpected line %s", responseBuf);
		}

		size_t bytesAfter = responseOffset - lineSizeWithEOL;
		if (bytesAfter > 0) {
			// There are possibly more lines in the buffer
			memmove(responseBuf, &responseBuf[lineSizeWithEOL], bytesAfter);
			responseOffset -= lineSizeWithEOL;
			checkAgain = true;
		}
		else {
			responseOffset = 0;
		}
	}

	return checkAgain;
}


bool processCommandStateMachine(int code) {
	bool checkAgain = false;

	switch(commandState) {
	case BANNER_WAIT_STATE:
		if (code == 220) {
			// Send USER command
			commandClient.printf("USER %s\015\012", FTP_USER);
			commandState = USER_RESPONSE_WAIT_STATE;
		}
		else {
			Serial.printlnf("failed to connect, error %s", responseBuf);
			cleanupAndWait();
		}
		break;

	case USER_RESPONSE_WAIT_STATE:
		if (code == 331) {
			// Send PASS command
			commandClient.printf("PASS %s\015\012", FTP_PASSWORD);
			commandState = PASS_RESPONSE_WAIT_STATE;
		}
		else {
			Serial.printlnf("failed to login, error %s", responseBuf);
			cleanupAndWait();
		}
		break;

	case PASS_RESPONSE_WAIT_STATE:
		if (code == 230) {
			Serial.println("logged in successfully!");
			commandClient.printf("CWD %s\015\012", FTP_REMOTE_DIR);
			commandState = CWD_RESPONSE_WAIT_STATE;
		}
		else {
			Serial.printlnf("failed to login, error ", responseBuf);
			cleanupAndWait();
		}
		break;

	case CWD_RESPONSE_WAIT_STATE:
		if (code == 250) {
			Serial.printlnf("directory set to %s", FTP_REMOTE_DIR);
			commandClient.printf("PASV\015\012");
			commandState = PASV_RESPONSE_WAIT_STATE;
		}
		else {
			Serial.printlnf("failed to change directory, error ", responseBuf);
			cleanupAndWait();
		}
		break;

	case PASV_RESPONSE_WAIT_STATE:
		if (code == 227) {
			int a[6];
			// Skip over result code and look for numbers
			char *cp = &responseBuf[4];
			while(*cp && (*cp < '0' || *cp > '9')) {
				cp++;
			}
			if (sscanf(cp, "%u,%u,%u,%u,%u,%u", &a[0], &a[1], &a[2], &a[3], &a[4], &a[5]) == 6) {

				dataAddr = IPAddress(a[0], a[1], a[2], a[3]);
				dataPort = (a[4] * 256) + a[5];
				// Serial.printlnf("data addr=%s port=%d", dataAddr.toString().c_str(), dataPort);

				prepareSendData();
				// Serial.printlnf("STOR %s", sendFilename.c_str());

				commandClient.printf("STOR %s\015\012", sendFilename.c_str());
				commandState = STOR_RESPONSE_WAIT_STATE;
				sendReady = false;
				dataState = DATA_CONNECT_STATE;
			}
			else {
				Serial.printlnf("failed to parse PASV response ", responseBuf);
				cleanupAndWait();
			}
		}
		else {
			Serial.printlnf("PASV command failure, error ", responseBuf);
			cleanupAndWait();
		}
		break;

	case STOR_RESPONSE_WAIT_STATE:
		if (code == 150) {
			Serial.printlnf("STOR got OK to send data, response %s", responseBuf);
			sendReady = true;
		}
		else {
			if (code == 226) {
				Serial.printlnf("** STOR successful, response %s", responseBuf);
			}
			else {
				Serial.printlnf("** STOR failed, error ", responseBuf);
				dataState = DATA_CLEANUP_STATE;
			}

			commandClient.printf("QUIT\015\012");
			commandState = QUIT_RESPONSE_WAIT_STATE;
		}
		break;

	case QUIT_RESPONSE_WAIT_STATE:
		if (code == 221) {
			Serial.printlnf("** QUIT successful, response %s", responseBuf);
		}
		cleanupAndWait();
		break;
	}

	return checkAgain;
}

void processDataStateMachine() {

	switch(dataState) {
	case DATA_IDLE_STATE:
		break;

	case DATA_CONNECT_STATE:
		Serial.printlnf("making data connection to addr=%s port=%d", dataAddr.toString().c_str(), dataPort);
		if (dataClient.connect(dataAddr, dataPort)) {
			Serial.println("data connection established");
			dataState = DATA_SEND_WAIT_READY;
			sendTime = millis();
		}
		else {
			Serial.println("** data connection failed");
			dataState = DATA_CLEANUP_STATE;
		}
		break;

	case DATA_SEND_WAIT_READY:
		if (sendReady) {
			dataState = DATA_SEND_STATE;
		}
		if (millis() - sendTime >= SEND_TIMEOUT_MS) {
			Serial.println("** data timeout waiting for 150 response");
			dataState = DATA_CLEANUP_STATE;
		}
		break;

	case DATA_SEND_STATE:
		if (dataClient.connected()) {
			if (sendBufOffset < sendSize) {
				size_t toWrite = sendSize - sendBufOffset;
				if (toWrite > MAX_TCP_WRITE) {
					toWrite = MAX_TCP_WRITE;
				}
				int result = dataClient.write((uint8_t *)&sendBuf[sendBufOffset], toWrite);
				if (result == -16) {
					// Buffer full, try again later (Photon only)
				}
				else
				if (result <= 0) {
					Serial.printlnf("error sending %d", result);
					dataState = DATA_CLEANUP_STATE;
				}
				else {
					sendBufOffset += result;
				}
			}
			else {
				// All data successfully sent
				unsigned long elapsed = millis() - sendTime;

				Serial.printlnf("** data sent successfully, %u bytes in %lu ms", sendSize, elapsed);
				dataState = DATA_CLEANUP_STATE;
			}

			// Discard any received data on the client connection to make sure the connection closes gracefully.
			// There shouldn't be any, just being safe.
			while(dataClient.available()) {
				dataClient.read();
			}
		}
		else {
			Serial.println("** data closed unexpectedly");
			dataState = DATA_CLEANUP_STATE;
		}

		if (millis() - sendTime >= SEND_TIMEOUT_MS) {
			Serial.println("** data timeout sending");
			dataState = DATA_CLEANUP_STATE;
		}
		break;

	case DATA_CLEANUP_STATE:
		dataClient.stop();
		dataState = DATA_IDLE_STATE;
		break;
	}
}


The Problem

While I successfully transferred 100 % of the file data, there was this oddity:

I successfully close the data connection using dataConnection.stop() which internally issues a AT+USOCL=2 and it seems to succeed. But then nothing happens.

Failure:

   846.092 AT send      16 "AT+USOWR=2,512\r\n"
   846.102 AT read  >    3 "\r\n@"
   846.152 AT send     512 ")*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLM"
   846.338 AT read  +   17 "\r\n+USOWR: 2,512\r\n"
   846.348 AT read OK    6 "\r\nOK\r\n"
data sent successfully, 1024 bytes in 1212 ms
0000846388:DEBUG: virtual void TCPClient::stop() (192):_sock 2 closesocket
socketClose(2)
   846.388 AT send      12 "AT+USOCL=2\r\n"
   846.658 AT read OK    6 "\r\nOK\r\n"
socketFree(2)
timeout in command mode
cleanupAndWait
0000870448:DEBUG: virtual void TCPClient::stop() (192):_sock 1 closesocket
socketClose(1)
   870.448 AT send      12 "AT+USOCL=1\r\n"
   872.729 AT read OK    6 "\r\nOK\r\n"
socketFree(1)

In a normal, successful transaction, the next thing that happens is that bytes arrive on the command connection +UUSORD: 1,24. This doesn’t happen in the failure scenario.

Success:

   774.411 AT read  +   17 "\r\n+USOWR: 2,512\r\n"
   774.421 AT read OK    6 "\r\nOK\r\n"
data sent successfully, 1024 bytes in 1312 ms
0000774461:DEBUG: virtual void TCPClient::stop() (192):_sock 2 closesocket
socketClose(2)
   774.461 AT send      12 "AT+USOCL=2\r\n"
   774.731 AT read OK    6 "\r\nOK\r\n"
socketFree(2)
   775.471 AT read  +   17 "\r\n+UUSORD: 1,24\r\n"
Socket 1: handle 1 has 24 bytes pending
socketRecv: _cbUSORD
   775.491 AT send      15 "AT+USORD=1,24\r\n"
   775.501 AT read  +   41 "\r\n+USORD: 1,24,\"226 Transfer complete.\r\n\""
   775.511 AT read OK    6 "\r\nOK\r\n"
0000775521:DEBUG: virtual int TCPClient::available() (145):recv(=24)
got line 226 Transfer complete.
got code 226
STOR successful, response 226 Transfer complete.

I don’t have a good way to determining for sure whether the server fails to send the command response or whether the Electron isn’t receiving it, but I’d vote for failure to receive.

  • If I try to send another command like QUIT on the command connection, there is no response.
  • I ran the test with a Photon and the same server, and it always works.
  • The file is correctly stored by the server and there’s a log entry that was uploaded in the FTP server log.

Anyway, there’s probably a bug somewhere, but it wasn’t hard to work around, and like I said, FTP is horrible.

Summary

It appears that you can send little back-and-forth transactions all you want, but it’s best to do it using state machines because you have to be careful about blocking the loop or weird timing issues specific to the FTP protocol.

There is some weirdness relating to closing the FTP data connection causing problems with the command connection, but this can be worked around and the data is still sent successfully.


New library FTPSendFile
More user application memory needed
FTP with electron
Particle Electron FTP upload
#13

@rickkas7 Wow, you really have been busy, thanks for that, I will work through that today and see where I get to.
Yes, my version of the ftp program used a state engine approach, it was the only way I could organise the delays properly. I ended up with something that looked like a RTOS based on state engines and that runs OK.
Thanks so much, I will report in


#14

@mountainecho thanks for those comments, after I see if rickkas’s contribution helps I will push farther into the ublox docs, I have been trying the alternative of using their at commands to do the ftp directly but the Cellular.command() function is not as flexible as I would like to really get that going.


#15

@rickkas7 I have just copied your program and created an ftp.ino file from it, compiled in Particle Dev Atom program and uploaded it to the Electron running 0.5.3, I don’t seem to get files uploaded. I was just wondering if you could run it on your end with the following credentials. Don’t worry about security or creaming files, it is a test logon
const char *FTP_HOST = “ftp.nrie.com.au”;
const int FTP_COMMAND_PORT = 21;
const char *FTP_USER = “robyn@nrie.com.au”;
const char *FTP_PASSWORD = “relish321”;
const char *FTP_REMOTE_DIR = “FTP”;
Thanks


#16

I hope it works out for you. I should thank you for inspiring me to take a look at the Ublox doc because I did see a UDP direct link timeout value that will allow me to stop sending round 1024 packets.


#17

I’m not quite sure what’s wrong. I’m getting a 220 (success) after the USER command, which I didn’t handle. There’s a reason why I said this wasn’t production code!

Socket 1: handle 1 was created
0000035072:DEBUG: virtual int TCPClient::connect(IPAddress, uint16_t, network_interface_t) (80):socket=1
0000035073:DEBUG: virtual int TCPClient::connect(IPAddress, uint16_t, network_interface_t) (98):_sock 1 connect
socketConnect(1,port:21)
    35.083 AT send      31 "AT+USOCO=1,\"173.248.187.5\",21\r\n"
    35.603 AT read OK    6 "\r\nOK\r\n"
0000035603:DEBUG: virtual int TCPClient::connect(IPAddress, uint16_t, network_interface_t) (100):_sock 1 connected=1
    36.334 AT read  +   18 "\r\n+UUSORD: 1,322\r\n"
Socket 1: handle 1 has 322 bytes pending
socketRecv: _cbUSORD
    36.354 AT send      16 "AT+USORD=1,128\r\n"
    36.374 AT read  +  146 "\r\n+USORD: 1,128,\"220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------\r\n220-You are user number 26 of 500 allowed.\r\n220-Local time is now\""
    36.386 AT read OK    6 "\r\nOK\r\n"
0000036386:DEBUG: virtual int TCPClient::available() (145):recv(=128)
got line 220---------- Welcome to Pure-    36.387 AT read  +   18 "\r\n+UUSORD: 1,194\r\n"
Socket 1: handle 1 has 194 bytes pending
FTPd [privsep] [TLS] ----------
    36.397 AT send      15 "AT+USOWR=1,24\r\n"
    36.407 AT read  >    3 "\r\n@"
    36.457 AT send      24 "USER robyn@nrie.com.au\r\n"
    36.597 AT read  +   16 "\r\n+USOWR: 1,24\r\n"
    36.607 AT read OK    6 "\r\nOK\r\n"
got line 220-You are user number 26 of 500 allowed.
failed to login, error 220-You are user number 26 of 500 allowed.
cleanupAndWait
0000036608:DEBUG: virtual void TCPClient::stop() (192):_sock 1 closesocket
socketClose(1)
    36.618 AT send      12 "AT+USOCL=1\r\n"
    36.888 AT read OK    6 "\r\nOK\r\n"
socketFree(1)

I added this to processCommandStateMachine():

	case USER_RESPONSE_WAIT_STATE:
		if (code == 220) {
			Serial.println("logged in successfully!");
			commandClient.printf("CWD %s\015\012", FTP_REMOTE_DIR);
			commandState = CWD_RESPONSE_WAIT_STATE;
		}
		else
		if (code == 331) {
			// Send PASS command
			commandClient.printf("PASS %s\015\012", FTP_PASSWORD);
			commandState = PASS_RESPONSE_WAIT_STATE;
		}

If I skip sending the password, it won’t let me create files.

got line 220----    34.957 AT read  +   18 "\r\n+UUSORD: 1,194\r\n"
Socket 1: handle 1 has 194 bytes pending
------ Welcome to Pure-FTPd [privsep] [TLS] ----------
    34.967 AT send      15 "AT+USOWR=1,24\r\n"
    34.977 AT read  >    3 "\r\n@"
    35.027 AT send      24 "USER robyn@nrie.com.au\r\n"
    35.167 AT read  +   16 "\r\n+USOWR: 1,24\r\n"
    35.177 AT read OK    6 "\r\nOK\r\n"
got line 220-You are user number 29 of 500 allowed.
logged in successfully!
    35.177 AT send      14 "AT+USOWR=1,9\r\n"
    35.188 AT read  >    3 "\r\n@"
    35.238 AT send       9 "CWD FTP\r\n"
    35.368 AT read  +   15 "\r\n+USOWR: 1,9\r\n"
    35.378 AT read OK    6 "\r\nOK\r\n"
socketRecv: _cbUSORD
    35.398 AT send      16 "AT+USORD=1,128\r\n"
    35.418 AT read  +  146 "\r\n+USORD: 1,128,\" 18:31. Server port: 21.\r\n220-This is a private system - No anonymous login\r\n220-IPv6 connections are also welcome on this serve\""
    35.430 AT read OK    6 "\r\nOK\r\n"
0000035430:DEBUG: virtual int TCPClient::available() (145):recv(=128)
got line 220-Loc    35.431 AT read  +   17 "\r\n+UUSORD: 1,66\r\n"
Socket 1: handle 1 has 66 bytes pending
al time is now 18:31. Server port: 21.
failed to change directory, error 
cleanupAndWait

I also tried sending a password on a 220 response, even though a password required should require a 3xx response, and that didn’t work either.

I didn’t mention it above, but my test FTP server is vsftpd running under Ubuntu Linux.

I’m not sure what’s wrong. It’s mysterious!


#18

@rickkas7 yes that is about as far as I get with your program also.
After the USER login i look for a 3** response and then send the PASS, 80% of the time that works and I get a successful logon. It is the 20% that confuses me and why I suspected the short text “PASS relish321” was not being sent out.
When I do get in the responses are unreliable. For instance “TYPE ASCII” will fail more often than not, if it does get past that it seems to negotiate the longer strings like connecting to the data port and sending the file records
If you have any ideas at all I would be very appreciative


#19

I’m pretty sure it’s a server issue. It looks like it’s treating some login attempts as anonymous, no identification required (220 from USER), even though it should be requesting a password (331). Then it rejects any further operations as not allowed from anonymous.


#20

@rickkas7 you could be right, I tested it on a completely different server
const char *FTP_HOST = “ftp.agscan.net”;
const int FTP_COMMAND_PORT = 21;
const char *FTP_USER = “nrieAlternative”;
const char *FTP_PASSWORD = “Altern@t1v3”;
and it seemed to send up a file.
I am not sure what I can find out about the issue with our main server, I don’t even know where it is located, somewhere on the east coast of US by its time zone. But it gives me something to work on, it would be nice if I could get a fix there.
Thanks for all your help, I will be back with whatever I can find