The File Transfer Protocol (FTP) was one of the first efforts to create a standard means of exchanging files over a TCP/IP network, so the FTP has been around since the 1970's. The FTP was designed with as much flexibility as possible, so it could be used over networks other than TCP/IP, as well as being engineered to have the capability with exchanging files with a broad variety of machines.
The base specification is RFC 959 and is dated October 1985. There are some additional RFCs relating to FTP, but it should be noted that even as of this writing (December 2001) that most of the new additions are not in widespread use. The purpose of this document is to provide general information about how the protocol works without getting into too many technical details. RFC 959 should be consulted for details on the protocol.
The protocol can be thought of as interactive, because clients and servers actually have a conversation where they authenticate themselves and negotiate file transfers. In addition, the protocol specifies that the client and server do not exchange data on the conversation channel. Instead, clients and servers negotiate how to send data files on separate connections, with one connection for each data transfer. Note that a directory listing is considered a file transfer.
To illustrate, we'll just present (an admittedly contrived) example of how the FTP would work between human beings rather than computer systems. For our example, we'll assume we have a client, Carl Clinton, who wishes to transfer files from Acme Mail Service that manages his post office box. Below is a transcript of a phone call between Carl Clinton and Acme Mail Service.
Clinton: (Dials the phone number for the mail service) Service: "Hello, this is the Acme Mail Service. How may I help you today?" Clinton: "Hello, this is Carl Clinton. I would like to access mailbox number MB1234." Service: "OK, Mr. Clinton, I need to verify that you may access mailbox MB1234. What is your password?" Clinton: "My password is QXJ4Z2AF." Service: "Thank you Mr. Clinton, you may proceed." Clinton: "For now, I'm only interested in looking at the bills and invoices, so look at the folder marked "bills" in my mailbox." Service: "OK." Clinton: "Please prepare to have your assistant call my secretary at +1 402 555 1234." Service: "OK." Clinton: "Now call my secretary and tell him the names of all the items in the bills folder of my mailbox. Tell me when you have finished." Server: "My assistant is calling your secretary now." Server: "My assistant has sent the names of the items." Clinton: (Receives the list from his secretary and notices a bill from Yoyodyne Systems.) "Please prepare to have your assistant send to my fax machine +1 402 555 7777."
Service: "OK." Clinton: "Now fax a copy of the bill from Yoyodyne Systems." Server: "My assistant is calling your fax machine now." Server: "My assistant has finished faxing the item." Clinton: "Thank you, that is all. Good bye." Server: "Goodbye."
Now let's look at how this same conversation would appear between computer systems communicating with the FTP protocol over a TCP/IP connection.
Client: Connects to the FTP service at port 21 on the IP address 172.16.62.36. Server: 220 Hello, this is the Acme Mail Service. Client: USER MB1234 Server: 331 Password required to access user account MB1234. Client: PASS QXJ4Z2AF Note that this password is not encrypted. The FTP is susceptible to eavesdropping! Server: 230 Logged in. Client: CWD Bills Change directory to "Bills." Server: 250 "/home/MB1234/Bills" is new working directory. Client: PORT 192,168,1,2,7,138 The client wants the server to send to port number 1930 on IP address 192.168.1.2. In this case, 192.168.1.2 is the IP address of the client machine. Server: 200 PORT command successful. Client: LIST Send the list of files in "Bills." Server: 150 Opening ASCII mode data connection for /bin/ls. The server now connects out from its port 20 on 172.16.62.36 to port 1930 on 192.168.1.2. Server: 226 Listing completed. That succeeded, so the data is now sent over the established data connection. Client: PORT 192,168,1,2,7,139 The client wants the server to send to port number 1931 on the client machine. Server: 200 PORT command successful. Client: RETR Yoyodyne.TXT Download "Yoyodyne.TXT." Server: 150 Opening ASCII mode data connection for Yoyodyne.TXT. The server now connects out from its port 20 on 172.16.62.36 to port 1931 on 192.168.1.2. Server: 226 Transfer completed. That succeeded, so the data is now sent over the established data connection. Client: QUIT Server: 221 Goodbye.
When using FTP, users use FTP client programs rather than directly communicating with the FTP server. Here's our same example using the stock "ftp" program which is usually installed as /usr/bin/ftp on UNIX systems (and FTP.EXE on Windows). The items the user types are in bold.
ksh$ /usr/bin/ftp ftp> open ftp.acmemail.example.com Connected to ftp.acmemail.example.com (172.16.62.36). 220 Hello, this is the Acme Mail Service. Name (ftp.acmemail.example.com:root): MB1234 331 Password required to access user account MB1234. Password: QXJ4Z2AF 230 Logged in. ftp> cd Bills 250 "/home/MB1234/Bills" is new working directory. ftp> ls 200 PORT command successful. 150 Opening ASCII mode data connection for /bin/ls. -rw-r--r-- 1 ftpuser ftpusers 14886 Dec 3 15:22 Acmemail.TXT -rw-r--r-- 1 ftpuser ftpusers 317000 Dec 4 17:40 Yoyodyne.TXT 226 Listing completed. ftp> get Yoyodyne.TXT local: Yoyodyne.TXT remote: Yoyodyne.TXT 200 PORT command successful. 150 Opening ASCII mode data connection for Yoyodyne.TXT. 226 Transfer completed. 317000 bytes received in 0.0262 secs (1.2e+04 Kbytes/sec) ftp> quit 221 Goodbye.
As you can see, FTP is designed to allow users to browse the filesystem much like you would with a regular UNIX login shell or MS-DOS command prompt. This differs from other protocols that are transactional (i.e. HTTP), where a connection is established, clients issue a single message to a server that replies with a single reply, and the connection is closed. On the other hand, client programs can be constructed to simulate a transactional environment if they know in advance what they need to do. In effect, FTP is a stateful sequence of one or more transactions.
The client is always responsible for initiating requests. These requests are issued with FTP command primitives, which are typically 3 or 4 characters each. For example, the command primitive to change the working directory is CWD.
The server replies are specially formatted to contain a 3-digit result code first, followed by a space character, followed by descriptive text (there is also a format for multi-line responses). The protocol specifies that clients must only rely upon the numeric result code, since the descriptive text is allowed to vary (with a few exceptions). In practice, the result text is often helpful for debugging, but is generally no longer useful for end users.
Although it is not required by protocol, in effect clients must always login to the FTP server with a username and password before the server will allow the client to access the service.
There is also a de facto standard for guest access, where "anonymous" (or "ftp") are used as the username and an e-mail address is customarily used as the password in a way for a polite netizen to let the server administrator know who is using the guest login. Because users do not want to divulge their e-mail addresses to protect against unsolicited bulk e-mail, this has subsequently evolved to the point where the password is just some arbitrary text.
The protocol has built-in support for different types of data transfers. The two mandated types are ASCII for text (specified by the client sending "TYPE A" to the server), and "image" for binary data (specified by "TYPE I").
ASCII transfers are useful when the server machine and client machine have different standards for text. For example, MS-DOS and Microsoft Windows use a carriage return and linefeed sequence to denote an end-of-line, but UNIX systems use just a linefeed. When ASCII transfers are specified, this enables a client to always be able to translate the data into its own native text format.
Binary transfers can be used for any type of raw data that requires no translation. Client programs should use binary transfers unless they know that the file in question is text.
The protocol does not have any advanced support for character sets for pathnames nor file contents. There is no way to specify UNICODE, for example. For ASCII, it is 7-bit ASCII only.
Unfortunately, the burden of deciding what transfer type to use is left to the client, unlike HTTP, which can inform the client what type of data is being sent. Clients often simply choose to transfer everything in binary, and perform any necessary translation after the file is downloaded. Additionally, binary transfers are inherently more efficient to send over the network since the client and server do not need to perform on-the-fly translation of the data.
It should be noted that ASCII transfers are mandated by the protocol as the default transfer type unless the client requests otherwise!
Although it was purposely designed into the protocol as a feature, FTP's use of separate data connections cause numerous problems for things like firewalls, routers, proxies which want to restrict or delegate TCP connections, as well as things like IP stacks which want to do dynamic stateful inspection of TCP connections.
The protocol does not mandate a particular port number or a direction that a data connection uses. For example, the easy way out would have been for the protocol's designers to mandate that all data connections must originate from the client machine and terminate at port 20 on the server machine.
Instead, for maximum flexibility, the protocol allows the client to choose one of two methods. The first method, which we'll call "Active", is where the client requests that the server originate a data connection and terminate at an IP address and port number of the client's choosing. The important thing to note here is that the server connects out to the client.
| Client: | "Please connect to me at port 1931 on IP address 192.168.1.2, then send the data." |
| Server: | "OK" |
Or, the client can request that the server to assign an IP address and port number on the server side and have the client originate a connection to the server address. We call this method "Passive" and note that the client connects out to the server.
| Client: | "Please tell me where I can get the data." |
| Server: | "Connect to me at port 4023 on 172.16.62.36." |
The active method uses the FTP command primitive PORT, so the first example using the actual FTP protocol would resemble this:
| Client: | PORT 192,168,1,2,7,139 |
| Server: | 200 PORT command successful. |
The passive method uses the FTP command primitive PASV, so the second example using the actual FTP protocol would resemble this:
| Client: | PASV |
| Server: | Entering Passive Mode (172,16,62,36,133,111) |
It should be noted that FTP servers are required to implement PORT, but are not required to implement PASV. The default has traditionally been PORT for this reason, but in practice it is now preferred to use PASV whenever possible because firewalls may be present on the client side which often cause problems.
The protocol provides a means to only transfer a portion of a file, by having a client specify a starting offset into the file (using the REST primitive, i.e. "restart point"). If an FTP session fails while a data transfer is in progress and has to be reestablished, a client can request that the server restart the transfer at the offset the client specifies. Note that not all FTP servers support this feature.
The base standard of the FTP protocol provides two types of listings, a simple name list (NLST) and a human-readable extended listing (LIST). The name list consists of lines of text, where each line contains exactly one file name and nothing else.
The extended listing is not intended to be machine-readable and the protocol does not mandate any particular format. The de facto standard is for it to be in UNIX "/bin/ls -l" format, but although most servers try to emulate that format even on non-UNIX FTP servers, it is still common for servers to provide their own proprietary format. The important thing to note here is that this listing can contain any type of data and cannot be relied upon. Additionally, even those that appear in "/bin/ls -l" format cannot be relied upon for the validity of the fields. For example the date and time could be in local time or GMT.
Newer FTP server implementations support a machine-readable listing primitive (MLSD) which is suitable for client programs to get reliable metadata information about files, but this feature is still relatively rare. That leaves the simple name list as the only reliable way to get filenames, but it doesn't tell a client program anything else (such as if the item is a file or a directory!).
Despite a rich feature set, there are some glaring omissions. For example, the base specification doesn't even provide for clients to query a file's size or modification date. However, most FTP servers in use now support a de facto extension to the specification which provides the SIZE and MDTM primitives, and even newer servers support the extremely useful MLSD and MSLT primitives which can provide a wealth of information in a standardized format.
There is also no 100% accurate way for a client to determine if a particular pathname refers to a file or directory, unless MLSD or MLST is available. Since the protocol also does not provide a way to transfer an entire directory of items at once, the consequence is that there is no 100% accurate way to download an entire directory tree.
The end result is that FTP is not particularly suited to "mirroring" files and directories, although FTP client programs use heuristics to make calculated guesses when possible.
Despite the guesswork that clients can use for determining metadata for files to download, there's little they can do for files that they upload. There is no standard way to preserve an uploaded file's modification time. FTP is platform agnostic, so there aren't standard ways to preserve platform-specific metadata such as UNIX permissions and user IDs or Mac OS file type and creator codes.
Separate connections for data transfers are also a mixed blessing. For high performance it would be best to use a single connection and perform multiple data transfers before closing it. Even better would be for a method to use a single connection for both the control connection conversation and data transfers. Since each data connection uses an ephemeral (random) port number, it is possible to "run out" of connections. For details on this phenomenon, a separate article is available.
It is important to note that the base specification, as implemented by the vast majority of the world's FTP servers, does not have any special handling for encrypted communication of any kind. When clients login to FTP servers, they are sending clear text usernames and passwords! This means that anyone with a packet sniffer between the client and server could surreptitiously steal passwords.
Besides passwords, potential attackers could not only monitor the entire conversation on the FTP control connection, they could also monitor the contents of the data transfers themselves.
There have been proposals to make the FTP protocol more secure, but these proposals have not seen widespread adoption.
Therefore, unless the IP protocol layer itself is secure (for example, encrypted using IPsec), FTP should not be used if sensitive login information is to be exchanged over an insecure network, or if the files containing sensitive material are being transferred over an insecure network.
© Copyright 2001 by Mike Gleason, NcFTP
Software.
All Rights Reserved.