.;Connected Machines Project paper for Fall 1982 DECUS. DR0:[103,110]PAPER.RNO .; .; Authors: Ben Burch .; Vicky White .; .; A product of the BISON Group, FERMILAB. .; .fill .b .sp 2 .center;An RSX-11M Device Driver Implementing .center;Network Protocols on the DR-11W .sp 1 .b 2 .c;D. Burch, V. White .c;Computing Department .c;Fermi National Accelerator Laboratory .c;Batavia, Il 60510 .sk 5 ^& .c;ABSTRACT \& .b .p At Fermilab, DR-11Ws have been used as high speed data links to interconnect PDP-11s under both RT-11 and RSX11-M. An implementation for VAX-11s (under VMS) is planned. Using this hardware, several processors can be used to provide distributed data collection, data monitoring and control, in real-time, for physics experiments. .sk 1 .p This paper discusses a device driver which we have implemented under RSX-11M V3.2. This driver allows task-to-task logical connections to be established between a pair of connected PDP-11s. This driver implements several unique software mechanisms including non-standard ASTs, driver maintained queues, and task rundown support. It allows a network operation to be processed with less than 6 ms software overhead. .sk 1 .p The reasons for choosing this method of implementation are discussed, and problems solved are described. .sk 1 .p Plans for the future development of utility software will be briefly mentioned. .b.sk 3 .b ^& .hl 1 Introduction \& The Fermi National Accelerator Laboratory (Fermilab) is a national laboratory dedicated to pure research in the field of high energy physics. Research is performed by a large number of experimental groups each using a dedicated minicomputer for data acquisition. Experiments at Fermilab are based on the use of a proton accelerator (proton synchrotron) which provides an intense beam of protons for one second out of every ten second accelerator cycle. During this one second period (known as the accelerator "spill"), the minicomputer for each experiment acquires as much data as possible from the experimental apparatus. The data is accumulated in the form of "events", each containing the information recorded by the experimental apparatus for a single interaction between the elementary particle beam and the atoms in a target. Depending on the experiment, the size of an event can vary from one word of data to several thousand words of data. In many experiments, the amount of data which can be accumulated during each spill is limited by the rate at which the computer can transfer the data from the experimental apparatus. Data acquisition rates can run .PAGE as high as 300,000 words of data per spill, with typical rates being about one quarter of this. .p In addition to accumulating data and storing them, normally on magnetic tape, the online computers must monitor the accumulated data and carry out direct checks of the hardware to verify the proper functioning of the entire experimental apparatus. .p Fermilab is currently embarked upon a program of improvement in order to meet the future needs of high energy physics. This program involves the construction of a new proton synchrotron that will provide protons with energies of 1 TEV (One Trillion Electron Volts). The principal effects of this program upon the data acquisition environment are the increase of the spill time to twenty seconds out of every sixty, and a marked increase in the complexity of the apparatus of some experiments. .p In order to meet the data taking needs of this new era in high energy physics, it became apparent that current data acquisition systems were not adequate. After consideration of many alternatives, it was decided to implement a distributed processing system in which PDP-11s and VAX-11s are connected by a high speed DMA-type communications device. We chose the DR-11W. .p The DR-11W is a device capable of transmitting data buffers from one UNIBUS to another in DMA mode. It does so (in non-burst mode) at a rate of three microseconds per word. The peak (burst mode) rate is about two microseconds per word (8,000,000 baud). It is capable of operating only over short (fifty foot) distances, but this was not a limitation in our environment. The DR-11W is low in cost, readily available, DEC supported, and available from a second source. .hl 1 Link Protocol .p In this section we will consider the protocol steps involved in a normal transfer operation of a single block of data (referred to as a "packet") between one and 32,000 words in length. Exceptional cases such as errors, restarts, collisions, and timeouts will not be discussed here. We will note, briefly, the possible exceptions to this protocol which cannot be considered errors. For a more detailed description of this protocol please see associated paper [1]. .p Our link protocol supports both the transmission of blocks of data and single byte values known as Signals. A signal is transmitted with a shorter protocol exchange, and therefore is suitable for the rapid transfer of simple control data. Data blocks are moved in DMA mode, and protocol exchanges occur in programmed data transfer mode. .p The DR-11W is a half-duplex device, and therefore a bid for ownership of the link must occur before a transfer operation can take place. A link operation starts with one side of the link (the Sender) issuing a Request Link. It's partner (the Receiver) responds with a Link Acknowledge. .p Next, the Sender transmits a Packet Type Code (PTC) to the Receiver. The Receiver responds to the PTC either with a PTC acknowledge or a PTC NACK. The PTC NACK exists to allow a Receiver to reject certain types of link transactions. .p Next, the Sender transmits either a Word Count for the transfer to be performed, or a Signal. A signal can be identified because the sign bit is set and a the upper byte is filled with a particular pattern. The Receiver responds either with a Signal Acknowledge, with A Permitted Word Count, or with a Word Count NACK. The Permitted Word Count is the amount that the receiver actually wishes to receive. By means of this feature, a transmitter can send pieces of a large buffer to a receiver which does not have a buffer large enough to contain the complete message. The Word Count NACK allows a Receiver to reject a transmission which is for some reason not of the correct length. If the receiver sends a permitted word count, it first sets up it's DR-11W for the DMA operation. .p Following the receipt of the Permitted Word Count, the Sender sets up its DR-11W device for a DMA, and begins the DMA operation. .p After completion of the operation, the Receiver determines whether it has received all of the expected words, and also whether it has any messages waiting for the link, and from these determinations forms an End of Message Status Code. The Receiver then transmits the End Of Message Status Code to the Sender. The capability of a Receiver to request the link at the end of a link operation is important in cases where a Sender is much faster than a Receiver, and could turn around and win the link bid process after each transmission. It also reduces the number of protocol steps during times when the link is seldom idle. The End of Message Status Code completes the link operation. See Figure One for a diagram of this process. .hl 1 Program-to-Program Protocol .p Implied by the Link Protocol is the Program-to-Program or Network Protocol. This system does not address the concept of "route-through". It only provides for communication between a pair of connected machines. To allow all members of a network using this system to communicate with one another without providing application specific routing software, one would have to connect each machine to every other machine in the net. .p The Packet Type Code (PTC) identifies a logical link from some procedure to a Receiver. In a multi-tasking system, such as RSX or VMS, this value is associated to a particular task. A "Packet Type Affinity" is then said to exist. By this method, a data block can be sent from a buffer within a task running one machine directly to a buffer in a task running in the receiver machine. .p In order to make the operation of a system using the link as insensitive as possible to the type of operating system running in a connected machine, we chose numbers rather than task names to identify logical links. The PTC destination address mechanism provides a logical link in one direction only, that is, from a sender to a receiver. The receiver has no information about the sender of a message other than which LUN the message was received from, and therefore, which physical machine the sender resides in. .p For this reason, application code protocols must exist if the receiver of a message must know anything about the message's sender. .p Depending on the implementation of this protocol, it is possible for a task using the link to be informed of the nature of a packet before it specifies a buffer for the receipt of the packet. The task then has the option of specifying a sufficiently large buffer, some smaller buffer if wants less than the entire buffer, or rejecting the transaction entirely, which causes the link serving mechanism to issue a Word Count NACK. .p The signal mechanism is intended to provide a low overhead mechanism to transfer control information. In the case of RSX-11, the Receiver does not need to issue an I/O operation to receive a signal. In place of an I/O operation, an AST is forced to the task with the signal on the AST stack. .P Each Task willing to receive a message must declare which PTCs it intends to receive. Note that one and only one task may establish affinity for a particular PTC at a given time, but as soon as that affinity is removed (either explicitly or through the demise of the task), any other task may obtain an affinity for that PTC. .p Note that all data blocks are moved in DMA mode directly from one task's buffer into another task's buffer. No intermediate pools or buffers are involved. .hl 1 The RSX-11M Device Driver Implementation .p The RSX-11M implementation of the above protocols is in the form of a device driver. Tasks using the Communications Driver (CD:), communicate with it via the QIO directive. Currently, this driver has been implemented as either a loadable or resident driver for 18-bit mapped PDP-11s. It is capable of handling communications to any number of DR-11W connected processors, as well as task-to-task communication between tasks running in the machine in which the driver resides. .p This latter feature is interesting because it allows all programs in a system to communicate with each other via a standard interface. This allows functions in the distributed processing system to be independent of their position within the system, and allows debugging of software on a single machine. The internal communications option can be generated into a driver which does not support any physical DR-11W units. It can, therefore, run on a 22-bit mapped PDP-11. .p In order to create a Packet Type Affinity, a special I/O function is executed which specifies the PTC that the task needs to receive, and an AST address which will be used upon receipt of a signal or when a message arrives with no read queued for that PTC. As described above, this allows a task to accept link transactions without always having an outstanding I/O packet. This allows the task to be checkpointable. Two I/O functions are provided to remove the Packet Type Affinities for a task. One removes a specific Packet Type Affinity, while the other removes all Packet Type Affinities for the issuing task. .p I/O functions have been implemented to read packets, write packets, and send signals. The write function specifies a packet address, packet length, and destination PTC as I/O parameters. The unmodified form of the read function is intended to receive a currently pending packet (one for which the task has received an AST). Since there can be one and only one pending packet for each DR-11W unit, the read function specifies only the buffer address and length. Note that until the Receiver issues it's QIO and the transaction completes, no other I/O may occur on that physical unit. Therefore any long series of transactions in this mode will decrease link throughput. An I/O subfunction has been defined that will allow a read operation to be queued to a particular PTC. This read operation will remain queued until a transaction is received for the PTC to which it is assigned. This is the recommended mode when large amounts of data are to be transferred. .p When a task receives an AST informing it that a packet is pending for a PTC for which it has declared affinity, but it is unable to process that packet for some reason, it can issue a special I/O function which causes a Word Count NACK to be issued, thereby rejecting that transaction and freeing the link. .p At Fermilab, we saw the need for some tasks to periodically gain exclusive use of the DR-11W link, preferably without device driver intervention to avoid the system overhead time of performing a QIO. For this purpose, a subfunction bit has been defined for both the read and write QIO functions that will direct the device driver to "sleep", that is, to give up it's ownership of the DR-11W interrupt vector. The intended use of this function requires that a pair of cooperating tasks exist on either side of the link which is to be put to sleep. One of these tasks initiates the transaction by issuing a write-and-sleep operation to a pre-defined PTC. The other task responds to the receipt of a packet AST for that PTC by issuing a read-and-sleep operation. Upon completion of this transaction, the CD: driver in each machine removes it's interrupt vector and replaces it with the address of the nonsense interrupt handler. This fully interlocks the sleeping process. .p When the cooperating tasks have finished their use of the link, they each disconnect from the DR-11W interrupt and issue a special I/O function which restores the driver's interrupt vector and starts the link startup protocol. .p While the DR-11W interrupt is removed, link transaction I/O operations are rejected unless they contain a special subfunction bit which tells the driver to allow them to be queued until the interrupt is regained. .hl 1 Interesting Details of Implementation .P One of the more interesting details of the RSX-11 implementation of this protocol is that it does not depend on the Ancillary Control Processor (ACP) mechanism. Originally, we had intended that the device driver be a much more modest entity, and that an ACP would handle the individual steps of the protocol and maintenance of the PTC tables. When we investigated this path, we found that the overhead time introduced by the ACP scheduling mechanism was longer than our target for total overhead time. .p In order to perform the complex protocol and housekeeping chores, the device driver itself had to be considerably more complex than most RSX-11M drivers are (or should be). .p The CD: driver handles all of it's own queuing of I/O request packets. Write type I/O operations are queued to the standard I/O queue when needed (when the driver is busy upon receipt of the packet). Read operations are handled quite differently. .p When a task creates a Packet Type Affinity, a Packet Descriptor Block (PDB) is allocated in the system pool. This PDB contains the TCB of the owning task, the PTC it refers to, and the AST address specified by the task. The PDB is then linked to the end of a singly linked list whose head is contained in a Unit Control Block (UCB) extention. Contained within the PDB, and initialized at creation time, is an I/O queue header. When a read operation is received for a particular PTC, that is, with the queue subfunction bit set and the third I/O parameter referring to a PTC for which the issuing task has declared affinity, the I/O packet is queued to this PDB I/O queue. For a picture of these pool structures, see figure two. .p One particularly nasty feature of the DR-11W when used as an interprocessor communications device is that interrupts from a connected machine may be lost if the Interrupt Enable (IE) bit in the device's CSR is turned off. This feature dictates some of the more interesting elements of the CD: design. .p The CD: driver maintains two independent execution streams. One stream handles the I/O functions that transfer data. It is driven by the completion of I/O operations if the device is busy, and started by either the receipt of a QIO packet or by an unsolicited link transaction. Transfer functions are terminated through the $IODON executive subroutine. The second execution stream handles so-called control or housekeeping functions. These functions are never queued, and are executed immediately upon receipt of the QIO packet which specifies them. Control functions are terminated through the use of the $IOFIN executive subroutine. .p Because an interrupt from the DR-11W may occur at any time during the protocol and in any driver state, the management of interrupt and fork-level code is complex. Interrupts can arrive faster than we expect them to, or some error may have occurred in the machine the driver is communicating with, causing a link startup code, or simply garbage, to be received. To handle these situations, the driver follows certain rules such that it's current state is always known while it is executing at below the priority of the device interrupt. At the interrupt entrypoint, a dispatch/action state table is consulted. If any currently executing fork-level code needs to be informed of the result of the interrupt, state flags are set which the fork-level code is required to consult before it terminates it's operations. This function is simplified through the use of central routines which effect the entry to and exit from a particular state. .p As was previously mentioned, the CD: informs a task of the receipt of a signal or an unsolicited transaction through the use of an AST. This non-standard AST block is allocated from pool, and queued to the task using the $QASTT executive subroutine. The AST stack contains the PTC which generated the AST, the unit number of the interrupting unit, and either the word count for the data packet or the signal. Including the unit number and the PTC allows one AST routine to suffice for a task which connects to several units and/or has multiple Packet Type Affinities. .p Because the CD: driver maintains non-standard I/O queues, it must intercept and process I/O Kill functions. Any write QIO packets that are on the system I/O queue for the unit are removed by the executive, and then the driver is called to complete the operation by scanning the PDB I/O queues. .p When a task exits, it is important that any pool structures which it has created are removed. To allow the CD: driver to perform this function, an RSX-11M I/O function known as IO.CLN is used. In an RSX-11M system, IO.CLN is issued by the I/O rundown service when a task exits with a file open. The intention is that the FILES-11 ACP will intercept the function and close the file. The CD: driver marks the second word of the LUN entry in the header of any task which creates a Packet Type Affinity. This fools the I/O rundown service into believing that a file is open on that LUN, and it sends the driver an IO.CLN I/O request packet. The driver responds to this packet by removing all PDBs owned by that task. .p An interesting build-time option of this driver is the trace feature. If the trace feature is present, the driver looks through the Partition Control Block (PCB) chain for a partition called CDTRAC. If this partition is available, some words at the beginning of it are initialized, and the partition is marked as busy so that nobody tries to load anything over it. At key points in the driver's code, a TRACE macro is invoked. This macro creates a call to a tracing subroutine and passes a parameter list to it. The trace subroutine checks a flag in the UCB to see if the trace partition was initialized, and if so it checks a bit in the partition to see if tracing was enabled (the trace enable bit is set by an external task). If tracing was enabled, it moves the current clock tics into the next free position in the trace partition (defined by pointers at the beginning of the partition) followed by the parameter list. A dump program has been written which decodes the data in the trace partition and displays them on a terminal or a lineprinter, and which can enable or disable the trace feature. This allows rapid fault isolation. This code and the dumping task are sufficiently modular that the writer of any new driver should consider using them. .hl 1 Timing and Size of Code .p The CD: driver may be generated with any combination of several options. It may contain (or not) physical DR-11W support, intraprocessor (internal to a single machine) communication support, and the trace feature. We assembled the driver using the combinations of these options to determine code size. The results were: .list .le;Physical, Trace, and Intraprocessor - 2859. words. .le;Physical, no Trace, and Intraprocessor - 2011. words. .le;Physical, Trace, no Intraprocessor - 2601. words. .le;Physical, no Trace, no Intraprocessor - 1805. words. .le;No Physical, Trace, and Intraprocessor - 1383. words. .le;No Physical, no Trace, and Intraprocessor - 899. words. .end list .p The size of the pool structures needs to be considered when planning a sysgen for a system which will use this driver. The following numbers are for a loadable driver for a non-multi-user system. The Device Control Block (DCB) is fifteen words. The size of the Unit Control Block (UCB) is thirty-three words, and must be repeated for each unit. The size of the Status Control Block (SCB) is twelve words and must be repeated for each unit. Each Packet Descriptor Block (PDB) takes up seven words. .p The following average timings were made for communications between an 11/34 and an 11/50; .list .le;One word packets - 5.7 ms/Transfer. .le;Ten word packets - 5.8 ms/Transfer. .le;500 word packets - 7.3 ms/Transfer .le;1000 word packets - 8.8 ms/Transfer. .le;2000 word packets - 11.9 ms/Transfer. .le;4000 word packets - 17.9 ms/Transfer. .end list .p These timings compare favorably with those for other high speed devices. We timed transfers to an STC-1900 tape drive at 6250 BPI, and found that a 4000 word transfer took 24.3 ms on average. A 4000 word transfer to a TU-10 (800 BPI) took 247.5 ms on average, to no one's surprise. .p We ran the intraprocessor version of the driver on a PDP-11/70, and recorded an overhead time of less than 5 ms. .p Note that when the Trace feature is in use, the overhead per transfer goes up markedly. .hl 1 Miscellanious Asides .p There are a few things that ought to be noted in passing. .p The RT-11 implementation of this protocol is compatible with the RSX-11M implementation, permitting applications under the two operating systems to communicate. For information about the RT-11 software, see associated paper [2]. .p The intraprocessor communications feature uses the $BLXIO executive subroutine to transfer data from one task buffer to another. If this feature is to be used in your system, you should specify the maximum size for the $BLXIO vector in your SYSGEN options generation session. .p The link startup protocol depends upon having a source of essentially random numbers to function dependably. We used the location $IDLCT which is a count of idle cycles updated by the rotating data light support routine. Unless you wish to modify the driver, we suggest that you include this option in your system generation. We routinely run our system on 11/34s and 11/60s which do not have data lights, and have experienced no problems with that to date. .p We have created a subroutine package, called CDPACK, which provides an operating system independent, FORTRAN-callable interface to the communications services. Currently, versions of this package written for RT-11 and RSX-11M are completed, and a VMS version will be produced as VMS support is added. .p We are currently adapting our Data Aquisition and Analysis systems to use the DR-11W services. Our future projects will probably include remote access to CAMAC device drivers through the use of ACPs and remote servers, File Transfer, and some sort of virtual terminal support. .p The current version of the CD: driver, the trace dumping task, and CDPACK will be on the RSX-11 Special Interest Group tape for this DECUS symposium. .hl 1 Conclusion .p The DR-11W-based communications architecture developed at Fermilab is suitable for situations in which rapid data transfer is more important than such features as error detection and retry, although such features could be added with an application layer. This architecture is currently being used to develop applications to support the high energy physics community at Fermilab, and its widespread use at Fermilab is forseen. .br.sk 1 .center;^&REFERENCES\& .list .le;P.Heinicke, J.Biel, D.Burch, R.Dosen, M.Pyatetsky, D.Ritchie, V.White, 1982 "High Speed Interprocessor Data Links Using The DR11-W", ^&1982 Fall DECUS U.S. Symposium, Anaheim, CA.\& .le;M.Pyatetsky, P.Heinicke, D.Ritchie, V.White, 1982 "Using the DR-11W DMA Device for Interprocessor Communications", ^&1982 Fall DECUS U.S. Symposium, Anaheim, CA.\& .end list .page .literal Simplified Link Protocol .end literal .br.sk 3 .literal Sender Receiver ------ -------- Request Link ======================= Link ACK ====================== Packet Type Code (PTC) ======================= PTC ACK or PTC NACK ====================== Word Count or Signal ======================= Permitted Word Count, or Signal ACK, or Word Count NACK ====================== N Data Words (DMA) ----------------------- End-of-Message Status ====================== .end literal .br.sk 3 .literal figure 1 .end literal