================================================================================ Note 26.0 MicroVAX II multicomputing No replies FURILO::JACKSON 700 lines 19-AUG-1985 11:36 -------------------------------------------------------------------------------- +---------------+ +-----------------+ | d i g i t a l | | uNOTE # 026 | +---------------+ +-----------------+ +----------------------------------------------------+-----------------+ | Title: The MicroVAX II CPU Multicomputing | Date: 28-JUN-85 | | Capability | | +----------------------------------------------------+-----------------+ | Originator: Mike Collins | Page 1 of 12 | +----------------------------------------------------+-----------------+ The MicroVAX II CPU may be configured as an arbiter CPU or as one of three auxiliary CPU's. Therefore it is possible to configure a Q-bus system with multiple MicroVAX II CPUs. This 'multicomputing' capability is the topic of this MicroNote. Multicomputing Introduction The multicomputing capability is a hardware design feature of the MicroVAX II. It allows a maximum of four MicroVAX II CPUs to reside on the same Q-bus. Figure 1 below is a block diagram which shows how 2 of a possible 4 CPUs may be configured on the bus. One of the processors will be the arbiter and the others will be auxiliaries. NOTE There is no shared memory between processors, therefore this system should not be considered as symmetrical multiprocessing. Each processor may use expansion memory modules provided that there are sufficient Q/CD slots in the backplane. See the configuration section for more detail on the configuration rules. +---------------------+ +---------------------+ | +---------+ | | +---------+ | | | | | | | | | +---------+ +---------+ +---------+ +---------+ | Arbiter | | | |Auxiliary| | | |Processor| | Memory | |Processor| | Memory | | | | | | | | | | | | | | #1 | | | | | | | | | | | +---------+ +---------+ +---------+ +---------+ | | | | ----------------------------------------------------------- Q-bus ----------------------------------------------------------- Page 2 Figure 1 - Multicomputing Configuration CAUTION Digital's 32-bit operating systems DO NOT support the multicomputing feature. Users who wish to customize their systems to take advantage of the multicomputing capability may do so but are responsible for designing the software system. The following topics which are pertinent to the multicomputing design will be described in detail: 1. MicroVAX II Memory System 2. Interprocessor Communications Register 3. Arbiter/auxiliary Processor Differences 4. Bootstrapping an Auxiliary Processor 5. Configuration Rules REFERENCES The MicroVAX 630 CPU Module User's Guide (P/N EK-KA630-UG) and MicroNote #22 have more detailed information on all aspects of the MicroVAX II CPU and memory system. MicroVAX II Memory System -------------------------- Before describing the details of the multicomputing feature it is necessary to understand the memory system of a MicroVAX II. The memory system of the MicroVAX II is unique. Unlike previous Q-bus processors, the memory modules do not communicate to the CPU over the Q-bus. Instead, the communication and transfers occur over a dedicated, closely coupled interconnect. This design allows for very fast reads and writes to memory (400 nsec). Not only are the transfers very fast, but none of this CPU-memory activity appears on the Q-bus. Therefore the Q-bus is strictly for I/O. Since memory is not directly on the Q-bus it is necessary to allow DMA devices on the Q-bus to access memory. This feature is called the Q-bus I/O map. It is a mechanism which maps Q-bus addresses (22-bit physical address) to the local memory system of a MicroVAX II (24-bit physical address). This same concept is used on larger UNIBUS based VAXes. When one or more auxiliary processors are added to a system, remember that each has its own local memory system. Therefore it is intended that they will each operate out of their own memory. This design is Page 3 well suited for applications which can be easily partitioned between multiple processors such that only messages are passed between them. It would be possible for a processor to operate out of memory on the Q-bus but this would be unconventional from a systems standpoint and would decrease system performance significantly. Digital's operating systems do not support the use of standard Q-bus memories in a MicroVAX II. With only one processor on the Q-bus, the system software has complete control over DMA transfers. The software controls the contents of the Q-bus map registers. First, it determines whether the register is enabled and second, if it is enabled, where the DMA transfers are directed in the local memory system. In a multiple processor configuration (since each has its own set of Q-bus I/O map registers) it is now possible for multiple map registers to respond to a single Q-bus address. It is the responsibility of the system software running on each processor to cooperate and assure that one and only one map register will respond to any Q-bus address. Otherwise multiple memory locations may respond and the results are indeterminate (i.e. multiple BRPLYs on the Q-bus for a single address). A similar situation occurs if standard Q-bus memories are added to a MicroVAX II system. The system software is again responsible for assuring that for any address on the Q-bus which is 'shared' by a Q-bus memory board and a Q-bus I/O map register, the map register is disabled to allow only a Q-bus memory reply. Q-bus memories would not be added to 'typical' systems. However there are some special applications which may require this capability. For example, a graphics system where the Q-bus memory is the bit map of a graphics display. On power-up, the MicroVAX II boot ROM will check for the presence of Q-bus memory and clear the valid bits of the corresponding mapping registers. This will prevent the above situation from occurring as long as the system software does not alter the state of these valid bits at a later time. Interprocessor Communications Register -------------------------------------- The interprocessor communications register (ICR) is the primary mechanism of the MicroVAX II CPU which enables multiple processors to cooperate and reside on the same Q-bus. Figure 3 describes the ICR and the bit definitions. The ICR provides the following four functions: 1. Any processor on the bus may interrupt another without using the Q-bus interrupt lines. This is done by setting the appropriate bit in the ICR of the second processor. The CPU will then interrupt at IPL 14 (HEX) with an interrupt vector of 204 (HEX). Page 4 2. The ICR also has a bit which controls external access to local memory via the Q-bus map. When set this bit allows external devices to access the local memory system. When reset, this bit prevents the local memory system from responding to any Q-bus address references. 3. If a processor is an auxiliary then the ICR provides a mechanism which allows it to be halted by other CPUs. This feature is used at power-up to coordinate the bootstrap process. 4. Parity errors are identified when one occurs during accesses to the MicroVAX II's local memory. The address for the ICR is located in the Q-bus I/O page address space and may be accessed by any device which can become Q-bus master. The ICR is byte-addressable. Since it is possible to put a maximum of 4 MicroVAX II CPUs on one Q-bus they each have their own unique ICR. Table 1 lists each processor and the associated ICR address. Table 1 - Interprocessor Communication Register Address Assignments +--------------+----------------------+----------------------------+ | Processor | 22-bit Octal Address | 32-bit Hexadecimal Address | +--------------+----------------------+----------------------------+ | Arbiter | 17 777 500 | 2000 1F40 | +--------------+----------------------+----------------------------+ | Auxiliary #1 | 17 777 502 | 2000 1F42 | +--------------+----------------------+----------------------------+ | Auxiliary #2 | 17 777 504 | 2000 1F44 | +--------------+----------------------+----------------------------+ | Auxiliary #3 | 17 777 506 | 2000 1F46 | +--------------+----------------------+----------------------------+ 1 1 1 1 1 1 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 +--+-----------------+--+--+--+--+-----------+--+ | | 0 0 0 0 0 0 | | | | | 0 0 0 0 | | +--+-----------------+--+--+--+--+-----------+--+ | | | | | DMA QPE ----+ | | | | | | | | AUX HLT -------------------------+ | | | | | | DBI IE -------------------------------+ | | LM EAE ----------------------------------+ | | DBI RQ -------------------------------------------------+ Figure 3 - Interprocessor Communications Register Page 5 Bit(s) Mnemonic Meaning <15> DMA QPE DMA Q22-Bus Address Space Parity Error. This read-only bit is set if Memory System Error Register bit <04> (DMA QPE) is set. The DMA QPE bit indicates that a parity error occurred when an external device (or CPU) was accessing the MicroVAX II CPU local memory. <14:09> - Unused. Read as zeros. <08> AUX HLT Auxiliary Halt. On an auxiliary MicroVAX, AUX HLT is a read-write bit. When set, typically by the arbiter CPU, it causes the on-board CPU to transfer program control to the Halt Mode ROM Code. On an arbiter MicroVAX II, AUX HLT is a read-only bit which always reads as zero. It has no effect on arbiter CPU operation. <07> - Unused. Read as zero. <06> DBI IE Doorbell Interrupt Enable. This bit, when set, enables interprocessor doorbell interrupt requests via ICR <00>. When the on-board CPU is Q22-Bus master, DBI IE is a read-write bit. When an external device (or CPU) is bus master, DBI IE is a read-only bit. DBI IE is cleared by power up, by the negation of DCOK and by writes to the Bus Initialize Register. <05> LM EAE Local Memory External Access Enable. This bit, when set, enables external access to local memory (via the Q22-Bus map). When the on-board CPU is Q22-Bus master, LM EAE is a read-write bit. When an external device (or CPU) is bus master, LM EAE is a read-only bit. LM EAE is cleared by power up, by the negation of DCOK and by writes to the Bus Initialize Register. <04:01> - Unused. Read as zeros. <00> DBI RQ Doorbell Interrupt Request. If ICR <06> (DBI IE) is set, writing a "1" to DBI RQ sets DBI RQ, thus requesting a doorbell interrupt. If ICR <06> is clear, writing a "1" to DBI RQ has no effect. Writing a "0" to DBI RQ has no effect. DBI RQ is cleared when the CPU grants the doorbell interrupt request. DBI RQ is held clear whenever DBI IE is clear. Page 6 When a processor is interrupted via its ICR the interrupt vector is 204 (Hex). This vector is used when the interrupting device is the arbiter, auxiliary 1, auxiliary 2, auxiliary 3, or any device which may become Q-bus master. Therefore it is the responsibility of the interrupt service routine to determine which device caused the interrupt since the same vector is used no matter what device set the DBI RQ bit. This interrupt occurs at the same interrupt priority level (IPL) as IRQ4 on the Q-bus. NOTE Following such an interrupt, the MicroVAX II sets the IPL to 14 (Hex). This is different from what happens after a standard Q-bus interrupt. When a Q-bus interrupt occurs on IRQ4, the processor raises the IPL to 17 (Hex) and it is the responsibility of the interrupt service routine to lower the IPL later on. Arbiter/Auxiliary Processor Differences --------------------------------------- There are several differences between arbiter and auxiliary processors. It is important to understand these differences when configuring multiple processors into a system. 1. The arbiter MicroVAX II arbitrates bus mastership in accordance with the Q22-Bus DMA protocol. The arbitration logic is disabled on an auxiliary MicroVAX II. 2. Both the arbiter and auxiliary MicroVAX II request bus mastership via the Q22-Bus DMA Request protocol. a. They both assert BDMR on the Q22-Bus. b. The arbiter MicroVAX II receives DMGI from its arbitration logic. But an auxiliary receives DMGI from its Q22-Bus BDMGI pin. c. Only the auxiliary MicroVAX II actually asserts BSACK on the Q22-Bus. 3. The arbiter MicroVAX II asserts the Q22-Bus BINIT signal when DCOK is negated and when its CPU software writes to its Bus Initialize Register. An auxiliary MicroVAX II never asserts Q22-Bus BINIT, but receives BINIT and uses it to initialize the MicroVAX chip and to clear all internal registers which are cleared by the negation of DCOK. 4. The physical address of the Interprocessor Communication Register is different for each of the four MicroVAX II arbiter/auxiliary configurations. Page 7 5. An auxiliary MicroVAX II can be halted by setting bit <08> (AUX HLT) of its Interprocessor Communication Register. On an arbiter MicroVAX II this feature is disabled and AUX HLT is a read-only bit which always reads as zero. 6. The CPU halts are controlled by the external connector HLT ENB input. However, the external halts which are affected differ somewhat for the arbiter and auxiliary MicroVAX II modules. If the HLT ENB signal is asserted (low) the a MicroVAX II CPU will halt and enter the console program if: a. The program executes a halt instruction in kernel mode. b. The console detects a break character. c. Arbiter CPU only - the Q-bus halt line is asserted. d. Auxiliary CPU only - the Interprocessor Communication Register AUX HLT bit is set. If the HLT ENB signal is negated then: a. The halt line and break character are ignored and the ROM program responds to a halt instruction by restarting or rebooting the system. b. If the MicroVAX CPU is an auxiliary, the ROM program responds to the assertion of the ICR AUX HLT bit by rebooting. The state of the HLT ENB bit can be read by software via the Boot and Diagnostic Register. 7. Each arbiter or auxiliary MicroVAX II module can field interrupt requests from its interval timer, from its console device, and from its interprocessor doorbell. Only the arbiter MicroVAX II can field interrupts from Q22-Bus interrupt request lines IRQ7-4. 8. The arbiter asserts BIAKO on the Q22-Bus when it responds to a Q22-Bus interrupt request. An auxiliary asserts BIAKO on the Q22-Bus when it receives the assertion of BIAKI in order to pass it through to devices after it. 9. Although both the arbiter and auxiliary MicroVAX II modules contain the same time-of-year clock and battery back-up circuitry, it is assumed that the auxiliary will be configured without batteries and that its clock will never actually be enabled. This configuration will ensure a single time base for the system. If an auxiliary needs to set its time-of-year clock it can do so by referencing the arbiter's clock. 10. An arbiter processor can bootstrap from a variety of devices but an auxiliary will always boot using the ROM bootstrap protocol. See the following section for a detailed description Page 8 of the bootstrap process. Bootstrapping an Auxiliary Processor ------------------------------------ Since there are distinct differences between the operation and capabilities of arbiter and auxiliary processors, it is necessary to bootstrap them differently. An arbiter processor bootstraps in the conventional manner from one of several devices but an auxiliary boots using only one method. On power-up both processors initialize themselves and perform some self tests. At this point the primary bootstrap (VMB) begins to execute. An arbiter processor may bootstrap from any one of the following devices: 1. DU type device (RX50, RDxx, RC25 or RAxx) 2. TK50 tape 3. ROM bootstrap protocol 4. Ethernet (DEQNA) An auxiliary processor will not attempt to boot from disk, tape or ethernet. It will always boot via the ROM bootstrap protocol, which is described below. In order to synchronize the bootstrap process, after power-up and initialization an auxiliary processor will not boot until it is allowed to do so by the arbiter. The controlling device is not required to be the arbiter, it could be another auxiliary or DMA device which is bus master, but it is the most logical point of control. The following steps summarize the bootstrap procedure for a system with an arbiter and one (or more) auxiliaries: 1. Both types of processors initialize themselves after power-up. 2. Self-diagnostics execute to check out major sections of the CPU. 3. The arbiter boots from one of the four device types listed above. a. While the arbiter is booting the auxiliary completes its diagnostics and waits to start its bootstrap process. b. The auxiliary clears the valid bit in each of its Q-bus map registers to prevent accidental transfers to or from its local memory. Page 9 c. The auxiliary loops doing nothing until the AUX HLT bit in its ICR is cleared. When this bit is finally cleared, the auxiliary boots via the ROM bootstrap protocol. 4. Somewhere in the system the appropriate data structure has been created to allow the auxiliary to boot via the ROM bootstrap protocol. This can be done several ways: a. The bootstrap code is actually in ROM on an MRV11-D module. b. The bootstrap code is loaded into non-volatile RAM. c. The arbiter CPU loads the bootstrap code into its own local memory then sets up some Q-bus map registers so that this data can be seen by the auxiliary CPU over the Q-bus. This method is the most flexible since the bootstrap code can be changed easily. 5. When the arbiter is ready and knows auxiliary boot code is available, it allows the auxiliary CPU to bootstrap by clearing the AUX HLT bit in its ICR. ROM Bootstrap Protocol. To locate a PROM bootstrap, VMB searches the Q22-bus address range from high to low addresses by page (512 bytes per page) looking for readable memory. If the first six longwords of any such page contains a valid PROM "signature block" (see figure 4), VMB passes control directly to the bootstrap code in the PROM, it does not copy the PROM code to local memory for execution as it does for all other secondary bootstraps. Note that while defined as an MRV11 PROM or equivalent bootstrap, VMB does not actually require that the signature block or the bootstrap code be in PROM. It could be in ROM, nonvolatile RAM or it could be loaded into another MicroVAX II's RAM and mapped to the Q22-bus thus enabling an auxiliary to see it. RB: +---------------+---------------+---------------+---------------+ + 0: | check byte | any value | 0 | 18 (hex) | +---------------+---------------+---------------+---------------+ + 4: | any value | 1 | 0 | +-------------------------------+---------------+---------------+ + 8: | size of PROM in pages | +---------------------------------------------------------------+ + 12: | must be zero | +---------------------------------------------------------------+ + 16: | offset into PROM to start execution | +---------------------------------------------------------------+ + 20: | sum of previous three longwords | +---------------------------------------------------------------+ Figure 4: PROM Bootstrap Memory Format Page 10 RB + 0: This byte must be 18 (hex). RB + 1: This byte must be 0. RB + 2: This byte may be any value. RB + 3: This byte must be the ones complement of the sum of the previous three bytes. RB + 4: This byte must be zero. RB + 5: This byte must be 1. RB + 6: These two bytes may be any value. RB + 8: This longword contains the size in pages of the PROM. RB + 12: This longword must be zero. RB + 16: This longword contains the byte offset into the PROM where execution is to begin. RB + 20: This entry is a longword containing the sum of the previous three longwords. Configuration Rules ------------------- In order to configure multiple MicroVAX II CPUs onto one bus, 4 issues must be addressed: 1. Q-bus slot requirements for the CPU and MS630 memory boards. 2. Compatible enclosures or boxes. 3. Bus termination. 4. Using a PDP-11 CPU as the Arbiter. 1. Q-bus slot requirements. To operate properly both the MicroVAX II CPU and all versions of the MS630 memory modules MUST be inserted in Q-bus slots which feature Q-bus signals on the A/B side and the CD interconnect on the C/D side. The MicroVAX II CPU and any expansion memory modules must all be adjacent in the backplane. WARNING The MicroVAX II CPU WILL be damaged if inserted into a slot which has Q-bus signals on the C/D side. Page 11 2. Compatible enclosures or boxes. Since Q/CD slots are required, there are only 4 enclosures which are compatible with these modules. They are: a. BA23 (8 slots total, first 3 are Q/CD) b. BA123 (12 slots total, first 4 are Q/CD) c. BA11-S (9 slots total, all 9 are Q/CD) d. BA11-N (9 slots total, all 9 are Q/CD) This enclosure is electrically compatible but since it is older it is only 18-bit compatible and has not been tested for FCC compliance in a system package. Since the BA11-S is the 22-bit, updated version, the BA11-N enclosure is not recommended. 3. Bus termination. Each MicroVAX II processor has 240 ohm termination. When multiple CPUs are placed into one system, take care to abide by the Q-bus specification regarding termination and configuring multiple box systems. REFERENCE Reference MicroNote #29, entitled 'Q-bus Expansion Concepts' and appendix A of the Microsystems Handbook (P/N EB-26085-41) for more information concerning Q-bus configuration rules. The BA11-S enclosures are best suited for multiple CPUs because they have backplanes with the Q22/CD configuration in all slots. Therefore multiple CPUs and expansion memory can be accommodated most easily. The other potential boxes, the BA23 and BA123, are not as flexible because they only have 3 and 4 Q22/CD slots respectively. When using multiple MicroVAX II CPUs in one system the goal should be to configure each backplane with 120 ohm termination if the number of backplanes used is one or two. In the three backplane case, only the first and third backplanes should have 120 ohm termination. This implies that no MicroVAX II CPUs should be configured into the middle backplane of a three backplane system (three backplanes is the maximum number allowed for the Q-bus). The following configurations assume the BA11-S enclosures will be used. Two CPU System: A two CPU system is the easiest case to configure. As mentioned earlier, each CPU has 240 ohm termination. With two CPUs in the same backplane, the proper overall termination of 120 ohms is achieved. The first CPU has 240 ohms in parallel with 240 ohms on the second CPU which is 120 ohms, assuming there is no termination on the backplane (which is Page 12 true for the BA11-S enclosure). Three CPU System: A three CPU system should be configured in 2 backplanes to maintain 120 ohm termination in each. The first 2 CPUs should be configured in the first box which terminates that backplane properly as in the case above. The third CPU should be placed in the second backplane. The modules of the expansion cable set must be terminated differently. The module of the expansion cable set in the first backplane should add no additional termination to the system while the module in the second backplane should be terminated with 240 ohms. Therefore the second backplane is also terminated in 120 ohm (240 on the processor in parallel with 240 on the expansion module). Four CPU System: A four CPU system should also be configured in 2 backplanes. The first backplane will have 2 CPUs for 120 ohm termination and the second backplane will also have 2 CPUs for 120 ohm termination. But the expansion cable set connecting the two backplanes will not be the same as in the three CPU system. In this case there should be no termination on either of the expansion modules. These same basic rules should be followed if the BA23 enclosure is used. However the configurations become more constrained because of the relatively few number of Q/CD slots. The configuration also becomes more complicated because the BA23 has built in termination on the backplane (termination resistors are socketed and therefore removable). 4. Using a PDP-11 CPU as the Arbiter. The arbiter can be a Q-bus PDP-11 CPU as well as a MicroVAX II CPU. In this configuration only the 3 auxiliary MicroVAXes could be added to the system. Although there are tremendous architectural differences between a PDP-11 and MicroVAX CPU, they both perform the arbiter function on the Q-bus and therefore such a configuration is possible. Two issues arise from this 'mixed' configuration. 1. All Q-bus PDP-11s to-date have their main memory on the Q-bus. In order to talk to an auxiliary MicroVAX CPU's local memory, part of the Q-bus physical address space must be reserved for mapping to that local memory. 2. The PDP-11 processor will not have an interprocessor communication register because that feature is unique to the MicroVAX II CPU. Therefore, if it is necessary to have the capability for an auxiliary CPU to interrupt the arbiter, an additional device must be added to the system. This device will look like the interprocessor communication register for an arbiter and will convert and ICR interrupt into a conventional Q-bus interrupt (there is no capability which allows an auxiliary to interrupt the arbiter via the standard Q-bus interrupt protocol).