Design considerations for using Ethernet rings
Industrial Control and Automation networks generally use unique topology.Unlike typical office Ethernet 'star' networks where a multi-port switch provides point-to-point links to the other nodes, the control layer of an industrial network is usually based on a ring layout.A ring or daisy-chain simplifies cabling and can extend Ethernet's reach without going to the expense of a fibre-based system. Mike Jones examines the design implications.
Reduction in the required cabling of an Ethernet ring network provides benefits welcome to the automotive and aerospace industry. Less cabling means weight reductions which has a direct impact on fuel efficiency and hence, overall costs. It also suits the layout of the typical automation cell where production tends to take place in a linear progression down the factory floor.
The introduction of Ethernet over lightweight Plastic Optical Fibre (POF) has added a further dimension by lowering of costs both in money and weight. An example applications is shown in Fig. 1 using a simple surveillance system distributed around a building perimeter. By linking each camera/sensor as a physical network ring around the perimeter, such techniques will prevent the need for a series of longer cable runs each routed back to a central switch. It also makes it easier to keep to the 100m link limit specified in IEEE 802.3 section 40.7. The basic building block in a ring network is the 3-Port switch. As we will discuss shortly the Ethernet ring requires managing to ensure that one of the links is 'broken' to prevent endless message looping.
Fig. 1.A ring network requires managing against the endless packet loop
However, there are two major issues as a result of implementing an Ethernet ring network; management of the loop and increased latency. A ring network also limits bandwidth, but this can be deemed as a minor issue as the 100Mbps offered by Fast Ethernet is usually more than adequate for control-layer networks.
Managing the Ethernet Ring
Unlike a token-ring network, which has been the basis for much of the IEEE 802.3 specification evolution, it is in fact forbidden to configure Ethernet as a true ring. Any loops within an Ethernet network will result in the duplication of packets that are forwarded in endless loops, quickly degrading the bandwidth and efficiency of a network. There is no available common standard that specifically deals with the management of Ethernet rings. Proprietary network market solutions provide product differentiation and various techniques can be used for the implementation of network ring management. However, when looking for a standards-based solution for ring management, Spanning Tree or Rapid Spanning Tree Protocol may fit the bill.
Spanning Tree Protocol (STP) is defined in IEEE 802.1d and creates a 'spanning tree' within a mesh network of Ethernet switches by blocking duplicate links to form a single active path between any two network nodes. The redundancy in the network can provide automatic backup paths if any of the active links fail.
Fig. 2. Managing network loops with Spanning Tree Protocol
This management of the redundant links is very desirable to reduce the down time industrial networks. For example, the network shown in Fig. 2 contains a loop between links 2, 3, 4 and 5. Here STP will force one of these links into a blocked state to break the loop. In this case link 4 is blocked, which can then act as the backup link. If any of the other active links 2, 3 or 5 subsequently fail, the protocol will disable the blocked state to enable full connectivity between the switches. Note in this example there is no redundancy backup for link 1.
STP is conducted by each switch in the network regularly exchanging information on the status of network, every two seconds. This information is carried in special data packets called Bridge Protocol Data Units (BPDUs), identified by the Ethernet multicast address 01:80:C2:00:00:00. There are three stages of operation of the protocol:
Electing a root switch;
- Finding all paths to the root switch and determining the 'least cost' paths;
- Disabling all other root paths.
The root switch is elected either manually or by using the unique Bridge Identification or BID. The BID is a concatenation of a given priority number (0 to 65535) and MAC address of the switch. The switch with the lowest priority number is elected to become the root switch, which is designated as the centre of the tree. Each switch will then determine all the different paths to the root switch and select the 'least cost' paths using the same algorithmic process. All other paths to the root switch will be disabled or blocked.
When a device is first powered up in a network operating STP, it will not immediately start to forward data. Instead it goes through a series of states by processing the BPDUs to determine the topology of the network, which can take up to 50 seconds. The STP switch port states are as follows:
- Listening - The switch processes BPDUs and awaits possible new information that would cause it to return to the blocking state;
- Learning - While the port does not yet forward packets it does learn source addresses from incoming packets and updates the Forwarding Table accordingly;
- Blocking - A port that would cause a switching loop. No user data is sent or received but it may go onto forwarding mode if the other active links fail and the spanning tree algorithm determines the port may transition to the forwarding state. BPDU packets are still received in a blocking state;
- Forwarding - A port receiving and sending data ie. normal operation. STP still monitors incoming BPDUs that would indicate it should return to the blocking state to prevent a loop;
- Disabled - Any port can be manually disabled by the network management.
Figure 3 illustrates the process flow through the STP states.
Fig. 3. Flow of STP states
Although the STP algorithm is executed in software it is a Layer 2 (Ethernet) protocol. Support will be required by the Ethernet switch to support the five states. BPDU packets received by the switch also need to be identified and automatically passed to the processor port to process. All of our managed Ethernet switch components support such hardware features.
One of the problems with using STP to provide redundancy in industrial networks is switchover time. A delay of just under a minute to recover from a fault condition is far too long for automation networks. Faster spanning tree convergence is provided by the more recent IEEE 802.1w Rapid Spanning Tree Protocol (RSTP) to reduce the down time to around two seconds. However, even RSTP does not meet the demands of all customers, where sub-100ms failover time is not an uncommon requirement.
Virtual LAN support
To improve switchover performance such protocols need to be adapted to specific custom solutions. By considering that a ring is a single subset of the vast mesh network configurations, management performance can be improved, whilst simultaneously reducing the complexity.
The next section demonstrates how VLAN (Virtual LANs) support in managed Ethernet switches, like Micrel's KSZ8893MQL (3-port) and KSZ8995MA (5-Port), can help reduce the network management effort and costs.
VLAN is a method of creating independent logical networks within a physical network. By blocking specific ports or switching paths VLAN can configure and manage Ethernet ring networks with minimal software support. There are two main types of VLAN, port-based and tag-based.
With port-based VLAN, each switch port is manually configured to select which other local ports are members of its VLAN. Ingress packets are only allowed to be forwarded to other port members of the VLAN.
Fig. 4. IEEE 802.1q Tag Format
With tag-based VLAN as defined by IEEE 802.1q, a 4-byte tag containing a 12-bit VLAN identifier (VID) is added to each packet (see Fig. 4). A programmed VLAN table is held by the switch which lists the port members against each valid VID.
Identifying nodes using port VLAN
Like with all solutions the key is to select a master (or root) node and for it to then discover the network topology. For a ring network this means learning the address and order of each node with respect to the master. An example of how to identify and order the slave nodes, using a simple port-based VLAN implementation is given in Fig. 5. Here the Master node sends out a broadcast 'invite request' to each of the Slave nodes (SW1,2,3) to reply, providing essential log information such as ID, address, etc. Dynamically configuring the Slave node port VLAN setting, results in 'known' nodes passing the request through to the next 'unknown' Slave node in the ring. This unknown Slave node will forward the incoming request to port 3, the processor port, due to the VLAN configuration.
Fig. 5. Using VLAN to setup a simple redundant ring network
Latency in ring networks
Generally, real-time Ethernet performance is performed by time-multiplexing the communication on the network between the different nodes. Critical isochronous data transfer is performed at the start of the cycle, followed by non-critical TCP/IP traffic. Such methods guarantee the real-time performance of the network while allowing compatibility with enterprise TCP/IP networks. Multiple Industrial Ethernet standards have emerged based on such techniques, for example Profinet and Ethernet Powerlink, with one critical factor of concern: node latency.
Ethernet ring topology networks can degrade real-time performance due to increased latencies. Network latency is increased as a result of data passing through each node in the ring before reaching its destination. The latency through an Ethernet switch is directly proportional to the packet size, due to the store and forward architecture. The total latency can be calculated by the simple formula below:
Total Latency = (Packet size x 8) /rate + forwarding delay
The forwarding delay in the switch is normally small, constant and independent of packet size. A typical figure for any Micrel-based Ethernet switch is around 2.7µs. Hence, the switch latency for a 64-byte packet will be approximately 7.8µs compared to a 124µs delay when a 1518 byte is forwarded. From this we can conclude the following tips when designing an Ethernet ring network:
Fixing the size of the packets in a network will provide constant switch latency;
Switch latency can be reduced by reducing the packet size to a minimum.
Priority schemes do not actually reduce the packet latency through a switch. Priority, often know as QoS (Quality of Service) only reduces relative packet latency during congestion periods. Congestion will occur when, using the example in Figure 6, both the processor (at port 3) and the ingress port 1 wish to forward a packet onto the ring (port 2). In this case one of the packets will be subject to additional delay, whilst waiting in the egress queue. Priority schemes, typically IEEE 802.1p, can help higher priority packets jump the queue. The worst case scenario occurs when packets are dropped and thus require re-transmission (via TCP or similar). IEEE 802.1p uses the same tag, as defined by 802.1q (see Figure 3) to classify each packet with a priority level by decoding the 3-bit Priority code. So to reiterate, priority schemes can only reduce the additional network latency variations but cannot reduce the fixed switch latency.
To reduce latency jitter in the network, the Ethernet Powerlink Group recommends using 100Base-TX/FX Ethernet repeater hubs. Repeater hubs deploy a 'Cut-through' architecture which significantly diminishes latency by forwarding the incoming packet to all ports (with the exception of the ingress port) before receiving the complete packet. Lower latency is offered compared to store&forward switches, independent of packet size.
However, repeater hubs today are rare and lack features such as tag-VLAN common to managed switches. Repeater hub implementation tends to be FPGA-based with external Ethernet PHYs, increasing risk, cost, board space and time-to-market. A solution can be found by using KSZ88xx family of switch and controller silicon which supports a low latency repeater mode. This mode provides a maximum port-to-port latency of 310ns and total deviation of a mere 40ns, substantially easing the design problem for critical applications.
Total node-to-node latency, although important, can be less critical than latency variations, known as latency jitter. Fixed latencies can be compensated for in a network while variations cannot, directly resulting in reduced accuracy for real-time packet delivery.
To overcome latency jitter many Industrial Ethernet standards are starting to adopt higher layer protocols such as IEEE1588 to operate each node from a common clock source. IEEE 1588 uses UDP (User Datagram Protocol) packets over IP on the Ethernet network to provide synchronisation in a network down to a microsecond. Such performance fulfils the stringent real-time requirements for motion control applications. (See Dirk Mohl's article on page 10 of this issue.)
To achieve synchronisation with the rest of the network each node must determine which clocking source it should use. All nodes perform the 'Best Master Clock' algorithm to select timing. If the node is to be a master to many of the nodes in the ring then a high precision source, such as GPS, is used. If the node is not designated as a master then it will extract timing from the network using the IEEE 1588 protocol. Failure to extract timing from the network will result in using the onboard local oscillator. The master is responsible for providing all the slaves in the network with an advertisement of it's position. If the slave does not receive any advertisement from the master then it will designate itself as the master.
To synchronise the master and slaves IEEE 1588 operates PTP (Precise Time Protocol) based on IP multicasting. The switch needs to identify PTP packets and timestamp the ingress and egress PTP packets. In the ingress direction the switch will forward the incoming PTP packets directly to the processor port. The timestamp point occurs after the SOF (Start of Frame), at the first bit of the DA (Destination Address).
IEEE 1588 time stamping can be implemented in software via the processor or hardware using a FPGA sitting between the switch and PHYs. However, software methods are less accurate as it is difficult to compensate for the non-deterministic switch latency. Synchronisation accuracy of typically 10µs to 100µs is achievable in software compared to less than 1µs using a hardware approach.
The traditional point-to-point 'star' Ethernet network topology has been replaced by the favoured ring for many industrial applications. The Ethernet ring network brings the added benefits of a reduction in cable, reducing both weight and cost, as well as simplifying the cabling installation. However, these advantages are not without their drawbacks as a result of using ring topology.
It is generally forbidden to configure Ethernet in a true ring. Loops in the network will quickly result in bandwidth degradation due to an increasing number of packets circulating in the endless loops. As we have shown protocols, such as Spanning Tree, and Ethernet filtering and VLAN features can be effectively used to manage the loop to prevent such catastrophic failure.
The second issue for Ethernet ring networks is a concern for realtime applications due to their increased packet latency. Low latency repeater modes, as uniquely supported by Micrel's KSZ8893MQL/FQL 3-Port Switch Family, can offer one solution. Alternatively networks can be synchronised using protocols like IEEE 1588 to overcome such latency issues in real-time applications.
Mike Jones is senior FAE with the semiconductor company Micrel