Performance metrics for Industrial Ethernet

Industrial Ethernet * is appealing for mission-critical use such as control and safety. The application context demands a fresh look at the performance issues involved. The University of Michigan's Engineering Research Centre for Reconfigurable Manufacturing Systems has been exploring the issue with a reconfigurable factory test bed using fully networked control, diagnostics and safety capabilities. Work has included a comparative evaluation of two common Industrial Ethernet protocols, and development of a cost-based methodology and calculator for evaluating network implementation and re-configuration choice. It has also shed new light on the link between software protocol and network determinism.
By James Moyne and Dawn Tilbury

As Industrial Ethernet becomes more prevalent on the manufacturing floor it is increasingly being considered as a universal networking solution^1,2. Its dominance in the 'value-add' diagnostics domain is now spreading to the mission critical domains of control and even safety networking³. While the capability of Ethernet has been proven acceptable in the office and high level manufacturing environments, issues of performance must be considered as we push this technology into areas such as I/O, motion control and safety⁴.

The University of Michigan's Engineering Research Centre for Reconfigurable Manufacturing Systems (ERC-RMS) has been exploring the applicability of Ethernet to the factory floor, focusing on application domains of diagnostics, control and safety⁵. Key results of these efforts are summarised in this paper. Specifically, after describing the issue of network performance and providing background information on the ERC-RMS effort, industrially relevant Ethernet performance metrics are presented, followed by results of Ethernet performance evaluation against some of these metrics. This paper concludes with a summary of key points.

* [To the author's knowledge there is no standardised definition of Industrial Ethernet. In this paper we use the term to mean Ethernet protocols that are applied in a manufacturing setting. Thus for example it could refer to specific solutions such as Modbus/TCP or just the application of 'traditional' Ethernet to manufacturing data collection and control.]

Network performance tradeoffs
The advantages of industrial networked systems over their 'hard-wired' point-to-point predecessors are well documented and include decreased wiring and maintenance costs, expanded flexibility of control architectures and diagnostics capabilities, and increased reliability, re-configurability and interchangeability³. However these advantages come at a cost of potential end-to-end performance reduction that results from the fact that many nodes are sharing the communication medium, and individual nodes must support added software communication protocol layers for the encoding and decoding of network messaging. This performance loss is illustrated with the network performance curve, termed the 'Lian Curve', shown in Fig. 1. Here we see that digital control performance improves as sample time is decreased (i.e., sample rate is increased). However when this digital control is networked, the performance begins to worsen if the sampling rate is increased beyond an optimal point. This is because network and/or node congestion begins to introduce delays and delay variability. A performance goal in any network application is to determine the ideal operating region 'sweet spot' and stay in, or just to the left of, that sweet spot as the network is reconfigured (e.g., as nodes are added).

For the most part all time-based network performance issues can be traced back, at least in-part, to this Lian Curve phenomenon. Additional information on the Lian Curve can be found in references 6 and 7.

The ERC-RMS was established by the National Science Foundation in 1996 to pursue advancements in reconfigurable manufacturing⁵. Today the ERC has a broad membership base of automotive manufacturers and suppliers. It focuses on providing solutions that will allow manufacturers and suppliers to respond rapidly by 'reconfiguring' to changes in demand, product, operations, and factory floor dynamics such as system failures. A key research focus of the ERC relates to networks and network performance because these networks, if properly implemented, can be important tools to facilitate re-configurability.

Fig. 1: Lian Curve: Illustrating Networked Control System performance as a function of sampling time

A key deliverable result of this work has been the reconfigurable factory test bed RFT, illustrated in Fig. 2a, which is used to demonstrate research results, investigate the collaboration of these results, and serve as an education and technology transfer mechanism⁸. With respect to the latter, the RFT uses industrial quality solutions wherever possible so that a focus can be placed on the optimal utilisation of these tools. Along those lines, as shown in Figures 2b and 2c, the RFT employs common industrial network technologies of DeviceNet and Profibus, OPC (over switched Ethernet), and SafetyBUS, for networked capabilities of control, diagnostics and safety respectively.

Fig. 2a: ERC-RMS Reconfigurable Factory Test bed illustrating the overall architecture.

Figures 2b (top) and 2c: ERC-RMS Reconfigurable Factory Test bed: control, diagnostics and safety functionalities

Ethernet performance metrics
The need for Ethernet performance metrics is driven by the desire to expand Ethernet throughout the factory, across domains of control, diagnostics and safety, and all the way down to the I/O level. Considering common questions that are asked about Industrial Ethernet when moving into new domains provides insight to help derive and prioritise these metrics. Typical questions include:

• Can I use Ethernet down to the I/O level?
• Should I partition my networks at different levels?
• Should I put safety, control and diagnostics on one, two or three networks?
• Where is the delay and delay variability occurring?
• What are the industry de facto standards?
• What is the cost-tradeoff of a decision?
• What is the complexity-tradeoff of a decision?
• What is the performance cost of security (e.g., VPN) or common Industrial Ethernet application protocols (e.g., OPC)?
• What are the differences between the Industrial Ethernet technology varieties? Which differences are important?

The ERC-RMS conducted a network performance workshop in 2006 to map and resolve these types of question into a ranking of issues that should be considered when applying Industrial Ethernet⁹. The output of that workshop, which reflects contributions from network technology providers, users and researchers is summarised in Table 2a.

As noted earlier, the ERC-RMS has been conducting a number of research activities focused on evaluating Industrial Ethernet performance. These activities are itemised in Table 2b along with a mapping to the performance issues identified in the 2006 workshop; this mapping illustrates that the ERC-RMS research focus is well-aligned with current issues facing the Industrial Ethernet community.

In the next section of this paper, key results from each of these research activities (underlined in Table 2b) are summarised. Further information on all topics identified in Table 2b can be found in the literature referenced at the end of this paper.

Issue	Issue
I-01	Node performance is most important performance consideration Speed, jitter, reliability, determinism Prioritisation on the network is key
I-02	Ease of use and diagnostic tools From a plant-floor personnel perspective
I-03	Cost versus security tradeoff e.g., wireless safety, security and cost trade-offs
I-04	Time synchronisation support From a plant-floor personnel perspective
I-05	Cost of complexity e.g., one or two networks for safety and control?
I-06	Network handling of power and topology
I-07	Fault handling

Table 2a: Top Industrial Ethernet Performance Issues (resulting from 2006 Industrial Ethernet Performance Workshop9)

Research Activity	Performance Issue
	1 - 0 1	1 - 0 2	1 - 0 3	1 - 0 4	1 - 0 5	1 - 0 6	1 - 0 7
Enterprise industrial networking solutions ERC core activity with Reconfigurable Factory Testbed Control, Diagnostics and Safety multi-tier networks implementation Software delays and delay variability	x	-	-		-	-	-
Industrial Ethernet technology evaluation Ethernet fieldbus best practices and comparative analysis Technology tradeoffs, and performance analysis	x	x				-	-
Industrial network time synchronisation IEEE 1588, approaches, benefits, current issues with implementation Overhead for control and diagnostics protocols on top of Ethernet (e.g., UDP, VPN and OPC) Factory simulation of network traffic; analysis of impact of time synchronisation	x		-	x
Network (reconfiguration) cost calculator Cost tradeoffs of network decision (hardware / install, engineering / maintenance and performance) Network partitioning for safety, control and diagnostics Tradeoffs of putting safety and control on a dedicated versus shared network			x		x

Table 2b: ERC-RMS Key Industrial Ethernet Research Activities and Mapping to Top Industrial Ethernet Performance Issues ('X' with green shading indicates Performance issue is part of core activity;'-' with yellow shading indicates aspects of issue are addressed with activity.) An overview of results from the underlined activities is provided in the next section.

Selected performance evaluation results
Software delays and delay variability - Building on earlier results of performance analysis of DeviceNet, which indicated the significant contribution of device delay to overall end-to-end performance¹⁰, the ERC research team constructed the simple test bed, depicted in Fig. 3, to investigate the relative contributions of network, switch and node delay in a typical Ethernet end-to-end UDP communication. (User Datagram Protocol is an unacknowledged form of Ethernet communication that is often used for high speed applications such as streaming video.) The results, also shown here, illustrate that end-to-end delay and delay jitter (not shown) are dominated by node delay. Thus node delay and jitter should be evaluated and incorporated into the design of any high speed or time-critical Ethernet system.

One key advantage of Ethernet in manufacturing applications is the leveraging of Ethernet application layer protocols developed to support features such as security, standardised data reporting and web-services communication (XML). Unfortunately, these applications create overhead affecting performance and, since the typical commercial domain of applicability of some of these applications is not manufacturing, there is little financial pressure to optimise them for manufacturing systems performance.

Figures 4a and 4b provide a good indication of the delay and delay variability introduced by two common application layer Ethernet protocols, namely OPC (OLE for Process Control) for standardised data reporting, and VPN (Virtual Private Network) for security¹⁰. Observing UDP Ethernet communications as a baseline, it is clear that both VPN and - especially - OPC add significant delay and delay jitter to the end-to-end communication. Thus careful consideration should be given to the performance of application level components of the Ethernet 'stack' when designing for performance.

Fig. 3: Contributions to end-to-end delay in an Industrial Ethernet system. Delays in microseconds

UDP round-trip delays (100Mb/s switched network) Mean = 0.33ms, max = 1.89ms Stdev = 0.03ms Network round-trip time: 0.035ms	OPC round-trip delays (100Mb/s switched network) Mean = 1.5ms, max = 16.8ms Stdev = 0.81ms
UDP round-trip delays (100Mb/s switched network) Slope = 0.4 µs/bit - Theory: 0.32 µs/bit Intercept: 0.285ms	VPN round-trip delays (100Mb/s switched network) Slope = 0.4 µs/bit Intercept: 1.07ms Encryption, no compression

The plots of Fig. 4b (two bottom plots) illustrate the overhead due to VPN as a function of packet size (right) as compared to a UDP baseline (left). The round-trip times were measured for a number of messages at various packet sizes. The results show that VPN adds slightly under 1ms of round trip time to the minimum size packet. The increase in delay as a function of data size (i.e., the slope) here is similar to that of normal UDP communications but can increase depending on the VPN encryption technique used.

* [Many Industrial Ethernet protocols are available, with Modbus/TCP the most widely used2. However in this instance the manufacturer was interested in comparing these two industrial Ethernet protocols.]

Fig. 5: Round-trip timing measurements for Profinet (5a) and EtherNet/IP (5b). Here the round trip times for packets between two PLCs are plotted for a large number of packets so as to obtain a time distribution. The tables represent the consolidation of a number of these graphs where the testing environment is modified in terms of switches between sender and receiver, data size transmitted, and network loading (the plots shown represent the baseline case). Bimodal performance behaviour of EtherNet/IP communications is thought to be due to lack of synchronisation between components.11, 12

Fig. 5a Profinet
Fig. 5b EtherNet/IP

Best practices and comparative analysis
In an effort to understand the tradeoffs between Industrial Ethernet protocols, two common protocols, EtherNet/IP and Profinet, were compared in the areas of architecture, technology, performance, ease of use, diagnostics capabilities and network management*. As part of this effort parallel multi-layer switched Ethernet test beds were developed using each of these technologies, where the network layout was representative of the structure being used at a leading automotive manufacturer. The results indicate that both protocols and protocol devices are fairly similar and are adequate to the task of providing industrial networking capabilities at the PLC level and higher^11,12. However distinct differences were observed, such as those illustrated in Fig. 5, indicating additional improvements in device performance may be needed if the solution is to be deployed down to the I/O level.

Cost tradeoffs in network design
There are a number of tradeoffs involved when considering conversion to Industrial Ethernet, and in making decisions about reconfiguring existing Ethernet systems. Ultimately the impact of such decisions is financial, whether directly in terms of component, installation and maintenance costs, or indirectly in terms of lost performance costs. An investigation was conducted into these cost tradeoffs to see if they could be quantified and the results consolidated. In short, we wanted to know how to arrive at the best decisions for Ethernet deployment. The result was the development of an extensible network Return-On-Investment cost calculator13,5. The calculator allows cost components to be weighted according to the needs of the application environment and then consolidated in a normalised fashion so that technical tradeoffs can be evaluated as cost decisions. Results from the calculator indicate that networking decisions are highly application dependant and often counter-intuitive.

For example, Fig. 6 illustrates the decision process for determining if a single shared network or two separate dedicated networks should be used to support safety and control functionality in a system. Figure 6a illustrates an application environment where hardware and maintenance costs are weighted heavily, pointing to a shared network as the best choice, while Figure 6b illustrates the same application environment with performance costs dominate, pointing to dedicated separate networks as the optimal solution.

Fig. 6: Analysis of shared versus dedicated networks for safety and control networking where hardware/maintenance costs dominate (6a) and performance costs dominate (6b).

Summary
As the move to Industrial Ethernet continues on the manufacturing floor, a key issue of concern is end-to-end performance. In evaluating this performance it is important to consider not only network and network component delay, but also node software delay and delay variability or jitter. The impact of the Ethernet application protocol overhead on support capabilities such as security (VPN) and standardised diagnostics (OPC) cannot be overlooked; these application protocols will continue to be an issue because the performance metrics to which the protocols are being optimised do not necessarily align with the needs of factory floor usage. Our evaluation of two common Industrial Ethernet protocols reveals that they are indeed better suited to industrial use than traditional Ethernet and perform satisfactorily in a typical manufacturing control scenario. However, performance improvements are needed, especially in the area of component time synchronisation and speed/jitter. The development and application of a network cost calculator indicates that the best design and reconfiguration choices (such single or multiple networks to support control and safety signals) are very application specific and often counter intuitive. The process of moving to Industrial Ethernet should be well-thought out and planned, paying attention to the specific performance requirements of the end-application.

Despite these reservations it appears that Ethernet can provide network solutions for diagnostics, control and safety in manufacturing if the planning processes outlined above are followed. The economics of these solutions will depend largely on the economy of scale that growing use of Industrial Ethernet will produce.

Looking ahead, one important consideration, from both a cost and performance perspective, is the emergence of wireless on the factory floor. Wireless brings new issues of performance (collisions, unreliable communication, security, etc.) not seen with wired switched Ethernet. Despite this, we believe that the enormous potential benefits of cost reduction and re-configurability provided by wireless will push the technology into all of the domains currently being considered for wired Industrial Ethernet. Thus while there will always be an application need for wired Ethernet, it is believed that wired Ethernet's share of the manufacturing networking market will first increase as it supplants non-Ethernet wired protocols, but then decrease as it is supplanted inturn by wireless technology.

Acknowledgements
The authors would like to thank the students who did much of the work on which this paper is based, especially J. Parrott, N. Kalappa, B. Triden, F.L. Lian, A. Thomas, M. Antolovic, and K. Acton. This work was supported in-part by the Engineering Research Centre for Reconfigurable Manufacturing Systems of the National Science Foundation under Award Number EEC-9529125.

James Moyne Ph.D is currently an Associate Research Scientist in the Department of Mechanical Engineering, and director of the Reconfigurable Factory Testbed at the University of Michigan. He is also Director of Advanced Process Control Technology at Applied Materials.

Dawn M.Tilbury Ph.D is with the faculty of the Mechanical Engineering Department at the University of Michigan, Ann Arbor, where she is currently an Associate Professor. Her research interests include distributed control of mechanical systems with network communication, logic control of manufacturing systems, performance management and control of computing systems, and uncertainty modelling in cooperative control.

References
[1] Jeanine Katzel and Mark T. Hoske,'Product Focus: Industrial Ethernet Protocols,' Control Engineering Magazine, (Dec 2006), available at http://www.controleng.com/article/CA6396565.html?industry=Process%2BControl&industryid=22072&spa cedesc=communityFeatures&
[2] Lars Larsson,'Fourteen Industrial Ethernet Solutions Under the Spotlight, The Industrial Ethernet Book, Issue 37 available at http://www.iebmedia.com/articles/articledisplay.asp?id=854
[3] James Moyne and Dawn Tilbury,'The Emergence of Industrial Control Networks for Manufacturing Control, Diagnostics and Safety Data,' IEEE Proceedings, 95(1): pp. 29-47; Special Issue on Technology of Networked Control Systems (Jan 2007).
[4] P. G. Otanez, J.T. Parrott, J. R.Moyne, and D. M.Tilbury,'The Implications of Ethernet as a Control Network,' in Proceedings of the Global Powertrain Congress,Ann Arbor, Sept (2002).
[5] http://erc.engin.umich.edu
[6] F.-L. Lian, J. R.Moyne, and D. M.Tilbury,'Performance Evaluation of Control Networks: Ethernet, ControlNet, and DeviceNet,' IEEE Control Systems Magazine, 21(1), pp. 66-83, (Feb 2001).
[7] F.-L. Lian, J. R.Moyne, and D. M.Tilbury,'Network Design Consideration for Distributed Control Systems,' IEEE Transactions on Control Systems Technology, 10(2), pp. 297-307, (Mar 2002).
[8] J.Moyne, J.Korsakas, and D.Tilbury,'Reconfigurable Factory Test bed (RFT):A Distributed Test bed for Reconfigurable Manufacturing Systems,' Japan-USA Symposium on Flexible Automation, Denver,CO, (July 2004).
[9] Acton, K.,Antolovic,M., Kalappa, N., Luntz, J.,Moyne, J.,Tilbury,D., 2006,'Practical Metrics for Evaluating Network System Performance,' UM-ERC/RMS Network Performance Workshop,Ann Arbor, Michigan, (April 2006). Report #: ERCRMS-TR-059-2006, available at