Performance metrics for Industrial Ethernet
Industrial Ethernet * is appealing for mission-critical use such as
control and safety. The application context demands a fresh look at the performance
issues involved. The University of Michigan's Engineering Research Centre for
Reconfigurable Manufacturing Systems has been exploring the issue with a reconfigurable
factory test bed using fully networked control, diagnostics and safety capabilities.
Work has included a comparative evaluation of two common Industrial Ethernet
protocols, and development of a cost-based methodology and calculator for evaluating
network implementation and re-configuration choice. It has also shed new light
on the link between software protocol and network determinism.
By James Moyne and Dawn Tilbury
As Industrial Ethernet becomes more prevalent on the manufacturing
floor it is increasingly being considered as a universal
networking solution1,2. Its dominance in the 'value-add' diagnostics
domain is now spreading to the mission critical domains of
control and even safety networking3. While the capability of Ethernet
has been proven acceptable in the office and high level manufacturing
environments, issues of performance must be considered as we push
this technology into areas such as I/O, motion control and safety4.
The University of Michigan's Engineering Research Centre for Reconfigurable
Manufacturing Systems (ERC-RMS) has been exploring the applicability of Ethernet
to the factory floor, focusing on application domains of diagnostics, control
and safety5. Key results of these efforts are summarised in this
paper. Specifically, after describing the issue of network performance and providing
background information on the ERC-RMS effort, industrially relevant Ethernet
performance metrics are presented, followed by results of Ethernet performance
evaluation against some of these metrics. This paper concludes with a summary
of key points.
* [To the author's knowledge there is no standardised definition of Industrial
Ethernet. In this paper we use the term to mean Ethernet protocols that are
applied in a manufacturing setting. Thus for example it could refer to specific
solutions such as Modbus/TCP or just the application of 'traditional' Ethernet
to manufacturing data collection and control.]
The advantages of industrial networked systems over their 'hard-wired'
point-to-point predecessors are well documented and include decreased
wiring and maintenance costs, expanded flexibility of control architectures
and diagnostics capabilities, and increased reliability,
re-configurability and interchangeability3. However these advantages
come at a cost of potential end-to-end performance reduction that
results from the fact that many nodes are sharing the communication
medium, and individual nodes must support added software communication
protocol layers for the encoding and decoding of network
messaging. This performance loss is illustrated with the network performance
curve, termed the 'Lian Curve', shown in Fig. 1. Here we see
that digital control performance improves as sample time is decreased
(i.e., sample rate is increased). However when this digital control is
networked, the performance begins to worsen if the sampling rate is
increased beyond an optimal point. This is because network and/or
node congestion begins to introduce delays and delay variability. A
performance goal in any network application is to determine the ideal
operating region 'sweet spot' and stay in, or just to the left of, that sweet
spot as the network is reconfigured (e.g., as nodes are added).
For the most part all time-based network performance issues can be
traced back, at least in-part, to this Lian Curve phenomenon. Additional
information on the Lian Curve can be found in references 6 and 7.
The ERC-RMS was established by the National Science Foundation
in 1996 to pursue advancements in reconfigurable manufacturing5.
Today the ERC has a broad membership base of automotive manufacturers
and suppliers. It focuses on providing solutions that will allow
manufacturers and suppliers to respond rapidly by 'reconfiguring' to
changes in demand, product, operations, and factory floor dynamics
such as system failures. A key research focus of the ERC relates to
networks and network performance because these networks, if properly
implemented, can be important tools to facilitate re-configurability.
Fig. 1: Lian Curve: Illustrating Networked Control System performance as
a function of sampling time
A key deliverable result of this work has been the reconfigurable factory
test bed RFT, illustrated in Fig. 2a, which is used to demonstrate research
results, investigate the collaboration of these results, and serve as an education
and technology transfer mechanism8. With respect to the latter, the
RFT uses industrial quality solutions wherever possible so that a focus can
be placed on the optimal utilisation of these tools. Along those lines, as shown
in Figures 2b and 2c, the RFT employs common industrial network technologies
of DeviceNet and Profibus, OPC (over switched Ethernet), and SafetyBUS, for
networked capabilities of control, diagnostics and safety respectively.
Fig. 2a: ERC-RMS Reconfigurable Factory Test bed illustrating the overall
Figures 2b (top) and 2c: ERC-RMS Reconfigurable Factory Test bed: control,
diagnostics and safety functionalities
The need for Ethernet performance metrics is driven by the desire to
expand Ethernet throughout the factory, across domains of control,
diagnostics and safety, and all the way down to the I/O level.
Considering common questions that are asked about Industrial
Ethernet when moving into new domains provides insight to help derive
and prioritise these metrics. Typical questions include:
- Can I use Ethernet down to the I/O level?
- Should I partition my networks at different levels?
- Should I put safety, control and diagnostics on one, two or three networks?
- Where is the delay and delay variability occurring?
- What are the industry de facto standards?
- What is the cost-tradeoff of a decision?
- What is the complexity-tradeoff of a decision?
- What is the performance cost of security (e.g., VPN) or common Industrial
Ethernet application protocols (e.g., OPC)?
- What are the differences between the Industrial Ethernet technology varieties?
Which differences are important?
The ERC-RMS conducted a network performance workshop in 2006
to map and resolve these types of question into a ranking of issues that
should be considered when applying Industrial Ethernet9. The output
of that workshop, which reflects contributions from network technology
providers, users and researchers is summarised in Table 2a.
As noted earlier, the ERC-RMS has been conducting a number of
research activities focused on evaluating Industrial Ethernet performance.
These activities are itemised in Table 2b along with a mapping
to the performance issues identified in the 2006 workshop; this mapping
illustrates that the ERC-RMS research focus is well-aligned with current
issues facing the Industrial Ethernet community.
In the next section of this paper, key results from each of these research
activities (underlined in Table 2b) are summarised. Further information on all
topics identified in Table 2b can be found in the literature referenced at the
end of this paper.
is most important performance consideration
on the network is key
||Ease of use and
From a plant-floor
||Cost versus security
safety, security and cost trade-offs
From a plant-floor
||Cost of complexity
e.g., one or
two networks for safety and control?
|| Network handling
of power and topology
Table 2a: Top Industrial Ethernet Performance Issues (resulting from 2006
Industrial Ethernet Performance Workshop9)
Table 2b: ERC-RMS Key Industrial Ethernet Research Activities and Mapping
to Top Industrial Ethernet Performance Issues ('X' with green shading indicates
Performance issue is part of core activity;'-' with yellow shading indicates
aspects of issue are addressed with activity.) An overview of results from the
underlined activities is provided in the next section.
Software delays and delay variability - Building on earlier results of
performance analysis of DeviceNet, which indicated the significant contribution
of device delay to overall end-to-end performance10, the ERC research
team constructed the simple test bed, depicted in Fig. 3, to investigate the
relative contributions of network, switch and node delay in a typical Ethernet
end-to-end UDP communication. (User Datagram Protocol is an unacknowledged form
of Ethernet communication that is often used for high speed applications such
as streaming video.) The results, also shown here, illustrate that end-to-end
delay and delay jitter (not shown) are dominated by node delay. Thus node delay
and jitter should be evaluated and incorporated into the design of any high
speed or time-critical Ethernet system.
One key advantage of Ethernet in manufacturing applications is the
leveraging of Ethernet application layer protocols developed to support
features such as security, standardised data reporting and web-services
communication (XML). Unfortunately, these applications create
overhead affecting performance and, since the typical commercial
domain of applicability of some of these applications is not manufacturing,
there is little financial pressure to optimise them for
manufacturing systems performance.
Figures 4a and 4b provide a good indication of the delay and delay
variability introduced by two common application layer Ethernet
protocols, namely OPC (OLE for Process Control) for standardised
data reporting, and VPN (Virtual Private Network) for security10.
Observing UDP Ethernet communications as a baseline, it is clear that
both VPN and - especially - OPC add significant delay and delay jitter
to the end-to-end communication. Thus careful consideration should
be given to the performance of application level components of the
Ethernet 'stack' when designing for performance.
Fig. 3: Contributions to end-to-end delay in an Industrial Ethernet system.
Delays in microseconds
round-trip delays (100Mb/s switched network)
Mean = 0.33ms,
max = 1.89ms
Stdev = 0.03ms
round-trip delays (100Mb/s switched network)
Mean = 1.5ms,
max = 16.8ms
Stdev = 0.81ms
round-trip delays (100Mb/s switched network)
Slope = 0.4
- Theory: 0.32 µs/bit
round-trip delays (100Mb/s switched network)
Slope = 0.4
Fig. 4: Delay and delay variability - the impact of Ethernet
application layer protocols on performance. The experimental setup of Fig. 3 was
used and data was collected on the round-trip time for messages. Figure 4a (top
two charts) are plots of the round-trip delays recorded for each message (index).
4a illustrates the impact of OPC (right) compared to baseline Ethernet UDP (left)
end-to-end performance (note scales on graphs are different).This shows that OPC
adds over 1ms to the delay (a greater than 300% increase) as well as over 0.7ms
to the standard deviation.
The plots of Fig. 4b (two bottom plots) illustrate the overhead due to VPN
as a function of packet size (right) as compared to a UDP baseline (left). The
round-trip times were measured for a number of messages at various packet sizes.
The results show that VPN adds slightly under 1ms of round trip time to the
minimum size packet. The increase in delay as a function of data size (i.e.,
the slope) here is similar to that of normal UDP communications but can increase
depending on the VPN encryption technique used.
* [Many Industrial Ethernet protocols are available, with Modbus/TCP the most widely
used2. However in this instance the manufacturer was interested in comparing these two
industrial Ethernet protocols.]
Fig. 5: Round-trip timing measurements for Profinet (5a) and EtherNet/IP (5b).
Here the round trip times for packets between two PLCs are plotted for a large
number of packets so as to obtain a time distribution. The tables represent
the consolidation of a number of these graphs where the testing environment
is modified in terms of switches between sender and receiver, data size transmitted,
and network loading (the plots shown represent the baseline case). Bimodal performance
behaviour of EtherNet/IP communications is thought to be due to lack of synchronisation
between components.11, 12
Fig. 5a Profinet
Fig. 5b EtherNet/IP
In an effort to understand the tradeoffs between Industrial Ethernet
protocols, two common protocols, EtherNet/IP and Profinet, were
compared in the areas of architecture, technology, performance, ease
of use, diagnostics capabilities and network management*. As part of
this effort parallel multi-layer switched Ethernet test beds were
developed using each of these technologies, where the network layout
was representative of the structure being used at a leading automotive
manufacturer. The results indicate that both protocols and protocol
devices are fairly similar and are adequate to the task of providing
industrial networking capabilities at the PLC level and higher11,12.
However distinct differences were observed, such as those illustrated
in Fig. 5, indicating additional improvements in device performance
may be needed if the solution is to be deployed down to the I/O level.
There are a number of tradeoffs involved when considering conversion
to Industrial Ethernet, and in making decisions about reconfiguring
existing Ethernet systems. Ultimately the impact of such decisions is
financial, whether directly in terms of component, installation and maintenance
costs, or indirectly in terms of lost performance costs. An
investigation was conducted into these cost tradeoffs to see if they could
be quantified and the results consolidated. In short, we wanted to know
how to arrive at the best decisions for Ethernet deployment. The result
was the development of an extensible network Return-On-Investment
cost calculator13,5. The calculator allows cost components to be weighted
according to the needs of the application environment and then consolidated
in a normalised fashion so that technical tradeoffs can be evaluated
as cost decisions. Results from the calculator indicate that networking
decisions are highly application dependant and often counter-intuitive.
For example, Fig. 6 illustrates the decision process for determining
if a single shared network or two separate dedicated networks should
be used to support safety and control functionality in a system. Figure
6a illustrates an application environment where hardware and maintenance
costs are weighted heavily, pointing to a shared network as the
best choice, while Figure 6b illustrates the same application environment
with performance costs dominate, pointing to dedicated separate
networks as the optimal solution.
Fig. 6: Analysis of shared versus dedicated networks for safety and control
networking where hardware/maintenance costs dominate (6a) and performance costs
As the move to Industrial Ethernet continues on the manufacturing
floor, a key issue of concern is end-to-end performance. In evaluating
this performance it is important to consider not only network and
network component delay, but also node software delay and delay variability
or jitter. The impact of the Ethernet application protocol
overhead on support capabilities such as security (VPN) and standardised
diagnostics (OPC) cannot be overlooked; these application
protocols will continue to be an issue because the performance metrics
to which the protocols are being optimised do not necessarily align with
the needs of factory floor usage. Our evaluation of two common
Industrial Ethernet protocols reveals that they are indeed better suited
to industrial use than traditional Ethernet and perform satisfactorily
in a typical manufacturing control scenario. However, performance
improvements are needed, especially in the area of component time
synchronisation and speed/jitter. The development and application
of a network cost calculator indicates that the best design and reconfiguration
choices (such single or multiple networks to support control
and safety signals) are very application specific and often counter
intuitive. The process of moving to Industrial Ethernet should be well-thought
out and planned, paying attention to the specific performance
requirements of the end-application.
Despite these reservations it appears that Ethernet can provide
network solutions for diagnostics, control and safety in manufacturing
if the planning processes outlined above are followed. The economics
of these solutions will depend largely on the economy of scale that
growing use of Industrial Ethernet will produce.
Looking ahead, one important consideration, from both a cost and
performance perspective, is the emergence of wireless on the factory
floor. Wireless brings new issues of performance (collisions, unreliable
communication, security, etc.) not seen with wired switched Ethernet.
Despite this, we believe that the enormous potential benefits of cost
reduction and re-configurability provided by wireless will push the
technology into all of the domains currently being considered for wired
Industrial Ethernet. Thus while there will always be an application need
for wired Ethernet, it is believed that wired Ethernet's share of the manufacturing
networking market will first increase as it supplants
non-Ethernet wired protocols, but then decrease as it is supplanted inturn
by wireless technology.
The authors would like to thank the students who did much of the work
on which this paper is based, especially J. Parrott, N. Kalappa, B.
Triden, F.L. Lian, A. Thomas, M. Antolovic, and K. Acton. This work
was supported in-part by the Engineering Research Centre for
Reconfigurable Manufacturing Systems of the National Science
Foundation under Award Number EEC-9529125.
James Moyne Ph.D is currently an Associate Research Scientist in the Department
of Mechanical Engineering, and director of the Reconfigurable Factory Testbed
at the University of Michigan. He is also Director of Advanced Process Control
Technology at Applied Materials.
Dawn M.Tilbury Ph.D is with the faculty of the Mechanical Engineering Department
at the University of Michigan, Ann Arbor, where she is currently an Associate
Professor. Her research interests include distributed control of mechanical
systems with network communication, logic control of manufacturing systems,
performance management and control of computing systems, and uncertainty modelling
in cooperative control.
 Jeanine Katzel and Mark T. Hoske,'Product Focus: Industrial Ethernet Protocols,'
Control Engineering Magazine, (Dec 2006), available at http://www.controleng.com/article/CA6396565.html?industry=Process%2BControl&industryid=22072&spa
 Lars Larsson,'Fourteen Industrial Ethernet Solutions Under the Spotlight, The Industrial Ethernet
Book, Issue 37 available at http://www.iebmedia.com/articles/articledisplay.asp?id=854
 James Moyne and Dawn Tilbury,'The Emergence of Industrial Control Networks for Manufacturing
Control, Diagnostics and Safety Data,' IEEE Proceedings, 95(1): pp. 29-47; Special Issue on Technology of
Networked Control Systems (Jan 2007).
 P. G. Otanez, J.T. Parrott, J. R.Moyne, and D. M.Tilbury,'The Implications of Ethernet as a Control
Network,' in Proceedings of the Global Powertrain Congress,Ann Arbor, Sept (2002).
 F.-L. Lian, J. R.Moyne, and D. M.Tilbury,'Performance Evaluation of Control Networks: Ethernet,
ControlNet, and DeviceNet,' IEEE Control Systems Magazine, 21(1), pp. 66-83, (Feb 2001).
 F.-L. Lian, J. R.Moyne, and D. M.Tilbury,'Network Design Consideration for Distributed Control
Systems,' IEEE Transactions on Control Systems Technology, 10(2), pp. 297-307, (Mar 2002).
 J.Moyne, J.Korsakas, and D.Tilbury,'Reconfigurable Factory Test bed (RFT):A Distributed Test bed
for Reconfigurable Manufacturing Systems,' Japan-USA Symposium on Flexible Automation, Denver,CO, (July
 Acton, K.,Antolovic,M., Kalappa, N., Luntz, J.,Moyne, J.,Tilbury,D., 2006,'Practical
Metrics for Evaluating Network System Performance,' UM-ERC/RMS Network Performance
Workshop,Ann Arbor, Michigan, (April 2006). Report #: ERCRMS-TR-059-2006, available