Beckhoff: High-speed Ethernet from the IPC to the drives
Industrial Ethernet Book Issue 100 / 6
Request Further Info   Print this Page   Send to a Friend  

Building IoT-ready manufacturing networks. Part 1:

Building IoT-ready networks that enables data logging and acquisition, remote monitoring and cloud-based solutions requires careful planning and execution. Converged architectures that incorporate best practices and technology are delivering resiliency, wireless access, defense-in-depth security and cloud connectivity.

MANUFACTURING NETWORKS ARE EVOLVING towards a converged architecture with integrated IT and OT systems. Connecting devices and assets in a converged architecture enables data acquisition, remote monitoring and cloud based predictive analytics. This leads to lower costs, higher productivity, and greater visibility into plant operations.

However, the converged network design must ensure that security and performance requirements are not compromised as a result. This article will provide an overview of various use cases, best practices and architecture recommendations in the areas of resiliency, wireless, defense-in-depth security as well as cloud connectivity and integration.

By utilizing a resilient, scalable and secure network, business leaders can achieve various business outcomes which includes improving efficiency, minimizing downtime, increase OEE and enable real time operational visibility.

Zoning through segmentation.

Reference architecture

Modern industrial automation and control system (IACS) applications require a network infrastructure to be scalable, reliable, safe, secure and future-ready to support the industrial internet of things (IIoT). A structured and hardened architecture, with key tenets, helps create smaller connected LANs which restores the segmentation and natural boundaries found in legacy 3-tier networks. Key tenets include smart endpoints, segmentation (zoning), managed infrastructure, resiliency, time-critical data, wireless - mobility, holistic Defense-in-Depth security and convergence-ready solutions.

To understand the security and network systems requirements of an industrial automation and control system (IACS), this article uses a logical model to describe the basic functions and composition of an IACS application. The Purdue Reference Architecture and ISA 95 are common and well-understood models that use Levels to segment devices and equipment into hierarchical functions. The concept of operational Levels has been incorporated into many other models and standards in the industry. Based on this segmentation of the plant operations, standards such as IEC 62443, NIST 800-82 and ICS-CERT recommended practices organize the operational Levels into Security Zones.

This logical model identifies Levels of operations which are organized into Zones based on functionality and domains of trust. The Open Systems Interconnection (OSI) and Reference Model is also commonly referred to in discussing architectures. The OSI model refers to Layers of network communication functions. Unless specified, Layers refer to layers of the OSI model and Levels refer to operational levels of this logical model.

Cell/Area zone

The Cell/Area zone is a functional area within a plant facility; many plants have multiple Cell/Area zones. In an automotive plant, it may be a bodyshop or a sub-assembly process. In a food and beverage facility, it may be the batch mixing area. It may be as small as a single controller and its associated devices on a process skid, or multiple controllers on an assembly line.

Each plant facility defines the Cell/Area zone demarcation differently and to varying degrees of granularity. For the purposes of this paper, a Cell/Area zone is a set of IACS devices, controllers, etc. that are involved in the real-time control of a functional aspect of the manufacturing process. To control the functional process, they are all in real-time communication with each other. This zone has essentially three levels of activity occurring, as described in the following subsections.

Level 0: Process Level 0 consists of a wide variety of sensors and actuators involved in the basic manufacturing process. These devices perform the basic functions of the IACS, such as driving a motor, measuring variables, setting an output, and performing key functions such as painting, and so on. These functions can be very simple (temperature gauge) to highly complex (a moving robot). These devices take direction from and communicate status to the control devices in Level 1 of the logical model. In addition, other IACS devices or applications may need to directly access Level 0 devices to perform maintenance or resolve problems on the devices.

  • Drive the real-time, deterministic communication requirements
  • Measure the process variables and control process outputs
  • Exist in challenging environments that drive topology constraints
  • Vary according to the size of the IACS network from a small (10s) to a large (1000s) number of devices
  • Once designed and installed, are not replaced all together until the plant line is overhauled or replaced, which is typically ten or more years

Control System Engineers (operational technology - OT) such as electrical, process, and so on, and not the IT departments, typically design and implement these devices and the IACS networks that support them.

Level 1: Basic Control Level 1 consists of controllers that direct and manipulate the manufacturing process, which its key function is to interface with the Level 0 devices (e.g., I/O, sensors, and actuators). These applications are typically implemented and maintained by the OT organization.

Controllers act alone or in conjunction with other controllers to manage the devices and thereby the manufacturing process. Controllers also communicate with other functions in the IACS (for example, historian, asset manager, and manufacturing execution system) in Levels 2 and 3. The controller performs as a director function in the Industrial zone translating high-level parameters (for example, recipes) into executable orders, consolidating the I/O traffic from devices and passing the I/O data on to the upper-level plant floor functions.

Controllers produce IACS network traffic in three directions from a Level perspective:

  • Downward to the devices in Level 0 that they control and manage
  • Peer-to-peer to other controllers to manage the IACS for a Cell/Area zone
  • Upward to HMIs and information management systems in Levels 2 and 3

Level 2: Area Supervisory Control Level 2 represents the applications and functions associated with the Cell/Area zone runtime supervision and operation. These include operator interfaces or HMIs, alarms or alerting systems and control room workstations. Depending on the size or structure of a plant, these functions may exist at the site level (Level 3). These applications communicate with the controllers in Level 1 and interface or share data with the site level (Level 3) or enterprise (Level 4/5) systems and applications through the Industrial DMZ. These applications can be implemented on dedicated IACS vendor operator interface terminals, or on standard computing equipment and operating systems such as Microsoft Windows. These applications are typically implemented and maintained by the operational technology (OT) organization.

Cell zone components.

Industrial zone

The Industrial zone is comprised of the Cell/ Area zones (Levels 0 to 2) and site-level (Level 3) activities. The Industrial zone is important because all the IACS applications, devices, and controllers critical to monitoring and controlling the plant floor IACS operations are in this zone. To preserve smooth plant operations and functioning of the IACS applications and IACS network, this zone requires clear logical segmentation and protection from Levels 4 and 5 of the plant/ enterprise operations.

Level 3: Site Level 3, represents the highest level of the IACS. The systems and applications that exist at this level manage plantwide IACS functions. Levels 0 through 3 are considered critical to site operations. The applications and functions that exist at this level include the following:

  • Level 3 IACS network
  • Reporting
  • Plant historian
  • Detailed production scheduling
  • Site-level operations management
  • Asset and material management
  • Control room workstations
  • Patch launch server
  • File server
  • Other domain services
  • Terminal server for remote access support
  • Staging area
  • Administration and control applications

The Level 3 IACS network may communicate with Level 1 controllers and Level 0 devices, function as a staging area for changes into the Industrial zone, and share data with the enterprise (Levels 4 and 5) systems and applications through the Industrial DMZ.

These applications are primarily based on standard computing equipment and operating systems. Additionally, because these systems tend to be aligned with standard IT technologies, they may also be implemented and supported by personnel with Industrial IT (OT-IT) skill sets. These industrial personnel may belong organizationally to either OT or IT.

Enterprise Zone

Level 4: Site Business Planning and Logistics Level 4 is where the functions and systems that need standard access to services provided by the enterprise network reside. This level is viewed as an extension of the enterprise network. The basic business administration tasks are performed here and rely on standard IT services. These functions and systems include wired and wireless access to enterprise network services such as the following:

  • Access to the Internet Access to E-mail (hosted in data centers)
  • Non-critical plant systems such as manufacturing execution systems and overall plant reporting, such as inventory, performance, etc.
  • Access to enterprise applications such as SAP and Oracle (hosted in data centers)

Although important, these services are not viewed as critical to the IACS and thus the plant floor operations. Because of the more open nature of the systems and applications within the enterprise network, this level is often viewed as a source of threats and disruptions to the IACS network.

The users and systems in Level 4 often require summarized data and information from the lower levels of the IACS network. The network traffic and patterns here are typical of a network where approximately 90 percent of the network traffic goes to the Internet or to data center-based applications. This level is typically under the management and control of the IT organization.

Level 5: Enterprise Level 5 is where the centralized IT systems and functions exist. Enterprise resource management, business-to-business, and business-to-customer services typically reside at this level. Often the external partner or guest access systems exist here, although it is not uncommon to find them in lower levels (e.g., Level 3) of the model to gain flexibility that may be difficult to achieve at the enterprise level. However, this approach may lead to significant security risks if not implemented within IT security policy and approach.

The IACS must communicate with the enterprise applications to exchange manufacturing and resource data. Direct access to the IACS is typically not required. One exception to this would be remote access for management of the IACS by employees or partners such as system integrators and machine builders. Access to data and the IACS network must be managed and controlled through the Industrial DMZ to maintain the security, availability, and stability of the IACS. The services, systems, and applications at this level are directly managed and operated by the IT organization.

Plantwide Ethernet architecture

The Purdue, ISA, IEC and NIST have identified Levels of operations and key Security Zones for the IACS logical model. In addition to the Levels and Zones, an Industrial Demilitarized zone (IDMZ) between the Enterprise and Industrial zones as part of architecture.

The purpose of the IDMZ is to provide a buffer zone where data and services can be shared between the Enterprise and Industrial zones. The IDMZ is critical in maintaining availability, addressing security vulnerabilities, and abiding by regulatory compliance mandates. In addition, the IDMZ allows for segmentation of organizational control, for example, between the IT organization and manufacturing.

This segmentation allows different policies to be applied and contained. The manufacturing organization may apply security and quality-of-service (QoS) policies different from IT. The IDMZ is where the policies and organizational control can be divided.

Wired access

The Industrial Automation and Control Systems (IACS) network within the Cell/Area zone is the major building block of plant-wide architecture. This is the network that connects sensors, actuators, drives, controllers and any other IACS devices that need to communicate in real-time (I/O communication). This section outlines the key requirements and technical considerations for the Cell/Area zone and related IACS applications.

It is important to consider the Cell/Area zone as a separate entity of the Industrial zone. For most industrial applications, the Cell/Area zone is where the primary IACS activities are performed. The availability and performance requirements are most distinct in the Cell/Area zone. These requirements are different than those typically found in an IT network. In summary, the key design considerations are as follows:

Industrial Characteristics: The environmental conditions of the plant floor such as ability to withstand shock, vibration, humidity, dust, varying operating temperatures, must be taken into consideration because the equipment must be able to perform in these conditions. This drives the industrial characteristics of all the equipment, including the network infrastructure. The topology must be shaped to fit appropriately into the plant environment.

Interconnectivity and interoperability: Standardization on a single vendor′s IACS or industrial Ethernet network equipment within the Cell/Area zone may not be practical.

Real-time communications and network performance: Cell/Area IACS network must be designed to meet the latency and jitter requirements of the IACS it supports. This can impact the size of the LAN, the number of routing hops, the VLAN configuration, and a number of other network parameters. Typical communication ranges depends upon specific application requirements.

Availability: The availability of the Cell/Area zone is critical to the manufacturing process. Without a properly functioning Cell/Area IACS network, some or all of the plant operations may come to a halt. This can severely impact plant efficiency and the manufacturer′s bottom line. Availability itself is a function of equipment, infrastructure, configuration, software, etc.. For example, the network must also be able to recover from network impacting events, such as a connection break, faster than the cycle time of the IACS to avoid the system automatically shutting down. Availability impacts the network design, topology, and even the type of network infrastructure used.

Manageability: plant floor is usually not supported in the same manner as an IT network. The plant floor maintenance personnel tend not to have the same networking experience as IT. The setup and maintenance of network equipment and configuration must be simplified to meet the experience level of the plant floor maintenance personnel.

Security: OT/IT network convergence calls for evolved security policies for industrial networks which no longer remain isolated. IACS assets have become susceptible to the same security vulnerabilities (denial of service) as their enterprise counterparts. Protecting IACS assets requires a defense-in-depth security approach to assure the availability, confidentiality and integrity of IACS data.

Unmanaged versus managed: Although the cost of the network infrastructure may not represent a large proportion of the plant floor, the same cost reduction mentality is often applied as to other aspects of the manufacturing facility. Without clear understanding of the qualities of a managed, intelligent network, the additional hardware costs they represent may lead network developers to choose less intelligent solutions based purely on initial cost considerations; only later do they determine that the cheaper, unmanaged infrastructure cannot scale, perform, integrate, or be as easily maintained as an intelligent, managed network.

The Cell/Area zone is also distinct in that most of the network communication is of a local nature-one device communicating with another in the same vicinity. From a network perspective, the Cell/Area zone correlates primarily with a Layer 2, or local area network (LAN), network. In the campus design, the Cell/Area zone aligns with the access-layer and many of the recommendations and considerations are applied, albeit with a consideration for the plant floor and the IACS applications.

Below is a summary of recommendations:

  • Design small Cell/Area zones in a VLAN to better manage and shape the traffic.
  • Use managed switches offering diagnostics, segmentation, prioritization, resiliency, network address translation (NAT) and security.
  • All connections should be auto-negotiate for speed and duplex. Apply full-duplex communication to avoid collisions.
  • Use fiber Gigabit Ethernet ports for trunks/uplinks for distance, quick recovery, lower latency, and jitter.
  • Use Internet Group Management Protocol (IGMP) snooping/querier functions to control multicast traffic volume, preferably with the querier on the Layer-3 distribution switch.
  • Use resilient network topologies, ring, or redundant star
  • Understand the availability requirements of the manufacturing process and IACS to properly select, design and implement the network resiliency capabilities. The selected network resiliency may or may not meet these requirements depending on the type of IACS application. Implementer should design the IACS systems appropriately and understand the implications of a network event on the IACS applications.
  • Apply port security to Layer-2 Ethernet switch to limit use of open ports.

Cell zone components include the following:

  • Levels 0, 1, and 2 components; for example, devices, controllers, and HMIs
  • Layer-2 access switches
  • Layer-3 distribution switches or routers
  • Media to connect all of the above

Cell/Area Zone Traffic Flow

Traffic flow in a Cell/Area IACS network is largely determined by the design and implementation of the IACS. These systems produce very different traffic patterns than the client-server and Internet-based applications in the IT domain or enterprise network.

Cell/Area zone traffic flow.

For example, 80 to 90 percent of the Cell/ Area traffic is local as compared to a typical IT LAN in which perhaps less than 10 percent of the traffic is local. This is primarily driven by the cyclical I/O data being communicated on very short intervals (milliseconds) from devices to controllers and workstations/HMIs all on the same LAN or VLAN.

A network infrastructure should be designed to support the proper traffic flows. Features such as network segmentation can impact the network traffic flows and network performance.

Key considerations when designing traffic flows include the following. EtherNet/IP implementations have traditionally been unable to route multicast traffic since the time-to-live field in the IP packet is set to 1. The use of multicast for Implicit CIP I/O traffic is an application choice.

Explicit messaging data has always been unicast delivery via TCP. Devices and controllers configured for multicast delivery need to be located within the same Cell/ Area IACS network as these packets cannot be routed, meaning that any router will drop the packet before forwarding it outside of the subnet/VLAN. Devices and controllers configured for unicast delivery, Implicit I/O or Explicit messaging, do not need to be within the same Cell/Area zone as that communication is routable.

In the next issue, we′ll present the second part of this article, including how Industrial automation control system networks differ significantly from their IT counterparts in their need to support real-time communications.

Arun Siddeswaran, Manager, Solution Engineering, Cisco Systems, Gregory Wilcox, Global Business Development Manager for Rockwell Automation and Paul Didier, Global Solutions Architect for Cisco Systems.

Source: Industrial Ethernet Book Issue 100 / 6
Request Further Info    Print this Page    Send to a Friend  


Discover Siemens IWLAN

Get Social with us:

© 2010-2017 Published by IEB Media GbR · Last Update: 24.05.2017 · 26 User online · Legal Disclaimer · Contact Us