Which of the following types of computer is an oriented computer specially designed for scientific applications requiring gigantic amount of calculation?

1 Introduction

Supercomputers are large systems that are specifically designed to solve complex scientific and industrial challenges. Such applications span a wide range of computational intensive tasks, including quantum mechanics, weather forecasting, climate research, oil and gas exploration, molecular dynamics, and physical simulations, and require an amount of computing power and resources that go beyond what is available in general-purpose computer servers or workstations. Until the late 1990s, supercomputers were based on high-end, special-purpose components that underwent extensive testing. This approach was very expensive and limited the use of supercomputers in a few research centers around the world. To reduce the acquisition and operational costs, researchers started to build supercomputers out of “common-off-the-shelf” (COTS) components, such as those used in general-purpose desktop and laptop. This technology shift greatly reduced the cost of each supercomputer and allowed many research centers and universities to acquire mid- and large-scale systems.

Current supercomputers consist of a multitude of compute nodes interconnected through a high-speed network. Each compute node features one or more multicore/multithreaded processor chips, several memory modules, one/two network adapters, and, possibly, some local storage disks. More recently, accelerators, in the form of graphical vector units or field-programmable gate arrays (FPGA), have also made their way to mainstream supercomputing.

On the other hand, however, COTS components brought a new set of problems that makes HPC applications run in a harsh environment. First, COTS components are intrinsically more vulnerable than special-purpose components and, because of cost reasons, follow different production process and verification/validation paths than military or industry mission-critical embedded systems or server mainframes. This means that the individual probability that every single component may fail is higher than in other domains. Second, COTS components are not specifically design to solve scientific applications and their single-processor performance was lower than the vector processor, thus a larger number of processors are usually required to achieve the desired performance. Combining together an extremely large number of system components greatly increase the combined probability that at least one breaks, experience a soft error, or stop functioning completely or partially. Third, HPC workloads have been shown to be extremely heterogeneous and can possibly stress different parts of the supercomputers, such as memory, processors, or network. This heterogeneous behavior, in turn, increases the thermal and mechanical stress, effectively reducing the life span of each component and the entire system. The location and the facility where the supercomputer is installed play an important role in the resilience of the system. The cooling system of the machine room must be appropriate to reduce the temperature of densely-packet compute nodes with limited space for air flow. Modern supercomputers take cooling a step forward and provide liquid cooling solution embedded in the compute node racks. Moreover, recent studies confirm that supercomputer installed at higher altitudes, such as Cielo at Los Alamos National Laboratory (LANL), are more exposed to radiation and show higher soft error rates [1]. Finally, the mere size of current supercomputers makes it impossible to employ resilience solutions commonly used in other domains, both because of cost and practical reasons. For example, traditional resilience techniques, such double- or triple-module redundancy (DMR, TMR), are prohibitive in the context or large supercomputers because of the large number of compute nodes and compute nodes’ components. Moreover, additional redundant or hardening hardware might increase the overall cost considerably. Even a 10% increase in energy consumption might results in a large increase in the electricity bill if the system consumes a large amount of power.

Without specific and efficient resilience solutions, modern HPC systems would be extremely unstable, to the point that it is utopistic to assume that a production run of a scientific application will not incur in any error or will be able to terminate correctly. This chapter will review some of the most common resilience techniques adopted in the HPC domains, both application-specific and system-oriented. The chapter will also provide initial insights and research directions for future exascale systems.

12.1 Introduction

Supercomputer applications frequently produce enormous amounts of output data that must be analyzed and presented to understand the application outcome and draw conclusions on the results. This process, frequently referred to as “visualization”, can itself require supercomputing resources and is a fundamental modality of supercomputer usage.

Some of the principal reasons for visualizing data resulting from running an application on a supercomputer include debugging, exploring data, statistical hypothesis testing, and preparing presentation graphics. In some cases the output from running an application on a supercomputer will be something as simple as a single file with comma-separated values. However, it is much more likely that the output will be in a special parallel input/output (I/O) library format, like one of those mentioned in Chapter 10, to manage and coordinate the simultaneous output from multiple compute nodes to a single file.

This chapter discusses four key foundational concepts frequently needed as part of high performance computing (HPC) visualization: streamlines, isosurfaces, volume rendering through ray tracing, and mesh tessellations. Visualization is then practically explored through the use of five different visualization tools that are frequently used in the context of HPC: Gnuplot [1], Matplotlib [2], the Visualization Toolkit (VTK) library [3], ParaView [4], and VisIt [5]. Three of these tools (VTK, ParaView and VisIt) already incorporate the ability to use distributed memory parallel processing to accelerate the visualization process itself.

b High-Performance Supercomputers

Supercomputers are an indispensable technological tool for complex large-scale computing tasks and play an important role in the national economy and cutting-edge areas of science. We need to carry out independent R&D on supercomputers with higher overall performance, master high-performance computing, mass storage and low power consumption technologies, and develop large-scale application systems that utilize these technologies. We need to develop better high-performance supercomputers in a timely manner to satisfy the demand of the national economy and national defense development for high-performance, high-speed computing. We should endeavor to make our country one of the world's leaders in high-performance computing capabilities (see Fig. 15).32

Arrays of compute nodes

Supercomputers made out of large arrays of off-the-shelf CPUs started to appear in the late 1990s from companies such as Hitachi and Fujitsu, and this new trend became known as massively parallel computing. Today, all of the top supercomputers in the world use these massively parallel architectures and performance increases each year as the latest high-performance CPUs are utilized. These machines look similar to small data centers with racks and racks full of processor shelves interconnected by a high-performance switch fabric. Some of these machines have been built out of video gaming boards or graphical processing units, but today most of these machines are built using multicore CPUs including the Intel® Xeon Phi™ chip. Each year, the industry releases a list of the top 500 supercomputers in the world based on industry benchmark testing. For example, a recent leading entry in the industries Top 500 supercomputing list from China reached 33,900 trillion floating point operations per second using 3,120,000 processor cores with Intel Xeon E5 processors and Intel Xeon Phi processing chips implemented on 16,000 compute nodes. Connecting these compute nodes together is a challenge which we will discuss further in this chapter.

1.1 Supercomputer and Optical Interconnect

Supercomputers have been used for highly compute-intensive tasks such as quantum physics, weather forecasting, climate research, oil and gas exploration, molecular modeling, and physical simulations. The performance level has reached 10 petaFLOPS, or 10 quadrillion calculations per second, with the current leading edge computer, the “K computer,” installed in the Riken in Kobe, Japan. According to the TOP500 project, started in 1993 to provide a reliable basis for tracking and detecting trends in high-performance computing, supercomputers have maintained a performance improvement trend of 85–90% annually, almost doubling performance every year, as depicted in Fig. 1[1]. According to the prediction from this trend, supercomputers will be able to process a million trillion calculations per second (1 exaFLOPS) by the end of 2019, which may allow the simulation of the human brain [2].

The performance of computer systems has been enhanced through improvements in microprocessors, which were invented in the early 1970s, and advances in computer architectures based on pipeline processing and its multiplexed scheme. Before 2005, advances in microprocessor performance were attained partly due to increasing clock speed (500 MHz to 3.2 GHz) and shrinking gate width. However, performance improvement due to high clock speed has become an obstacle from the viewpoint of thermal design; thus, many-core architectures are used to accommodate the increasing demand on performance [3–5]. Note that the power consumption of supercomputers has been suppressed compared to the rapid improvement in performance, as shown in Fig. 1.

At the system level, optical interconnects were mostly applied to server-to-storage links in the early 1990s. Around 2005, they replaced the longer part of the electrical interconnects with rack-to-rack cluster links (IBM ASCI Purple System). In addition, in the beginning of 2010, the IBM Power 775 used a fiber cable optical backplane within the rack, as well as the rack-to-rack cluster fabric. In this case, optical modules in the form of a multi-chip module (MCM) were located on the same first level package as the router chip. The MCM contained 28 transmit and 28 receive modules, each of which had 12 channels running at 10 Gb/s per channel [6]. The adoption of optical interconnects depends on network topology, which is related to the latency or the time delay during data transmission. The IBM Power 775 adopted a two-stage all-to-all network (dragonfly) for providing low-latency, high-bandwidth connectivity random nodes in the system [6]. A 2.4 petaFLOPS supercomputer, Tsubame 2.0 of the Tokyo Institute of Technology, uses a full-bisection fat-tree network, which also has an advantage in terms of low-latency for numerical simulations. Dual-rail wideband active optical cables (QDR InfiniBand, 40 Gb/s) are used to connect compute nodes-to-nodes and nodes-to-storages to efficiently operate compute nodes. Tsubame 2.0 was recognized as the greenest production Supercomputer in the world in 2010 [7, 8].

Another typical network topology is the torus network, which consists of “nearest neighbor” interconnects, and has the advantage in terms of reducing the length of interconnects and allowing the use of conventional electrical interconnects, as has been introduced in the “K computer.” For nodes connected to further away nodes, multiple hops were necessary, resulting in greater latency. In future exascale computers, an enormous number of processor cores will be used in the order of ten million (107 ) to thousand million (109), and half of the power consumption might be shared by the network and memory [9]. To develop such a computer system, the number of cores per node, and that of nodes might be in the order of 1–10 thousand and 10–100 thousand, respectively [10]. Thus, the power reduction of the network is a major challenge not only among nodes but also within nodes, which should be accommodating with allowable latencies. This is the opportunity for optical interconnects to be widely used in future supercomputers.

Work with paying customers

SDSC, the financial management arm of the Chronopolis team, worked for several months to develop a recharge model. This model, based on a per terabyte per year charge, was designed to offset the costs for the basic maintenance of the Chronopolis infrastructure. This financial model has many components, including:

FTE necessary at the three Chronopolis partner sites;

costs to continue maintenance of current hardware infrastructure as well as costs to upgrade it on a reasonable time schedule;

costs to cover adding infrastructure in a timely manner if the capacity of the system demands it;

costs if necessary are added for ‘overhead’, which is required in the university setting.

6.9 Emerging Challenges: Petascale and Exascale Computing

Supercomputer systems are today created by putting together a very large number of nodes. This trend is likely to continue. We can envisage systems consisting of hundreds of thousands of compute nodes, each node consisting of hundreds of processing cores along with associated memory and I/O support. For example, one recent study evaluated a system consisting of 204,800 nodes, each node having 768 cores.

With an increase in the number of nodes, there comes an increase in the failure rate. The supercomputers of today typically have a mean time between failure of less than 40 hours. Large-scale and long-lived workloads, involving thousands of nodes and running for hundreds of hours or more, require checkpointing to make meaningful progress in such an environment. However, such workloads also generate large-sized checkpoints, with significant time and energy overheads. For example, one recent study (on the French Grid'5000 distributed computational platform) found that checkpointing on a node cost between 2520 and 3570 Joules per GByte (note the wide variation in energy consumption per node despite all the nodes being supposedly similar to one another). Checkpoints can be enormous in many applications that run on such systems. To cite two extreme cases, a checkpoint size of 160 TBytes has been quoted for the astrophysics application CHIMERA (used to simulate the core collapse of a supernova) and 20 TBytes in GTC (used in the study of plasmas for nuclear fusion). Storing such checkpoints can take considerable time; techniques to reduce such overheads are urgently required.

It is likely that a combination of approaches will be needed to reduce checkpointing overheads in such systems to manageable levels. First, faster nonvolatile memory is becoming available. Traditional disks are being replaced by flash drives, which allow for much faster access. The development of memristor and phase-change memories is proceeding apace. Second, as much checkpointing activity as possible can be overlapped with regular processing. Incremental checkpointing can be used, where just the information changed since the previous checkpoint is stored. Compiler analysis of data structures could identify “dead” data, which will never again be accessed by the workload: clearly, such data need not be included in the checkpoint. Furthermore, as we have already mentioned in this chapter, data can be compressed before being stored as a checkpoint; obviously compression will incur its own overhead. To reduce intercore synchronization overhead, coordinated checkpointing could be considered within individual sets of tightly coupled nodes, whereas message logging could be used for messages between loosely coupled nodes. Finally, some researchers have suggested a combined application of redundant calculation (using one or more shadow nodes to duplicate the activity of each “primary” node) and checkpointing to reduce the pressure on the checkpointed system.

Experimental results from actual supercomputer facilities will drive the optimal checkpointing policy, as some common assumptions may be observed to be untrue. We already noted above the wide disparity in energy consumption for taking checkpoints between supposedly similar nodes. To take another example, workers at the Oak Ridge National Laboratory recently reported that failures on their supercomputer were temporally correlated. Failures occurred not randomly over time, but bunched: one failure was soon followed by another. By the same token, a long period without a failure was likely to indicate that failure was less likely to occur in the near future. An obvious response to this is to lengthen the intercheckpoint interval if a failure has not been observed for some considerable time.

9.1 Introduction

Modern supercomputers employ several different modalities of operation to take advantage of parallelism both to exploit enabling technologies and to contribute to achieving the highest performance possible across a wide spectrum of algorithms and applications. Three of these most common hardware architecture forms present in a supercomputer are single-instruction multiple data (SIMD) parallelism, shared memory parallelism, and distributed memory parallelism. Shared memory and distributed memory parallelism are subclasses of the multiple-instruction multiple data (MIMD) class of Flynn's computer architecture taxonomy.

Each of these modalities is present in a modern supercomputer. While the unifying theme of a parallel computer architecture is parallelism, the ways in which a parallel algorithm exploits physical parallelism in each of these modalities can differ substantially. Some parallel algorithms are better suited for one kind of parallelism versus another. Often a completely different parallel algorithm will be needed, depending on the targeted parallel computer architecture structure. Frequently a combination of all three will be necessary for a parallel algorithm to achieve the highest possible performance that a supercomputer can provide.

In 2004 an influential set of seven classes of numerical methods commonly used on supercomputers were identified [1]. These are known as the “seven dwarfs” or “seven motifs”: dense linear algebra, sparse linear algebra, spectral methods, N-body methods, structured grids, unstructured grids, and Monte Carlo methods. These seven classes of algorithms represent a large segment of supercomputing applications today and many high performance computing (HPC) benchmarks are built specifically to target them. In addition to the original “seven dwarfs”, researchers have added other important emerging classes of numerical methods found in supercomputing applications, including graph traversal, finite state machines, combinational logic, and statistical machine learning [2]. Optimally mapping these numerical methods to a parallel algorithm implementation is a key challenge for supercomputing application developers.

Several classes of parallel algorithms share key characteristics and are driven by the same underlying mechanism from which the parallelism is derived. Some examples of these generic classes of parallel algorithms include fork–join, divide and conquer, manager–worker, embarrassingly parallel, task dataflow, permutation, and halo exchange. Some examples of each class are listed in Table 9.1.

Table 9.1. Examples of Generic Classes of Parallel Algorithms

This chapter examines a wide variety of parallel algorithms and the means by which the parallelism is exposed and exploited. While the specific implementation of the algorithm for SIMD or MIMD parallel computer architectures will differ, the conceptual basis for extracting parallelism from the algorithm will not. The chapter begins by examining fork–join type parallel algorithms and an example from the divide-and-conquer class of parallel algorithms, parallel sort. Examples from manager–worker type algorithms and a specific subclass of it, embarrassingly parallel, are then examined. Halo-exchange parallel algorithm examples are examined also including the advection equation and sparse matrix vector multiplication. A permutation example of Cannon's algorithm and a task dataflow example of a breadth first search algorithm complete the chapter.

3.6.4 Compilers and Compiling

Cluster supercomputers provide several suites of compilers and debugging tools to support the diverse user community using the cluster. The individual user environments are most frequently customized using a module system. Using modules, the compilers and relevant environment paths can be changed in a dynamic way transparent to the user. A list of the most common module commands is given in Table 3.3. An example of module usage on a Cray XE6 is provided in Fig. 3.10.

Table 3.3. A List of Commonly Uses Module Commands for Dynamically Controlling the User's Software Environment

An example of some usage of these commands is shown in Fig. 3.10. Brackets indicate required arguments, while angle brackets indicate optional arguments.

With the specific compiler flavor and version controlled by loading modules, compiling a source code usually translates into invoking a compiler wrapper and supplying compiler flags along with the source code in the same way as is done when compiling a serial (nonparallel) application. In cluster environments, the C compiler wrapper for applications using the MPI (see Chapter 8) is most frequently called mpicc.

7.2.1 CPU and Memory Resource Management

Current supercomputers and data centers (eg, Amazon EC2) generally consist of thousands of computing machines. At any time, there are tens of thousands of users running their high-performance computing applications (eg, MapReduce [12], Message Passing Interface (MPI), Spark [13]) on it. The efficient resource management of the computing resources such as CPU, memory is nontrivial for high performance and fairness. Typically, the resource management includes resource discovery, resource scheduling, resource allocation, and resource monitoring. Resource discovery identifies the suitable computing resources in which machines that match the user’s request. Resource scheduling selects the best resource from the matched computing resources. It actually identifies the physical resource where the machines are to be created to provision the resources. Resource allocation allocates the selected resource to the job or task of the user’s request; it means the job submission to the selected cloud resource. After the submission of the job, the resource is monitored.

There are a number of resource management tools available for supercomputing. For example, Simple Linux Utility for Resource Management (SLURM) is a highly scalable resource manager widely used in supercomputers [14]. It allocates exclusive and/or nonexclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (typically a parallel job) on a set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work. For data-intensive computing in data center, YARN [15] and Mesos [16] are two popular resource management systems.