Topic for Discussion Sessions
Reflections and Perspectives on Avenues to Performance
One can naturally consider the following two questions:
What have the main novelties been during the past decade, in all aspects of HPC?
What is our educated guess on the key developments for the next decade?
To organize our thoughts we can schematically partition the history of solving problems by computation into three broad paradigms:
Deployment of highly structured models, which significantly leverage domain-specific knowledge and insights coming from human intuition. Typically, data representations are developed that enable casting the problem in a homogeneous form. This paradigm encompasses most of the work that has been presented and discussed during the two decades of ScalPerf.
Deployment of traditional data regression models, which do not typically exploit domain specific knowledge. Examples are Support Vector Machines and Principal Component Analysis.
Deployment of recently developed AI/Machine-Learning models that enable data regularity to emerge, by massive computation on large data sets, with limited a priori domain-specific knowledge. Large Language Models are a crucial example, in very recent years. Experimentally, we see that patterns tend to emerge only at a rather large scale. Computationally, these models can be very expensive, with costs of the order of hundreds of millions of dollars. Avenues to reduce these costs are under active investigation.
A promising direction, which begins to be pursued in several business areas, consists in tailoring general models to focused application areas. Effective ways are searched to incorporate expert knowledge (for example, Bayesian kernels). In a way, the goal is a synthesis between Paradigms 1 and 3 described above.
While pondering on the target questions, within the outlined framework, it may be useful to identify the specific role of each layer of the computing stack, such as VLSI and other technologies, circuit design, machine architecture, compilers, performance tools, system software, application software, programming models, algorithms, theory of computation, etc. From which layers of the stack can we expect further performance improvements in the coming decade? And how will these layers evolve, to best serve the new computational paradigms? Do we expect basic matrix operations to be the dominant primitives, as in the last twenty years?