Saturday, 2 December 2017

Solar Atmosphere Wave Dynamics Generated by Solar Global Oscillating Eigenmodes

We have recently completed our work investigating the dynamics and upward propagation of waves which are generated by the solar global eigenmodes. A lot of the background work is documented on this blog under the pmode label.

In this post I describe a series of hydrodynamic simulations of a realistically stratified model of the solar atmosphere representing its lower region from the photosphere to low corona. The objective was to model atmospheric perturbations, propagating from the photosphere into the chromosphere, transition region and low corona. The perturbations are caused by the photospheric global oscillations. The simulations use photospheric drivers mimicking the solar p-modes. In this post I review the work and highlight some of the results. Finally we review future work to investigate how the mean magnetic field itself would change the coupling of the global solar acoustic modes to the overlaying magnetised atmosphere. 

The discovery of solar global oscillations is highly remarkable and has resulted in the technique of helioseismology which has revealed  Observations of William Herschel as early as 1778-1818 revealed a granular structure. Around 1904 studies by Pierre Janssen suggested these structures had a diameter of around 1500km. These were soon attributed to the convective cells. In the 1960's the spectrographic studies of Evans and Leighton were able to identify and measure the rising and falling motions of solar the solar atmosphere. Using  Doppler shifts for different spectral lines the average turbulent velocity was measured for different heights. Leighton employed a technique different to that of Evans his approach study the doppler shift for a single spectral line (Ca-K band) but to scan two images across the solar disk. The resulting dopplergrams were subtracted to reveal an oscillatory motion with a period of 300s and amplitude of 300-400m/s. It was realised that this was a global phenomena. A number of theories were suggested for example the granules  acted as a piston driving the acoustic osillations, in another theory (Lighthill and Lamb) it was suggested that the stratified solar atmosphere acts as a filter possessing characteristic cutoff frequencies. A third model, the resonant cavity model although it could maintain five minute modes the cut-off frequency at the temperature minimum of the solar atmosphere meant that the 5-minute modes could not escape into the solar atmosphere. Further observations (Frazier and Mein 1965) provided evidence that the oscillations were generated deep in the convective interior. In 1970 Ulrich's theory used a resonant cavity with its upper boundary just below the top of the convection zone. The theory provided a dispersion relation showing how waves of given frequency and wavenumber lay along define ridges. Initially these were not observed this was because it was necessary to observe the sun over a long time and a large area. With the rise of the global oscillations network and the big bear observatory  such observations were possible and spectra like that of Ulrich's phase diagram were observed e.g. see below, the new discipline of helioseismology had arrived and we were beginning to understand the nature of solar global oscillations. As a student I remember a lecture on stellar interiors, it was claimed that this was a good area of study because it's difficult to look inside a star to verify your theory, therefore you have a greater freedom. Perhaps this speaker knew about the developments in helioseisomology it has changed things for our own star it's a developing and exciting field and that comment years ago fueled my curiosity.  A well worthwhile read is the book Sunquakes: Probing the Interior of the Sun by Jack Zirker (see reference 12).

It was recognised that turbulence in the convection zone generates acoustic noise continuously with some waves reflected back into the cavity by the photosphere. Thus the photosphere is driven up and down by evanescent waves leaking through the upper boundry with a spectrum which has a peak at five minutes. Since the advent of coronal seismology  many space-based high-resolution solar observations e.g. SOHO, TRACE, SDO and IRIS (to name but a few) have provided evidence for wave phenomena in the solar atmosphere. The power spectra reveal strong 3-5-minute oscillations in all channels and in- clude some longer period modes too. Using SDO/AIA data we show in Figure (1) the power spectrum in nine AIA passbands for randomly selected single pixels in and Active Region (AR), Quiet Sun (QS), and a Coronal Hole (CH) on a randomly chosen day (22 August 2010) during solar minimum. The power spectra are derived by studying image sequences at solar minimum for the different solar regions e.g. AR, a typical QS region and a CH.

The power spectrum in nine AIA passbands for single pixels in AR (black solid), QS (red solid), and CH (blue solid).

These results demonstrate the ubiquity of the observed 3- and 5-minute oscillations in all channels and regions and may serve as evidence of a global excitation mechanism. These observations are our strong motivation to model whether global p-modes may penetrate in the atmosphere.

Numerical MHD Simulations

To investigate these phenomena we ran a range of numerical hydrodynamical simulations of a model solar atmosphere for the quiet sun. We studied both the motion and the energy propagation characteristics for a range of frequencies and mode numbers.  The different driver modes are illustrated below for a vibrating membrane. This model is oscillator is located at the base of the model and is coincident with the temperature minimum.

The solar global modes of oscillation can be understood from the gravitating slab model discussed in the blog post "Our Wobbling Star". The modes observed at the solar surface are characterised by two indices l and m. l is the number of nodes between the poles whilst the azimuthal mode number m is the number of modes around the circumference. There are many thousands of modes some of which are illustrated below.


Research-IT Services Used

The completion of this work has been dependent on a range of research-IT services hosted and supported by The University of Sheffield. Research support was required throughout the full lifecycle of the project.  This work was funded by the Science and Technology Facilities Council (STFC), UK this funding was crucial to develop the codes used to perform the simulations and the analysis.
The computational models were run using the smaug code using the GPUs hosted on the iceberg high performance computer. Reference 11 provides details about the development and implementation of the code. We used revision 257 of SMAUG which can be checked out from the ccpforge code repository, this was checked out using the following subversion command.

svn checkout -r 257

A record of the project  was maintained using the github repository
This was used for a number of things
  •  to monitor any code changes and modifications (which would be checked back into the main repository on ccpforge)
  • The different simulations that were performed
  • The analysis applications  for visualisation and post processing
  • The final paper
The HPC services we used are heavily dependent  on GPU technology
A range of different software packages were used

A subset of the final results was published on the research data archive service provided by The University of Sheffield.

Results of Numerical Simulation

Since we are interested in the propagation of energy into the solar corona we computed the energy flux for different times and for different sections through the model. We also used time-distance plots to  characterise the modes of oscillation. Since we ran a large number of simulations with different modes and frequencies we attempted to summarise the results by preparing a collection of videos.

Each video shows the value of the vertical component of the plasma velocity (z-component) along different slices through the simulation box. The scale shows the velocity value in m/s. The green vectors show the velocity directions along a single slice through the simulation. The green surface at a height of 3.5Mm is the 2MK temperature isosurface. The simulation box is 4Mmx4Mm along the base and the height of the box is 5.7Mm. The driver for the simulations is located at a height of 0.5Mm.

Each video is labelled using 3 numbers. The first number is the driver period in seconds. The following 2 integers are each of the mode indices for the x and y direction respectively. Details of the full set of simulations is given in reference 1 see also the link to the repository at;

Videos of Magnetohydrodynamics Simulations of Solar Atmosphere Wave Dynamics Generated by Solar Global Oscillating Eigenmodes


Our results support the notion that solar global oscillations may be a driver for a range of global dynamical phenomena resulting in chromospheric and low coronal Doppler and intensity oscillations which, after all, may contribute to the non-thermal energy present in the solar atmosphere. We would like to emphasise that these upper atmospheric ubiquitous wave phenomena may not arise solely from the photospheric p-modes. On the contrary, a range of sources including, turbulent motions from convective cells, local nano-flares, small-scale Kelvin-Helmholtz instabilities, or continuous reconnection events in the magnetic carpet may contribute to their excitation.
Among others, we found that.
  1. There is consistency between the frequency-dependence of the energy flux in the numerical simulations and power flux measurements obtained from SDO;
  2. energy propagation into the mid- to upper-atmosphere of the quiet Sun occurs for a range of frequencies and may explain observed intensity oscillations for periods greater than the well known 3-minute and 5-minute oscillations;
  3. energy flux propagation into the lower solar corona is strongly dependent on the particular wave modes;
  4. agreement between the energy flux predictions of our numerical simulations and that of the two layer Klein-Gordon model supports our interpretation of the interaction of solar global oscillations with the solar atmosphere.

Future work

 An important caveat of the present work is the modelling of the active response of the atmospheric magnetic field. Although the plasma-β may be very low in the low corona, this approximation may serve an appropriate initial insight, nevertheless one needs to relax this condition and analyse how perhaps the mean magnetic field itself would change the coupling of the global solar acoustic modes to the overlaying magnetised atmosphere. Here, an interesting question would be to investigate whether slow or fast MHD waves are the key stakeholders in the re-distribution of the convective kinetic energy.

It has been recognised that magnetic fields influence the propagation of solar acoustic modes in a variety of ways (reference 8).  Hindman (reference  5) demonstrated the frequency shift of p-modes in vertical fields of different strengths. Cally and Goosens demonstrated that in the region of the solar atmosphere where the alfven speed matches the sound speed there is a significant transfer of energy between the different modes (see reference 7). A study of wave propagation in sunspot umbrae (see reference 4 below) reveals a suppression of wave power. The complexity of modelling physical processes in the highly stratified and magnetised makes it challenging to understand global oscillation phenomena. We have undertaken initial simulations with a uniform vertical field and using a (2,2) mode 300s driver. Some of the simulations have resulted in high frequency oscillations, these simulation artifacts can be filtered e.g. see the method used in reference 4. Our simulations clearly demonstrated the blockage of the 30s p-mode due to the cut-off for a magnetic field free atmosphere (see reference 6 below). However early results of computational simulations of the 30s mode using a magnetised solar atmosphere clearly demonstrate a leakage of energy in the solar corona the observed speed of propagation of the leaked mode is suggestive of mode conversion. Khomenko and Collados review the modelling and observation of sunspot waves in reference 9.

Further work will be undertaken using larger domains with greater resolution using the smaug+ code described in reference 13.


  1. Solar Atmosphere Wave Dynamics Generated by Solar Global Oscillating Eigenmodes
  2. Videos of Magnetohydrodynamics Simulations of Solar Atmosphere Wave Dynamics Generated by Solar Global Oscillating Eigenmodes
  3.  A Fast MHD Code for Gravitationally Stratified Media using Graphical Processing Units: SMAUG
  4. Magneto-acoustic Waves in Sunspots: First Results From a New Three-dimensional Nonlinear Magnetohydrodynamic Code 
  5. Driven Acoustic Oscillations within a Vertical Magnetic Field 
  6. On wave equations and cut-off frequencies of plane atmospheres 
  7. Three Dimensional MHD Wave Propagation and Conversion to Alfven Waves near the Solar Surface. I. Direct Numerical Solution 
  8. Absorption of p-Modes by Slender Magnetic Flux Tubes and p-Mode Lifetimes
  9. Oscillations and Waves in Sunspots 
  10. On coronal oscillations
  11.  A Fast MHD Code for Gravitationally Stratified Media using Graphical Processing Units: SMAUG
  12. Sunquakes: Probing the Interior of the Sun Hardcover – 29 Oct 2003 by J. B. Zirker (Author) 
  13. MHD code using multi graphical processing units: SMAUG+ 

Friday, 24 November 2017

Coronal loops heated by transverse waves

This week we had a talk about solar coronal heating with a particular focus on transverse waves in coronal loops the speaker was Tom Van Doorsselaere who visited Sheffield. The talk was introduced with an overview of recent observations of transverse waves in the solar atmosphere. He highlighted a claim that there were no waves in the corona! The time distance plot below 

A good introduction to transverse wave in the solar corona is given by the Warwick centre for Fusion, Space and Astrophysics.

Coronal Transverse Waves

Tom talked about coronal heating and identified mechanisms for heating the solar corona as well as convective heating and radiative transfer i.e. direct coronal heating he noted the important contribution of waves.

Tom summarized the current equilibrium models for coronal loops and reviewed some of the recent results from numerical simulations run by their group, e.g. see, Observational Signatures of Transverse Magnetohydrodynamic Waves and Associated Dynamic Instabilities in Coronal Flux Tubes.

Tom showed that coronal loops driven by transverse waves form a turbulent structure, via the Kelvin-Helmholtz instability or uniturbulence. The cascade of the wave energy to the small scales is beneficial for heating of solar coronal loops by waves. Tom described how forward modelling  was used to make the connection of the wave models to the observations. These results were generated using the FoMO application.

 Snapshots of the emission line flux (in inverted grayscale colors) in Fe xii 193 for model 1 (left panels) and in Fe ix 171 for model 2 (right panels) with the velocity field overlaid in rainbow colors. The snapshots follow the formation of the first KHI vortices (named TWIKH rolls). The color scale for the velocity field incorporates the extrema over the entire simulation.
Time–distance diagrams of the intensities in the Fe ix (top panel) and Fe xii (middle panel) lines of model 1 and Fe ix line (bottom panel) of model 2 for a slit placed perpendicularly to the loop at the apex and with a LOS angle of 45° and at numerical (highest) spatial resolution.

It was fascinating to see how models of many thin flux tubes resulted in mixing giving rise to a single flux tube. It was good to hear about the FoMo code this is something that Robertus has encouraged us to do i.e. reconstruct observations using our simulation data, here's how you do it.

Friday, 27 October 2017

Discoveries in the Great Solar Data Mountain

One of the key's to the discovery of the Higgs Boson by the Large Hadron Collider at CERN
was the recognition of the quantity of data required to reveal a signature of the Higgs Boson. This needed a novel approach to the analysis and storage of the mountain of data that would be generated by the Large Hardron Collider.... big data had arrived. Just as CERN was first with the world wide web it was also first with the recognition of the Big Data problem. The story started with the discovery of the neutral current (a.k.a. the Z boson) in 1973 by the Gargamelle bubble chamber. This was remarkable because three events were observed in over 1.4 million bubble chamber photographs taken over a two year period, we didn't have digital image processing back then! Now, research is massively dependent on digital image processing and it is fascinating to follow the enormous variety of research problems in the area of solar physics, these researchers make use of toolkits such as the IDL solar software library, the more recent SunPY project or even Matlab. Along with many disciplines, there are two massive problems in this research field
  1. Solar physics has a big data problem, how do researchers collaboratively analyse the mountain of data both historical and being generated by new satellites studying our nearest star.
  2. How do we ensure that the software we use continues to be fit for purpose?
One of the solutions to the first problem was addressed by two excellent talks to the solar physics group the first talk a couple of weeks ago was about deep learning for solar flare forecasting. Yudong Ye gave todays talk which was an introduction to machine learning and its application to space physics. Ye said that Machine learning is more and more useful in this data explosion era and could be a powerful tool to reveal hidden connections and pave the way to new discoveries.  Two quantities  noted currently are the Sloane Digital Sky survey  which holds 140TB of data for optical telescope sky surveys between 2000 and 2010. The NASA NCSS holds 32pB of climatological data covering the years upto 2013. Ye provided a clear introduction to machine learning covering its concepts, he identified the different categories and gave recent applications in space physics. With an example of deciding whether there is a strong geomagnetic storm (namely, the Dst index is less than -100) from ICME’s plasma and magnetic field parameters using a support vector machine. He explained step by step how a machine learning method was applied to the specific problem described above. Further details are in references 11-15 below.
The previous talk given by Xin Huang discussed a model for  deep learning based solar flare forecasting. Solar flares originate from the release of the energy stored in the non-potential magnetic field of active regions, the triggering mechanism for these flares, however, are still unknown. For this reason, conventional solar flare forecasting is probabilistic and based on the statistical relationship between the characteristic parameters of active regions and solar flares. In the deep learning method, forecasting patterns can be learned from the line-of-sight magnetograms of solar active regions. It is necessary to obtain observational data with sufficient size to train the forecasting model and test its performance. Huang described how a dataset was created from the line-of-sight magnetogarms of active regions observed by SOHO/MDI and SDO/HMI from April 1996 to October 2015 along with the corresponding soft X-ray solar flares observed by GOES. The MDI data was taken as the training set and the HMI data as the testing set. The experimental result indicated that (1) the forecasting patterns can be automatically reached with the training set and these patterns can also be applied to the testing set, which is reduced to be the MDI proxy data; (2) the performance of the deep learning forecasting model is not sensitive to the given forecasting periods (6 hour, 12 hour, 24 hour or 48 hour); (3) a reasonable forecasting model is achieved for solar flares with higher importance. Huang used a deep learning package called CAFFE and used a single NVIDIA GPU (see references 6-9) below. He described how a cascade of layers in a convolutional neural network were used for feature extraction. The trick with deep learning is to exploit readily trained networks and to make use of supervised learning.

This talk was rather inspirational I've known for a long time that the Matlab package provides machine learning toolbox. At the risk of a little knowledge being dangerous I decided to try one of the matlab deep learning demos with a GPU, which is a Demonstration of Image category classification using deep learning (ref 2). This was very easy to run and I attempted a simple image classification on a set of photographs, clearly this is very powerful. But this is open to all our users on the central HPC at the university of sheffield it's possible to run the matlab deep learning demos. ShARC features a range of deep learning and machine learning software which has been well used and tested by the RSE and machine learning groups at The University of Sheffield.

A further possibility for researchers is to use the new deep learning cluster, JADE, based at Oxford ( see reference 10 ). It is fortunate that The University of Sheffield is a partner in this project making access much easier for researchers to use this powerful and increasingly used technique to meet the challenge of the big data problem (see reference 19). We can look forward to some excellent adventures exploring the great solar data mountain! 
  1.  The discovery of the weak neutral currents
  2. Demonstration of Image category classification using deep learning with Matlab
  3. Neural network toolbox for Alexnet Network with Matlab  
  4. Neural network importer for CAFFE models
  5. Mathworks neural networks toolbox team
  6. CAFFE
  8. TORCH
  10. JADE
  11. Predicting Coronal Mass Ejections Using Machine Learning Methods
  12. Solar Flare Prediction Model with Three Machine-learning Algorithms using Ultraviolet Brightening and Vector Magnetograms 
  13. Space weather research group (Bradford)
  14. Automated Prediction of CMEs Using Machine Learning of CME – Flare Associations 
  16. Studying imagery from solar dynamics 
  17. Application of Convolution Neural Network to the forecasts of flare classification and occurrence using SOHO MDI data
  18. Application of a deep-learning method to the forecast of daily solar flare occurrence using Convolution Neural Network
  19. GPU Computing Sheffield 

Friday, 3 February 2017

Heliseismology - Introduction

I remember attending a lecture many years ago about the physics of stellar interiors with particular emphasis on the sun. It is amusing that the lecturer noted that one of the reasons for his interest was his freedom to construct models in the knowledge that it was very difficult to come up with the experimental evidence to verify the models! The field of helioseisomology and neutrino observations are changing this idea and we are gleening more information about the interior of our nearest star and other stars too.

Reka Jain gave an excellent presentation, her experience in this area is great and covers many levels of expertise including information inspiring young people [ref. 6]. Reka provided an overview of Helioseismology and  briefly discussed some recent successes of global and local Helioseismology.

My interest  in this area relates to the link between excitations in the solar atmosphere and the solar global oscillations see reference 13 and 14.

Helioseismology is the study of wave oscillations in the Sun. Observations of acoustic wave oscillations are used to make helioseismic studies of the interior of the Sun even though helioseismic techniques operate slightly differently on different length scales.The diagram below illustrates the variety of reflections and refractions occuring when acoustic waves propagate in the solar interior.

The helioseismic vibrations arise from convective and turbulent motions within the solar interior. We can identify two types of modes, pressure driven p-modes and gravity driven g-modes. Solving spherically symmetric equations of hydrodynamics (see our wobbling star) the modes can be understood in terms of spherical harmonics illustrated in the figure below. The predicted power spectrum gives rise to a series of distinct ridges 200000  modes have been detected of a possible million.

Reka's talk was particularly interesting because not only did she provide a clear overview  we heard about exciting advances including applications of helioseismology for studying
  • sunspot growth and evolution,
  • flux tubes convecting to the surface and the study of
  • neutrino observations to determine characteristics of g-mode oscillations.
  • neutrino propagation in stellar interiors to determine the properties of dark matter
Deubner [ref. 7] explained the p-mode oscillations using  the phase and group velocity of fluctuations of different solar spectral lines. Agreement with theoretical estimates was shown for acoustic waves trapped in the sun. With increasingly accurate models of the sun from helioseismology it is possible to use this higher quality data to study the propagation of neutrinos such probes can be used in studies of the difficult to find g-mode oscillations. One possibility is to use the sun as a probe for fundamental physics and cosmology. An important idea is the detection of dark matter and to understand the impact that this has onthe formation of stars [ref 8

ref. 8 Neutrion propagation in the interior
The Sun as a probe of Fundamental Physics and Cosmology The high quality data provided by helioseismology, solar neutrino flux measurements, spectral determination of solar abundances, nuclear reactions rates coefficients among other experimental data, leads to the highly accurate prediction of the internal structure of the present Sun - the standard solar model. In this talk, I have discussed how the standard solar model, the best representation of the real Sun, can be used to study the properties of dark matter, for which two complementary approaches have been developed: - to limit the number of theoretical candidates proposed as the dark matter particles, this analysis complements the experimental search of dark matter, and - as a template for the study of the impact of dark matter in the evolution of stars, which possibly occurs for stellar populations formed in regions of high density of dark matter, such as stars formed in the centre of galaxies and the first generations of stars.

 ref. 9 Predictions of solar cycle  changes in p-mode frequencies change with the solar cycle
The Sun's activity measured through many of its proxies varies in a periodic manner with an average duration of about 11 years. The empirical relations based on the periodicity are considered as the first generation methods to predict the maximum amplitude of the next solar cycle. These methods which are statistical in nature fall into two different categories: precursor methods and extrapolation methods and has been widely used in the later part of the 20th century. Recent advances include predictions based on non-linear methods and dynamo models, where the later predicts not only the maximum amplitude of the solar cycle but also the timing of the activity maximum. In this review, we focus on different prediction methods and compare their outcome for previous cycles with an emphasis on cycle 24. We further analyze and compare various predictions for solar cycle 25 and beyond.

The figures above and below illustrate the variation of the meridional flow.

reference 11,12 helioseismic detection of supergranulation
We present measurements of the Sun’s sub-surface convective flows and provide evidence that the pattern of supergranulation is driven at the surface. The pattern subsequently descends slowly throughout the near-surface shear layer in a manner that is inconsistent with a 3D cellular structure. The flow measurements are obtained through the application of a new helioseismic technique based on traditional ring analysis. We measure the flow field over the course of eleven days and perform a correlation analysis between all possible pairs of depths and temporal separations. In congruence with previous studies, we find that the supergranulation pattern remains coherent at the surface for slightly less than two days and the instantaneous surface pattern is imprinted to a depth of 7 Mm. However, these correlation times and depths are deceptive. When we admit a potential time lag in the correlation, we find that peak correlation in the convective flows descends at a rate of 10-40 m s-1 (or equivalently 1-3 Mm per day). Furthermore, the correlation extends throughout all depths of the near-surface shear layer. This pattern-propagation rate is well matched by estimates of the speed of downflows obtained through the anelastic approximation. Direct integration of the measured speed indicates that the supergranulation pattern that first appears at the surface eventually reaches the bottom of the near-surface shear layer a month later. Thus, the downflows have a Rossby radius of deformation equal to the depth of the shear layer and we suggest that this equality may not be coincidental.

We present measurements of the Sun's sub-surface convective flows and provide evidence that the pattern of supergranulation is driven at the surface. The pattern subsequently descends slowly throughout the near-surface shear layer in a manner that is inconsistent with a 3-D cellular structure. The flow measurements are obtained through the application of a new helioseismic technique based on traditional ring analysis. We measure the flow field over the course of eleven days and perform a correlation analysis between all possible pairs of depths and temporal separations. In congruence with previous studies, we find that the supergranulation pattern remains coherent at the surface for slightly less than two days and the instantaneous surface pattern is imprinted to a depth of 7 Mm. However, these correlation times and depths are deceptive. When we admit a potential time lag in the correlation, we find that peak correlation in the convective flows descends at a rate of 10 - 30 m s-1 (or equivalently 1 - 3 Mm per day). Furthermore, the correlation extends throughout all depths of the near-surface shear layer. This pattern-propagation rate is well matched by estimates of the speed of down flows obtained through the anelastic approximation. Direct integration of the measured speed indicates that the supergranulation pattern that first appears at the surface eventually reaches the bottom of the near-surface shear layer a month later. Thus, the transit time is roughly equal to a solar rotation period and we suggest this equality may not be coincidental. 


  1. Helioseismology
  2. Lecture notes on stellar oscillations
  3. Introduction to helioseismology
  4. Leibacher, A New Description of the Solar Five-Minute Oscillation
  5. Ulrich, The Five-Minute Oscillations on the Solar Surface
  6. Junior introduction to Helioseismology 
  7. Deubner, F.-L., Acoustic waves and the geometric scale in the solar atmosphere  see also Some properties of velocity fields in the solar photosphere. V - Spatio-temporal analysis of high resolution spectra
  8. Lopes, The Sun as a probe of Fundamental Physics and Cosmology
  9. Tripathy, Predictions of solar cycle
  10. Jain K, Tripathy, Hill, Solar Activity in Cycle 24 - What do Acoustic Oscillations tell us? 
  11. Greer, Hindman and Toomre, Helioseismic Imaging of Supergranulation throughout the Sun’s Near-Surface Shear Layer 
  12. Hindman, Greer, Toomre, Helioseismic Imaging of Supergranulation throughout the Sun's Near-Surface Shear Layer 
  13. Solar wave theory blog: helioseismology 
  14. Solar wave theory blog: solar global oscillations 
  15. A Comparison Between Global Proxies of the Sun's Magnetic Activity Cycle: Inferences from Helioseismology 

Monday, 16 January 2017

The Sun in the Clouds: Using Amazon EC2 for High Performance Computing

Over the last few years HPC on demand has become possible, there is a strong market lead from Microsoft with it's Azure service and by Amazon with EC2.

Traditional HPC systems are normally shared systems using scheduling software  to optimise utilisation. Although these systems satisfy the needs of specific communities with large computational demands (e.g. the computational solar physics community)  running such a system for a large diverse community  is a challenge. The main issues are the wait time for s system to come online, the necessity for scheduling jobs and the operating system and software stack is dictated by the user community.

On demand research computing infrastructures such as those delivered by Amazon-EC2 and Azure can reduce the time taken to get a system on line with the software stack required b y the researcher. Such an environment is ideal for researchers working interactively developing research applications, analysing and visualising data. Setting up such systems is much easier with open source deployment software such as Alces flight.

In this post  we use Alces flight and Amazon EC2 to set up a compute node with solar physics software developed at The University of Sheffield e.g. SMAUG, SAC and pysac. We use amazon ec2 nodes with GPUs to run smaug and compare the costs  and performance for running codes on local HPC infrastructure and on Amazon EC2. We describe how to set up the software stack on Amazon EC2 using Alces flight.

To run the benchmarks the following steps were followed
  1. Set up and configure account with Amazon EC2
  2. Set up machine using Alces flight
  3. Configure flight and install the required packages
  4. Configure smaug and run the benchmarks
  5. Performance results

Set up and configure account with Amazon EC2

Setting Up and Configuring Account with EC2 is straightforward registration by visiting the AWS page at

After logging in with you account you are presented with the console which has a large range of services. We will use the EC2 service to get started we need to set up a keypair which will be used to allow secure access to the machine. To configure this from the EC2 console select key pairs under the network and security group. This shows a list of active keypairs that you own, there is also a button to enable creation and deletion of the key pairs. We also import the keypairs from here to our local/client machine so that we can access our machine.

The amazon documentation is very good here is a link to the information on setting up a keypair.
to access my amazon instances it was necessary to import the Amazon key (pem file) and copy into the .ssh folder under my home directory, note that permissions for this key must be set correctly e.g. using the command
chmod 400 my-key-pair.pem
For the putty and winscp clients from windows it is necessary to convert the pem key to a ppk file using puttygen. Details are given on the Amazon docs

Set up machine using Alces flight

Using alces flight has been made much easier with some excellent articles and documentation. The Amazon news article provides a good overview. 
New in AWS Marketplace: Alces Flight – Effortless HPC on Demand
However, to get started the online documentation is once again very good.
Flight Appliance Documentation

When we are configuring the machine it's important to use the Amazon spot market and initially to use instances of type t2.micro this reduces set up costs to a minimum. As soon as the machine has been configured we set the instance type to either g2.2xlarge or p2.8xlarge for our GPU tests

Setting up the instance with Alces flight

  1. Login to you amazon account select the EC2 dashboard click on instances to see all your machines
  2. Visit the alces flight documentation at select the launching aws instance option e.g. step by step instructions
  3. Goto amzon aws market place in the search box type “alces flight” and select community edition select region for pricing e.g. EU (Ireland), Select delivery type e.g. AMI or HPC instance
  4. From the next dialog check versio, region and EC2 instance type also check the key pair used (this should correspond to the name of one of the keys in your AWS managment session check under network&security -> keypairs, there is an option to create a key pair here
  5. Clicking one click launch will fire up the following messsage - check your AWS console under instances - get coffee while it initialises
  6. Login using ssh -Y -i “yourkey.pem” alces@XX.XXX.XX.X (the pem downloaded from console and should be visible from where you login)

Usage instructions and connection information for the instance  can be obtained by ticking the activated instance in the EC2 control panel and clicking the usage instructions tab or by right clicking the activated instance in control panel and selecting Connect from the drop down menu.

Configure flight and install the required packages

 When you have logged into your instance, the node should start with the prompt unconfigured and the warning message that it is not yet operational. Use the command alces configure node. For running single GPU tests with smaug we just require a master node without any slave nodes.

The following commands return the configuration information

alces about node
alces about identity
To start a gnome session use the command:
alces session start gnome
The session will respond with connection information which can be used with client applications such as tiger VNC.
Typical commands used from Alces flight are:
Commands to use from alces flight

'module avail'            - show available application environments
'module add <modulename>' - add a module to your current environment

'alces gridware'          - manage software for your environment
'alces howto'             - guides on how to use your research environment
'alces session'           - start and manage interactive sessions
'alces storage'           - configure and address storage facilities
'alces template'          - tailored job script templates

'qstat'                   - show summary of running jobs
'qsub'                    - submit a job script
'qdesktop'                - submit an interactive session request

'aws help'                - show help for AWS CLI

's3cmd --help'            - show help for S3cmd
's3cmd ls [<bucket>]'     - list objects or buckets
's3cmd put <file> <s3>'   - put file into bucket
's3cmd get <s3> <file>'   - get file from bucket

To install the required packages we use the alces gridware install command. Alces gridware is a powerful and extremely useful utility. To see the diverse range of packages which may be installed see the repository at
The following lines can be placed in a script file which may be run and can be used for testing and experimenting with the applications that we will use 


#set up packages using alces gridware
alces gridware install compilers/gcc
alces gridware install ffmpeg
alces gridware install imagemagick
alces gridware install graphicsmagick
alces gridware install grace
alces gridware install paraview
alces gridware install nvidia-cuda
alces gridware install anaconda3

Further information for working with the ALCES cluster is at
To install the required sac and smaug applications for running our tests the following commands were run from a script file


#script to upload an install projects in an alces flight installation

cd ~
mkdir proj

svn checkout --username mikeg64
svn checkout --username mikeg64

git clone

Configure smaug and run the benchmarks

To install the version of smaug on github (version 788e630) used for the performance tests the following git commands were used

cd smaug
git checkout 788e6303aa2ac670f8490f6e193587cbedf7383a

cd smaug
git reset --hard 788e630
We will be using the Orszag-Tang model test to set this change to the src folder of the smaug distribution and type
make ot

For compiling using the version of CUDA on the amazon instance load the cuda compiler module installed with alces gridware. To find the path to the nvcc compiler type which nvcc. The path should be used in the make_inputs file it will be necessary to change the CUDA and CUDACCFLAGS as illustrated below.

Compilation using
CUDA = /opt/gridware/depots/77cbfdae/el7/pkg/libs/nvidia-cuda/7.5.18/bin/toolkit
CUDACCFLAGS = --ptxas-options=-v -arch sm_20  -maxrregcount=32 -DUSE_SAC

It is necessary to set the architecture flag correctly guidelines for setting the architecture flag for GPUs are documented by NVIDIA, the GPU feature list is described at

In the CUDA naming scheme, GPUs are named sm_xy, where x denotes the GPU generation number, and y the version in that generation. Additionally, to facilitate comparing GPU capabilities, CUDA attempts to choose its GPU names such that if x1y1 <= x2y2 then all non-ISA related capabilities of sm_x1y1 are included in those of sm_x2y2. From this it indeed follows that sm_30 is the base Kepler model, and it also explains why higher entries in the tables are always functional extensions to the lower entries. This is denoted by the plus sign in the table. Moreover, if we abstract from the instruction encoding, it implies that sm_30's functionality will continue to be included in all later GPU generations. As we will see next, this property will be the foundation for application compatibility support by nvcc.

sm_20 (deprecated)
Basic features
+ Fermi support
sm_30 and sm_32
+ Kepler support
+ Unified memory programming
+ Dynamic parallelism support
sm_50, sm_52, and sm_53
+ Maxwell support

After configuring the machine we stop the instance state and change the instance type to g2.2xlarge ready to run the compiled version of smaug on the GPU.

The GPU infor for our Amazon image is as follows
Device Grid K520 on Amazon g2.2xlarge
4132864 GDR memory
CUDA ver 3.0
8 multiprocessors
0.797GHz clock

The configuration files for the Orszag-tang test are contained in a public compressed archive at the following link
The shortened link is 


To use these we change the model size and config filename in the iosmaugparams.h file in the include directory.

Performance results

The table below shows the timings obtained for a range of different NVIDIA GPUs these are the timings in seconds for 100 iterations

The speed up factor compared to a single core of the Intel Xeon X5650 Westmere CPU is shown in the plot below

Although our results demonstrate the K520  is slower than our results for the M2070  and the K20 it still demonstrates the usefulness of the K520 for running smaller models for MHD.  We will report on the results for the larger Amazon GPU instances as tests are continued. The ease with which we have been able to access and set up GPUs is highly encouraging.

Further details about SMAUG and benchtesting are given in the publication below.

Useful References

Guide for researchers by Amazon
AWS Global Data Egress Waiver removing the worry of estimating network traffic charges and set up invoice billing.

New P2 Instance Type for Amazon EC2 – Up to 16 GPUs

AWS Marketplace - alces flight instances

A Fast MHD Code for Gravitationally Stratified Media using Graphical Processing Units: SMAUG

Magnetohydrodynamic code for gravitationally-stratified media