AWS Graviton2, Batch and multi-arch docker

Amazon Web Services (AWS) introduced a new 64-bit ARM Neoverse core which they named Graviton2 about a year ago with wide availability in June 2020. Marketing claims they provide up to 40% better price performance over x86-based instances. A footnote clarifies with “20% lower cost and up to 40% higher performance based on internal testing with varying characteristics of compute and memory requirements”. As you know, details can be important....

light-weight way to capture results from cloud HPC/HTC

For large HTC (or HPC) computations on the cloud, ‘spot instances’ (AWS-speak), ‘low-priority VM’ (Azure-speak) or ‘preemptible VM instances’ (GoogleCloud-speak) are the low cost options for compute. Of course, the challenge here is that these instances/VM can vanish at anytime. If you’re doing a large HTC task, you want to make sure you save your result (and/or checkpoint) files as soon as they are generated to persistent storage. Otherwise you lost the computation you just paid for....