How to convert to and from hex dumps using xxd
In an earlier article, I explained the use of compiled binary files to hide the code run within them:
https://daniel-ellis.medium.com/compiling-a-shell-script-to-binary-8dac07b770df
However, it is not possible to set environmental variables using this method. MD5 checksums were considered, however, these are NOT reversible, so naturally, we ended up using the hexadecimal representation of our text.
Conventionally xxd
is used to convert a text file — e.g.
echo "Magic Text " > text_dump
into a hex dump file. Although it is possible to convert into a binary dump using the -b
flag, this…
Unlike the interactive Graphical User Interface, there is also a CLI for use on headless machines or HPC clusters, when local memory requirements just don’t cut it. Here we go over some of the basic functionality:
Although the script works through normal python or Jupyter notebooks, the most intuitive way to explore data is through the IPython shell. This allows the user to run the script, and then manually manipulate the data afterwards.
There are a number of available arguments, and the easiest way to view these is by using the --help
option
A quickstart anaconda environment cheatsheet for everyday use.
We choose to create an environment named myenv
conda create --name myenv
A custom location (e.g. this directory under the name myenv)
conda create --prefix ./myenv python=3.7 --yes;
conda create --name myenv --clone env2clone
conda info --envs
conda activate myenv
Activating an environment at location ./myenv
conda activate ./myenv
conda deactivate
conda env remove --name myenv
conda env export > environment.yml
If we want to avoid version numbers we can use the following command instead:
conda env export | cut -f 1 -d '=' > environment.yml
conda env create -f environment.yml
conda…
With the introduction of two-factor authentication, we are now required to change the way we access our repositories. There are now two main methods to upload to Github: using ssh keys or access tokens. This page describes the latter.
user icon
(top-right)Settings
Developer Settings
Personal Access Tokens
Generate New Token
Copy and SAVE the token, you will not get to view it again without generating a new one!
If we wish to store our token for a fixed period in time we may cache
it with a predefined timeout.
…
How to create a fully customised colour mapping within matplotlib.
Our data is split into non-uniform categories, where a linear colour bar will be inefficient. In this tutorial, we cover how to create a custom colourmap with colours matching bins of our chosen sizes.
We start by importing the required packages
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
Next, we decide on our colours:
colours = ['#ffbe0b', '#fb5607', '#ff006e', '#8338ec', '#3a86ff']
And the bin sizes which correspond to them. — e.g. {0 → .3}, {0.3 → .7 } etc… I have intentionally chosen non-uniform bins.
…
Just some notes on getting started.
Bede is the N8’s Tier 2 Power and GPU-based high-performance computing (HPC) platform.
The filestore layout is similar to that of the ARC system. …
Notes on how to set up an interactive GPU session for computation on the Leeds System.
qrsh -l h_rt=2:0:0,coproc_v100=1,h_vmem=8G -pty y bash
A single V100 card, 10 CPU cores and 48GB system memory (one quarter of the available resource on a V100 node)
qrsh -l h_rt=2:0:0,coproc_k80=1 -pty y bash
A single K80 card, 12 CPU cores and 64GB system memory (half the available resource on a K80 node)
or
qrsh -l h_rt=2:0:0,coproc_p_100=1 -pty y bash
A single P100 card, 6 CPU cores and 64GB system memory (one quarter of the available resource on a P100 node)
module load cudanvidia-smi…
Sometimes we require multiple modules all of which communicate using the I2C channel. In this post, we discuss how to do exactly that using available GPIO pins.
We begin with installing the required libraries,
sudo apt-get install -y python-smbus i2c-tools
followed by enabling I2C using the advanced tab within raspi config
. I this does not make sense, have google ‘using I2C on Raspberry Pi’ and the device you are trying to connect. Make sure you can connect this successfully before continuing.
Generally, to set up we use pins 3 and 4 (see the diagram below) to connect a device. In…
You have a suite you wish to run, but need to constrain it such that it can run on your allocated queues — this guide explains how to break it into segments which allow you to do so.
The run I am using is u-ca218
. First, you need to open the rose editor and navigate to suite conf → run initialisation and cycling
using the menu on the left.
Here you will have a set of options including an initialisation file, calendar and other information.
Various two-factor authentication sites now require the use of an RSA key. For me, it is needed to log into the MONSooN supercomputer. Although this is often on my desk, it is still difficult to read when flat. For this reason, I created a stand and will detail the process here.
We begin by measuring our key. This is done in mm and the use of calipers can produce much more accurate results than a simple rule.
The main dimensions of my RSA key are approximately 20mm x 10mm with the screen being 5mm from the top and bottom.
Since…
Research Software Engineer specialising in Data Visualisation with a touch of HPC. — PhD in Atmospheric Chemistry and Masters in Theoretical Physics.