The fragmentor repository (branch
sc-3400-diamond-cluster) has a number of generic notes about setting up the database, here are some project-specific notes that may be too sensitive to include in the public repo’s documentation.
Address
Fragmentor server: 192.168.34.14, 130.246.81.18
Additional notes
We clone the fragmentation code and playbooks to the playground bastion: -
ssh playground-bastion
mkdir fragmentation
cd Fragmentation
git clone https://github.com/InformaticsMatters/fragmentor.git
cd fragmentor
And then setup the execution environment: -
sudo apt install python3.8-venv -y
python3 -m venv ~/.venv/fragmentor
source ~/.venv/fragmentor/bin/activate
pip install --upgrade pip
And setup environment variables to identify our S3 storage: -
export AWS_ACCESS_KEY=<Access Key ID>
export AWS_SECRET_KEY=<Secret Access Key>
export AWS_REGION=''
export S3_URL=https://s3.echo.stfc.ac.uk
With this done we should be able to setup a suitable parameters file (described
in the repo’s README) and then configure the ubuntu database server (and start and stop the database).
sysstats
There are playbooks in the https://github.com/xchem/fragalysis-stack-kubernetes repository that configure and collect statistics from machines using sysstats. The xch-bastion typically contains a clone of this repository from where the playbooks can be run.
Start by logging in to the bastion (ideally as wmy55374 where the clone and playbooks and private-key files are available).
Configuring
From the clone on the xch-bastion navigate to the topology directory and configure any inventory that you need to. Here we configure the fragmentation machines in the topology: -
cd git/fragalysis-stack-kubernetes/topology
ansible-playbook playbooks/sysstat.yaml --limit fragmentation
Collecting visual stats (SVG via sadf)
sysstats collects stats, saving them in a separate file for each day. We have a simplified playbook that allows you to collect the stats (as an SVG file) given a previous day (a look back). It generates the SVG on each machine and then pulls the file back to the bastion (into the local /tmp/svg directory).
Here, running the playbook from the topology directory, we collect files for “yesterday” (the default) for all the fragmentation machines: -
ansible-playbook playbooks/generate-sadf.yaml --limit fragmentation
Once complete you should have a number of files in the xch-bastion /tmp/svg directory. Their names will be logged by the ansible script’s output, so you should be able to see the filenames that have been collected.
Separate SVG files are created for each machine for its CPU and Memory statistics.
To collect files from two days ago set the look_back variable to -2: -
ansible-playbook playbooks/generate-sadf.yaml --limit fragmentation \
--extra-vars look_back=-2
Collecting textual stats (daily average)
sysstats collects stats, saving them in a separate file for each day. We have a simplified playbook that allows you to collect the stats given a previous day (a look back). It collects the daily average line on each machine and updates a text file. It then pulls the file back to the bastion (into the local /tmp/csv directory).
ansible-playbook playbooks/generate-summary.yaml --limit fragmentation
Average results are inserted into the file for each day, and the file grows with each new day. The result is a files with a series of daily averages, indexed by date. For example: -
2025-08-05,0.00,0.00,0.00,0.00,0.00,99.99
2025-08-06,0.00,0.00,0.00,0.00,0.00,99.99
It doesn’t matter if you collect the stats for the same day more than once, the ansible line-in-file module ensures that there is only one line that begins with the chosen date.
Once complete you should have a file in the xch-bastion /tmp/csv directory for each machine. Their names will be logged by the ansible script’s output, so you should be able to see the filenames that have been collected.
To collect files from two days ago set the look_back variable to -2: -
ansible-playbook playbooks/generate-summary.yaml --limit fragmentation \
--extra-vars look_back=-2
Keep the topology up to date
The topology/inventory.yaml file in the repo is crucial. Keep it up to date whenever you add or delete machines in the xchem-follow-up project. It also needs to be committed to GitHub.