Skip to topic | Skip to bottom

Provenance Challenge


Start of topic | Skip to actions

Crystallography Workflow


This page outlines the crystallography workflow for the Fourth and Last Provenance Challenge. It describes the workflow and how it maps the abstract scenario. This page also provides all the required information and data needed to run the workflow and take part in the challenge.

Brief Summary

Crystallography is the experimental science of determining the arrangement of atoms in solids. Crystallographic methods depend on the analysis of the diffraction patterns that emerge from a crystal sample that is targeted by X-ray beams. In the scenario described below, scientists perform a series of steps to produce a set of atom coordinates from a crystal, and then publish this on a public database. The raw data and conduct of an experiment which produced a crystal image are important for others to interpret the quality of that image.

This experiment is one performed by crystallographers working in King's College London. The process abstracts from the details but has been confirmed to be realistic, and the provenance questions are ones which have been confirmed as valuable to answer.

Example Workflow

In this section, we first show the general worklow for performing a crystallography experiment, then specify a single example of this workflow being executed.


Full size link

In the image above the red outlined boxes around areas of the workflow refer to areas which are mapped from the abstract scenario. All of these have their names to the left of the box. The light blue labels attached to each element of the workflow are to help explain the scenario as descried below.

We will now describe a single instance of the experiment. Here, the crystallographer had two crystal samples of the same protein and wanted to work out the structure of the molecule using x-ray diffraction techniques.

This section is split up into stages purely for clarification of explanation, i.e. no other semantics should be attached to the division. The letters in brackets refer to artifacts and processes in the general workflow above.

First stage.

Second stage

  1. Mosflm was then used to generate an unmerged reflection file(H) from the diffraction images(C) we just obtained(G).
  2. This file was then transformed(I) into a merged reflection file(J).
    1. 1. Before it can be merged we needed to rearrange it as the unmerged files are given in no particular order.
    2. 2. We then merged the file using SCALA, which is part of the CCP4 suite

Third Stage

Fourth Stage

Fifth Stage

The diagram below shows the cardinality relationship between the different data objects produced by running the work flow and how they are related to each other. This is provided in addition to the workflow to aid in understanding in what is produced from each stage of the workflow.


Sample OPM Graph

At the link below, we show a hand-made, mocked-up OPM graph to describe the above experiment. This should be used for further clarification as to the structure of the experiment.

We intend to refine this graph, in particular to show accounts of differing granularities.

Sample OPM Graph

Mapping from Abstract Scenario

Below, we map from each step in the abstract scenario to the procedures which comprise the crystallography workflow.

User Performs Action

This is mapped to the beginning of the workflow(A,B) because at the beginning of the experiment the crystallographer must choose which crystal they will use, mount it on the synchrotron and get some diffraction images.

User Decision Point 1

The first user decision point is mapped to inspect images(E) as after inspecting the images the crystallographer must make a decision as to whether the images are good enough and if the crystal is still ok to use and if not repeat the first part of the experiment again.

Exchange Between Services

The exchange between services happens between mosflm and the detector.(C,D,E)

Collection Manipulation

This is collection manipulation as the MTZ files are collections of data about spots which start off as unmerged(I) which contains data about the same spot multiple times. It then needs to be rearranged and merged to create a new file(J) which only contains one occurrence of each spot.

Running Service with Others' Data

This is mapped to this area of the workflow(L) because when the reflection data is processed to get coordinates and the statistical data used to tell if it is a good match. it is compared to the structure of similar proteins from the PDB. This is retrieved from the PBD using a search tool which matches the sequence data of proteins(K). This is then processed using services provided from CCP4.


The workflow is mapped to this section as this part is a linear workflow(M,N,P) which will be automated for our process there are not any particular areas of interest within this from a provenance point of view. This produces an image of the protein in it's current form.

User Decision Point 2

This is 'check satisfaction of coordinates'(Q) as within this step the user must decide if the coordinates(P) generated through coot are accurate enough and if they are not go back and change them.

Publish Data To URL

The data is submitted to the PDB which will then mean it will have a web reference.(U)

Collaborative Editing

This is mapped to the section involving the wiki(W,X,Y), as the wiki is used for discussion and editing of the report by multiple collaborator.

Citing Paper

The data which is in the PDB is now cited in the report(Z) produced by collaboration on the wiki.


Submitting to the biology journal(AA,AB) fits the credentials section, as to submit to the biochemistry journal, you need to be logged in the the appropriate sort of account.

Discovery By Query

Anyone is able to search the Biochemistry journal(AC) through the search link on the main page so the user can query for information about the protein solved in the scenario and discover the report(AD)

Social Collaboration

Social collaboration in the crystallography workflow is done through use of twitter(AE) to make comments and suggestions about the paper

File Formats used

Below, we describe the key data formats used and/or produced in the experiment.

Diffraction File Formats

Diffraction files come in many different formats which are decided by the manufacturer, some examples of these file types are *.img; *.mar; *.mccd; *.image; *.sfrm; *.osc. For our workflow, only one will be used to simplify things: *.img

The header metadata may be viewed if an image is opened in a text editor. Software such as Mosflm and HLK2000 read the image header and automatically extract the metadata when images are loaded. The metadata is used by the software to determine the experimental settings as part of the image integration and spot finding process

Visualisation Software

We use two types of visualisation software in the workflow. One is to visualise the diffraction images and the other to visualise the final crystal images.

For the workflow only Mosflm will be used for the visualisation of diffraction images to simplify the process.

Reflection files are stored as *.MTZ files. MTZ is a binary flat-file format containing reflection data and a header of metadata An MTZ file can be viewed using the CCP4 viewer program. If this is done then the file can be viewed as plain text.

These files come in 2 forms, merged and unmerged.

An unmerged reflection file consists of one big file which can be considered to be a collection of 'spots', where there will be multiple occurrences of the same spot within the unmerged file. Spots are represented as a position and intensity, where spots of the same position can have different intensities. Each spot carries a reference to the diffraction image from which it was extracted.

A merged file is still a collection of spots but each spot will only appear once and the intensity of each spot is the averaged intensities of every appearance of the spot from the unmerged file. This is done using SCALA from CCP4.

For the visualisation of the final crystal image we will use coot as it can read a number of reflection file formats and coordinate data.

CCP4 Scripts

Sequence files

Text files consisting of letters that represent a sequence of amino acids and describe how they are chained together to build a protein.

Coordinate Files

The coordinate data is present in the pdb file and is the geometry that defines how different atoms are linked together in the protein molecule. These are line separated plain text files


The packages listed below are required to run the entire Crystallography workflow. Some of these may not be required depending on which area of the workflow is being worked on by a particular challenge team.


This can be found at

It is required for the part of the workflow which specifies its use, but is also required to run Mosflm in the first section of the workflow.

CCP4 runs on Windows, Linux (generic x86 version or specific RedHat? EL4 version) and Macintosh. In addition to this, the source code is available as well.

It is simple to install, as there is a form on the website which will generate an archive containing all the packages required with instructions on how to install. This should be installed before Mosflm to avoid any problems.


Mosflm can be found at

This is required for the first section of the workflow. Mosflm comes in two versions, we will be using the command line version. Once CCP4 is installed this can just be run. It works on Linux, OSX and windows.


There exists builds for linux and windows. The source code is also freely available on the website.

Coot provides an interactive visualisation of coordinate files which outputs a new set of coordinates once manipulation is completed. Installing coot is easy, as all that needs to be done is to download the file, untar it, and then add a specified folder to your path environmental variable.

Execution Instructions

In this section, we will add more detailed instructions about executing the steps in the experiment yourself. Please refer to the section below for the input data used for each step.

Step 1: Visualise Images in Mosflm 1st Iteration

This requires Mosflm to have been installed.

Run ipmoslfm <commands.txt, in a directory containing commands.txt and the three image files listed for this step below.

Alternatively, to perform this interactively, run ipmoslfm and, at the prompt, enter each command listed in commands.txt in the directory containing the images.

IMPORTANT NOTE: This works on Linux but does not appear to work on Windows (the image files produced are empty), possibly due to a bug in the Windows version of Mosflm. We have contacted the authors of Mosflm to check why this might be.


Below, we provide sample data for the experiment instance as described above.

Step Procedure Data Role Item 1 Item 2 Item 3 Item 4 Executables Required
1 Visualise Images in Mosflm 1st Iteration Inputs First Diffraction Image Second Diffraction Image Third Diffraction Image Matrix File Mosflm
Outputs First Diffraction Image Visualisation Second Diffraction Image Visualisation Third Diffraction Image Visualisation  
2 Visualise Images in Mosflm 2st Iteration Inputs First Diffraction Image Second Diffraction Image Third Diffraction Image Matrix File Mosflm
Outputs First Diffraction Image Visualisation Second Diffraction Image Visualisation Third Diffraction Image Visualisation  
3 Using CCP4 To Produce Unmerged Reflection File Sort Inputs First Diffraction Image Second Diffraction Image Third Diffraction Image   CCP4
Outputs Unsorted Reflection File      
4 Merge Reflection Files:Sort Inputs Unsorted Reflection File       CCP4
Outputs Sorted Reflection File      
Parameters Sort Command Log File      
Merge Reflection Files:Scala Inputs Sorted Reflection File      
Outputs Scala Reflection File      
Parameters Scala Command Log      
Merge Reflection Files:Truncate Inputs Scala Reflection File      
Outputs Truncate Reflection File      
Parameters Truncate Command Log      
Merge Reflection Files:Free Input Truncate Reflection File      
Output Merged Reflection File      
Parameters Free Command Log      
5 Process Reflection Files To Produce Coordinates Inputs Merged Reflection File Sequence Data      
Outputs PDB File Without Added Details        
6 Visualise Final Crystal Image Inputs PDB File Without Added Details       Coot
Outputs Visualisation of Crystal Structure      
7 Add Extra Information To Coordinates File Inputs PDB File Without Added Details Sequence Data      
Outputs PDB File With Extra Details        

Provenance Questions

NB: We wish to cover the range of questions proposed at the challenge workshop in Troy. The list is still under development, and suggestions are welcome.

We wish to ask the following questions about the provenance of a crystal image.

Question 1: It is 10 years after the process was conducted, and the process has become obsolete. For a given published crystal image (named by web reference), what was the raw diffraction images from which the crystal image was produced? Assume that the public database can contain only the coordinate and reflection files, and data kept on the desktop PC which ran the process has gone and knowledge in people's heads has been forgotten.


Question 2: For a given crystal, how often did a crystallographer reject and reproduce coordinates (the later stages of the experiment)? This is important because difficulty in obtaining an adequate crystal image can indicate that the original diffraction data was poor quality.

In our experiment this process was repeated 4 times.

Question 3: During the experiment the first loop is executed more than once. This could be done for a number of reasons. Was it to retrieve more images to continue the experiment or to try a new crystal?

First iteration was to try a new crystal Second iteration was to gather more images to continue the experiment

Question 4: Sequence data is used to help solve the structure of the protein by finding a similar protein which has been solved and comparing the sequences, seeing how they differ. Where this is done what is the similar protein used and where was it sourced?


Question 5: The report has been published but how many times has it been edited before being published?

The report was edited 15 times?

Question 6: The identity and characteristics of the synchrotron may be lost over time if it is decommissioned. For a given PDB entry, what was the synchrotron used and what were the configurations for the experiment(Polarisation, wavelength and wavelength)?

Question 7: How many times has this data been cited in other reports? 4

Question 8: Who made the most edits to the report? ??

Question 9: After publishing the report it is discussed on twitter and edited again, how many times has the paper been edited as a result of discussion on twitter? 4?

Vocabulary and Ontology

In order to provide consistency in identifying types and concepts in provenance traces, we have prepared a common vocabulary of URIs. They are constructed as follows. The base URI for all the terms we define below is

The set of terms is as follows:

We have also prepared a tentative OWL file mapping concepts specific to the crystallography workflow to those in the abstract scenario. It is available here.

-- CarlBarton - 23 Jun 2010
to top

End of topic
Skip to action links | Back to top

You are here: Challenge > Crystallography

to top

Copyright © 1999-2012 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.