CHAPTER 1

Overview

 

 

1.1     Introduction

 

Precognition is an end-user software system that processes single crystal diffraction images and yields experimentally-observed structure factor amplitudes.  It is also an API of a small set of modules that can be incorporated into other software systems.  Precognition is suitable for processing diffraction images from small molecule, protein and virus crystals.  Each type of image processing requires very different implementation.

 

Diffraction image processing is the first step in a rather complex computational process leading to a crystal structure.  It deals with the greatest portion of data in the computational pipeline.  Automation level of an image processing software is important to the throughput of the entire process.  Precognition utilizes some pattern recognition techniques to reduce user intervention as much as possible, and therefore enhance the automation level.

 

Success of crystal structure analysis and clarity of the resulting structure critically depend on the quality of the observed structure factor amplitudes.  The result of diffraction image processing is a precognition of the outcome of the entire crystal structure analysis.  Various new theories are applied and new techniques are implemented to improve the quality of diffraction image processing and data reduction.

 

Precognition heavily relies on CCLÔ/CPLÔ (Crystallographic Concept LibraryÔ and Crystallographic Protocol LibraryÔ) of Renz Research, Inc., and utilizes a wide range of public-domain software components.  See 1.4 for details.  Precognition is mainly written in object-oriented C++ and Python languages with small portion of the code in C and FORTRAN.  All these languages are highly integrated.

 

Precognition can be interactively command-driven, or run through preprogrammed command scripts or a graphical user interface.  It can also be used as a small set of application program interface in forms of Python modules and classes that can be incorporated into other software systems.  These styles of user interface may also be mixed in one program, and the choice of style depends on the functionality of a specific program.

 

The current release focuses on Laue diffraction of single crystals by polychromatic X-rays.  Other areas of application are under development.

 

1.2     Edition, Version, Release, and Patch

 

The name of a command-driven executable is Porgram_E´.´.´_P´.´, where Program can be Precognition or Epinorm.  The letter E indicates an edition for small molecule, protein, virus crystals, etc.; three numbers ´ are version, release, and patch numbers, respectively.  The letter P indicates platform/operating system, and the other numbers show its version.  Precognition.py is the GUI version, which includes many additional functionalities that rely on graphics.  Version number of Precognition.py can be found from Help/About menu.

 

 

Figure 1.2.0.0.1 About Precognition.py.

 

A module of CPL called cpl.Precognition contains the major functions of the system.  This module can be incorporated into other software systems.  Version number of CPL is independent of Precognition and Precognition.py.  It is shown when CPL is imported.

 

1.3     Major Functionalities and Features

 

1.3.1 Major functionalities of Precognition, a pattern recognition software for Laue diffraction

 

·        Auto-indexing of Laue diffraction pattern

·        Geometric refinement of Laue pattern

·        Profile fitting of diffraction spots

·        Deconvolution of spatial-overlapping spots

·        Integration of diffraction spots

 

1.3.2 Major functionalities of Epinorm, a wavelength normalization and data scaling software for Laue diffraction

 

·        Wavelength normalization

·        Data scaling

·        Harmonic overlap deconvolution

·        Data reduction of partial reflections due to finite bandwidth

·        Data merging

 

1.3.3 Major features

 

·        Soft limits and initial source spectrum can be recognized before and after indexing of a pattern.

·        Indexing consistency check is available for re-indexing in middle of a set.

·        Residual pattern recognition (RPR) in geometry refinement.  Residual pattern consists of all residual vectors from predicted spot locations to observed ones.  This pattern varies according to the remaining errors in the parameters, and shrinks to a small Gaussian distribution when refinement is done well.  Precognition automatically recognizes residual pattern using the newly developed RPR technique, makes geometry refinement more robust.

·        A new refinement mode is added for solution of frequent crystal jumping.

·        Random walk-around of crystal orientation is available.  Number of steps and radius can be automatically adjusted.

·        Uncertainties of detector parameters can be input by user, so that freedom of these parameters are limited.

·        Manual geometry refinement is possible.

·        Geometric parameters can be fixed or freed during refinement.

·        Calibration of distance and cell constants after geometric refinement is possible.

·        Many integration modes are available.

·        Analytical profile fitting works with a numerical compensation, which improves the accuracy of integration for strong spots.

·        Negative integrated intensity becomes possible.

·        Connection to the older program LaueView is no longer the default, but still possible by explicit command option.

·        Initial values of scaling parameters can be set.

·        Save and restore intermediate results of wavelength normalization and scaling.

·        Temperature factors can be fixed without refinement during scaling.

·        Spike removal in l-curve.

·        l-curve becomes l-surface.  Wavelength normalization factor not only depends on wavelength, but also mosaicity and Bragg angle.

·        Frame-specific l-curves are possible.

·        Crystal mosaicity is refined.

·        Partials in Laue diffraction due to a finite energy bandpass, called energy partial, are reduced.

·        Negative integrated intensity may contribute to scaling and be included in final results.

·        Harmonic deconvolution is available.

·        Crystal mosaicity, energy resolution and energy partial of harmonic reflection are completely modeled in harmonic deconvolution.

·        Extremely-close spatial overlaps can be deconvoluted by the same algorithm as harmonic deconvolution.

·        All effects of energy partial are completely modeled in deconvolution of extremely-close spatial overlaps.

·        Friedel pairs can be separated during data scaling and/or merging if anomalous scattering should be considered.

 

1.4     A Tour of This Book

 

This manual has three parts, a user guide at the beginning of this book, reference in the middle, and some tutorials at the end.  This very first chapter collects discussions on general issues independent of each specific topic, such as conventions used throughout the book and software installation.

 

The user guide is split into several chapters.  It was written with average readers in mind.  One should learn the basics of X-ray diffraction from other sources, and have some experience of diffraction image processing.  I tried to insert some boxes of background information, and will continue to do so throughout future revisions, but by no means these theoretical background are complete.

 

Processing of diffraction images from single crystals is always an indexing-integration-scaling dance, although Laue diffraction, monochromatic oscillation, and Weissenberg have very different styles.  Chapter 2 is very important, since it sets the scene for the rest.  Many basic input commands and files are common to later chapters.  Given the basic inputs, the central topic in this chapter is indexing of Laue diffraction pattern.  Various different scenarios are discussed, some easy case and others troublesome.

 

Chapter 3 continues to deal with single crystal diffraction geometry, except this time in a refined manner.  Geometric refinement of diffraction pattern aims at precise prediction of diffraction spots on the active surface of detector.  This goal is achieved by correction of crystal lattice and detector parameters, but all geometric.  An additional quantity predicted in case of Laue diffraction is the central wavelength that a reflection is stimulated.  A large portion of this chapter discusses minimization of geometric errors, which are crucial to the next step.

 

Once a precise prediction of all diffraction spots on detector surface is available.  Spot integration is very much independent of diffraction geometry.  In Chapter 4, an array of integration algorithms is discussed, some faster, some more accurate.  All of them fall into two categories, summation and profile fitting.  Laue diffraction images demand more sophisticated algorithms to deal with the commonly occurring spatial overlaps.

 

Chapter 5 covers scaling, wavelength normalization, harmonic deconvolution, and deconvolution of extremely close spatial overlaps.  These apparent complex issues are essentially data reduction and merging from integrated intensities to structure factor amplitudes, the final form of data to be used in structure determination and refinement.

 

Chapter numbers 6 through 10 are reserved for future use.

 

The reference part is in Chapter 11.  This part is meant to be the definite description of each and every program, command, argument, and file format of the software.  This part is inevitably harder to read, and may contain information for programmers only.

 

The rest chapters are examples that show many details of the usage.  I arranged these chapters from easy to hard.  A beginner would like to follow these examples to get a jump start, however, it would be a good idea to check back the relevant parts in the user guide and reference, since better explanations might be found in these chapters.  This book, all testing data, command scripts, and log files are available at http://renzresearch.com/Precognition.

 

1.5     Conventions Used

 

1.5.1 Notations

 

s, S

scalars

f, F

scalar functions

c, C

complexes; c = a + ib

c = |c|

amplitude of a complex

cc, Cc

complex conjugates, if c=a + ib, cc = a - ib

v, V

vectors; v = (a, b, c)

vectors;  = (-a, -b, -c) if v = (a, b, c)

m, M

matrices

p, P

geometric points

(C, r)

circle with center C and radius r

|OP|

distance from point O to point P

ÐABC

angle

courier

code, program, command, argument, filename, path, URL, etc.

 

Table 1.5.1.0.1 Notations used.

 

1.6     Related Software

 

Precognition is an end-user application software written with CCLÔ/CPLÔ (Crystallographic Concept/Protocol LibrariesÔ), which in turn rely on many public domain software components.  This section describes where to obtain these software and how to install them.  This section provides a reference to these libraries and tools.  General users do not have to install all of them.

 

1.6.1 GCC

 

CCL, CPL and Precognition are compiled by GCC or GNU Compiler Collection (http://gcc.gnu.org).  The current release has not yet been tested by other compilers.

 

GCC can be obtained from ftp://ftp.gnu.org/gnu/gcc/gcc-3.4.1.  First, download gcc-3.4.1.tar.gz to a local hard drive.  Second, uncompress and untar the file by command

 

tar xzvf [path/]gcc-3.4.1.tar.gz

 

in a directory where GCC will be installed, usually recommended in /usr/local or /usr/local/rri/pub, where [path/] means optional.  On some systems, one may need two commands of gunzip and tar.  The GCC top directory, e.g., /usr/local/gcc-3.4.1, will be created.  In the top directory, type command

 

./configure [--prefix=`pwd`]

            [--enable-languages=c,c++,f77]

 

If the option is used, GCC will be installed in its top directory, otherwise, in /usr/local.  In most cases, the latter is recommended, but if multiple versions of GCC are needed, their top directories can be the choice.  If configure goes well, type make in the top directory to build the package.  This step will take a while.  Then type make install to install the files.

 

1.6.2 Python

 

Both CCL and CPL depend on the Python interpreter (http://python.org).  Although CCL is written in C++, it embeds some CPL components.  Python distribution Python-2.3.4.tgz can be downloaded from http://python.org/2.3.4.  The installation is exactly same as that of GCC, except the option of --enable-languages.  If it is desired to replace the existing Python version on your system, use /usr in the prefix option.  The documents are at http://python.org/ftp/python/doc/2.3.4/pdf-leter-2.3.4.tgz.

 

1.6.3 Numeric/Numarray

 

Numeric package provides multidimensional arrays.  This package is in transition to a new generation of package Numarray.  The latest distribution can be found at http://www.numpy.org.  Download Numeric-23.6.tar.gz and unpack it in a directory, say, /usr/local/Python-2.3.4/Extensions or simply /usr/local.  If the directory Extensions does not exist, make such directory.  A new directory Numeric-23.6 will be created upon unpacking.  In this top level directory, execute:

 

python setup.py install

 

Make sure that the python command is the very version Numeric is intended to be installed into.  Installation to one Python version will not be available to other versions on the same system.

 

Note: setup.py was modified to set use_dotblas to false:

 

library_dirs_list = []

libraries_list    = []

use_dotblas       = 0

include_dirs      = []

 

1.6.4 PIL

 

Python Imaging Library (PIL; http://www.pythonware.com/products/pil) adds image processing capability to Python.  The latest distribution of PIL Imaging-1.1.4.tar.gz can be downloaded from http://effbot.org/downloads.  Installation of PIL must be done after those of GCC and Python.  Change your working directory to where Python is installed, e.g., /usr/local/Python-2.3.4.  Make a directory Extensions, if there is not yet one.  In the directory Extensions, unpack PIL distribution using command:

 

tar xzvf [path/]Imaging-1.1.4.tar.gz

 

A new directory Imaging-1.1.4 will be created.  Move into Imaging-1.1.4/libImaging, and run the following configuration and make commands:

 

./configure

make

 

After these are done, move back to Imaging-1.1.4, and run:

 

python setup.py build

python setup.py install

 

Before the last command of installation, make sure the python command is indeed what you intend to use.  PIL installed by one python command will not be available for other Python releases on the same machine.

 

Note: error occurred during python setup.py build.  To solve the problem, two lines were inserted into Imaging-1.1.4/_imagingft.c as suggested in freetype.h:

 

#include <ft2build.h>  // insert

#include FT_FREETYPE_H // insert

#include <freetype/freetype.h>

 

1.6.5 Pmw

 

Pmw, Python Megawidgets (http://pmw.sourceforge.net), is used to create GUI components.  Pmw.1.2.tar.gz can be obtained from Source Forge.  Unpacking of this file in a directory, say, /usr/local/Python-2.3.4/Extensions or simply /usr/local, creates a new directory Pmw.  The parent directory of Pmw should be added to the environment variable PYTHONPATH before using it.  An alternative is to make a symbolic link of Pmw to /usr/local/rri/pub.

 

1.6.6 SWIG

 

SWIG, Simplified Wrapper and Interface Generator (http://www.swig.org), is used to wrap CCL into several modules of CPL.  Several SWIG shared libraries are required at runtime, even the user does not rebuild CPL.  Installation of SWIG is similar to that of GCC, except that one must run:

 

make

make –k runtime

make install-runtime

 

swig-1.3.21.tar.gz can be obtained from http://www.swig.org/download.html.

 

Note: swig-1.3.22.tar.gz is available, but did not work for me.

 

1.6.7 FFTW

 

Fast Fourier Transform in the West (FFTW, http://fftw.org) package is used by CCL and in turn by CPL.  Its shared libraries are required at runtime.  fftw-3.0.1.tar.gz can be downloaded from http://fftw.org/download.html.  Its installation procedure is as described in 1.6.1 GCC.

 

1.6.8 TNT

 

Template Numerical Toolkit (TNT, http://math.nist.gov/tnt) is a library contains only herder files.  It is required only when CCL and CPL are built.  First, download tnt094.zip from http://math.nist.gov/tnt/download.html, then type command

 

unzip [path/]tnt094.zip

 

in the directory where TNT will be installed.  /usr/local/include is recommended.  A subdiredctory tnt will be created, which contains a set of header files.

 

1.6.9 LAPACK and BLAS

 

Linear Algebra PACKage (LAPACK, http://www.netlib.org/lapack) and Basic Linear Algebra Subprograms (BLAS, http://www.netlib.org/blas) are low-level libraries used by CCL and CPL.  Rpms are available from http://www.netlib.org/lapack/rpms.  First, download the following rpm files to a local harddrive:

 

lapack-3_0-2_src.rpm

lapack-3_0-2_i386.rpm

lapack-man-3_0-2_i386.rpm

blas-3_0-2_i386.rpm

blas-man-3_0-2_i386.rpm

 

then use the command rpm –i [--prefix=path] package.rpm to install package in the directory path.  /usr/local is recommended.  By default, it installs to /usr.

 

1.6.10 GRACE

 

GRACE is a popular 2-dimantional plotting software that can be obtained from http://plasma-gate.weizmann.ac.il/Grace.  It is used to generate graphs to show some results from Precognition.

 

You may also download all the related rpms from http://www.if.usp.br/pub/unix/x11/xmgrace.  Then install these packages:

 

rpm -ivh pdflib-4.0.3-1.i386.rpm

rpm -ivh pdflib-devel-4.0.3-1.i386.rpm

rpm -ivh pdflib-perl-4.0.3-1.i386.rpm

rpm -ivh pdflib-python-4.0.3-1.i386.rpm

rpm -ivh pdflib-tcl-4.0.3-1.i386.rpm

 

rpm -ivh netcdf-3.5.0-1.i386.rpm

 

rpm -ivh fftw-2.1.3-1.i386.rpm

rpm -ivh fftw-devel-2.1.3-1.i386.rpm

 

The last two steps installing FFTW may not be necessary, if it is already installed as described in 1.6.4.  Finally, install grace

 

rpm -ivh grace-5.1.9-1.i386.rpm

 

Alternatively, grace-5.99.0.tar.gz can be found at ftp://plasma-gate.weizmann.ac.il/pub/grace/src/grace6.  Unpack and configure like this:

 

./configure --prefix=/usr/local

 

Here the option --prefix must be used.  Then make and make install, but still have to link /usr/local/grace/bin/xmgrace-5.99.0 to /usr/local/bin.

 

I am moving from GRACE to Gnuplot, but currently some utilities still depend on GRACE.

 

1.6.11 Gnuplot

 

Gnuplot (http://www.gnuplot.info) is currently optional.  Future releases of our GUI programs may use gnuplot.py as a graphic module.  Download and unpack gnuplot-4.0.0.tar.gz in directory /usr/local.  To install gnuplot, follow the procedure described in 1.4.1.  To install gnuplot.py, download and unpack gnuplot-py-1.7.tar.gz in directory /usr/local/Python-2.3.4/Extensions, move into the new directory gnuplot-py-1.7, and run command

 

python setup.py install

 

http://t16web.lanl.gov/Kawano/gnuplot/index-e.html is a great source of help.

 

1.7     Installation and Execution

 

1.7.1  Program Installation

 

Pre-compiled executables and shared libraries are distributed.  Uncompress and unpack the distribution tar-ball rriyyyymmdd.tgz at a desired location, e.g., /usr/local.  A new directory rriyyyymmdd will be created.  Make a symbolic link of this directory to rri.  Several environment variables must be set before appropriate execution.  Assume that the software system is installed in /usr/local/rri or it is a symbolic link to the top directory, the following lines shall be added to the .tcshrc file or equivalent:

 

setenv RRILICENSE   /home/renz/license.rri

setenv CCLHOME      /usr/local/rri/ccl

setenv CPLHOME      /usr/local/rri/cpl

setenv PRECOGNITION /usr/local/rri/gpr

setenv PYTHONPATH\

   "${CPLHOME}/..:${CPLHOME}:${CPLHOME}/ptr:\

    ${CPLHOME}/std:${CPLHOME}/mac:${CPLHOME}/ccl"

setenv LD_LIBRARY_PATH /usr/local/rri/pub/lib

 

Listing 1.7.1.0.1 Environment variables to be set.

 

In addition, the environment variable PATH shall include /usr/local/rri/bin and /usr/local/rri/pub/bin to allow access to the executables.  If a compatible version of Python (see 1.6.2) has been installed on your host, the last path is unnecessary.

 

1.7.2  License Installation

 

In the list above, RRILICENSE shall point to a license file in each user’s home directory.  The user’s account must have read and write permission on the license file.  To obtain a new license file, start a Python session after installation and setup of the environment variables, type the following commands in Python:

 

% python

Python 2.2.2 (#1, Feb 10 2003, 00:23:53)

[GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import cpl

 

|||)) Welcome to CPL M0.12.0

|||\\ Renz Research, Inc.

 

>>> cpl.license.getinfo()

 

SOURCE: cpl.license.getinfo

TYPE:   result

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To obtain a license on this host, please send file info.rri to Renz Research.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

Listing 1.7.2.0.1 Obtaining a host information file.

 

A file info.rri containing some information of the host is created.  Send this file to us to request a license file (renz@renzresearch.com).

 

If you have multiple computers, you may supply a filename other than the default to the function getinfo(), such as getinfo(’myComputer.rri’).  Please note that the filename must be a quoted string.

 

1.7.3  Execution

 

If you can generate the information file as described above, CPL is installed appropriately.  Most frequently, problems arise due to that some related software installed are inconsistent with the pre-compiled distribution.  See 1.6 Related Software for details.

 

If you see the copyright notice and RRI logo when launching a program, installation is correct and complete.  Again, frequent problems may be missing shared libraries or invalid license file.  Please note that the name of a specific executable program is subject to change, when the system evolves.  The following is an example of such executable program:

 

% precognition_S1.0.6

                                  _

                                 |_)

                                 | recognition

 

                                Release S1.0.6

                              all rights reserved

                                   *   *   *

                                John Zhong Ren

                              Renz Research, Inc.

                  P. O. Box 605, Westmont, IL 60559, U. S. A.

                         e-mail renz@renzresearch.com

                              tel (630) 230-0272

                              fax (435) 514-2645

 

 

|||)) Welcome to CPL M0.12.0

|||\\ Renz Research, Inc.

 

 

DPR: Input   I: File & keyboard input (e.g., image & parameters)

     Output  O: Save result in files

     Quit    Q: Exit DPR (default)

DPR:

 

Listing 1.7.3.0.1 Launching Precognition.

 

Each program in the system may take a set of command-line options.  These options will be discussed later.

 

A command-driven program in the system may support some global commands that can be executed at any level throughout the program.  The global commands are usually used to alter the general behavior of the program, for example, to screen various types of messages.

 

A sequence of commands for a command-driven program may be collected into an input file and fed to the program at once.  The input file can be an argument when the program is started:

 

% precognition_S1.0.6 dpr.inp

 

where dpr.inp is script of input commands.  See below for more detail.  An alternative is to load the comman