As there are many instrument vendors, there are many different file formats for spectral data. This approach was supported vendors who developed the instruments and software to operate them; thus, in order to go back and view spectra you need to use their proprietary software – which of course gets updated on a regular basis and generates revenue. However, the majority of software does have an export format for the data (and maybe some of the spectral metadata) that can be directly imported into Excel or other applications because this has been a need for users for many years. This is typically as Tab-Delimited Text (.txt file) or Comma Separate Variable (.csv file) files.
Over time many scientists have lamented (even complained) that it would be much better if all of the data collected on instruments were stored in a common format that would make it easy to view and share data. While .txt and .csv files can be used for both activities, there has always been a need for a file format that supports both the spectral data and the associated metadata that describes the instrument type and settings, samples, analyte(s) and any other contextual information that gives the data value. Historically, this has been achieved using the following specifications (what are listed below are major specifications with formal standards – there are others that are less formally defined).
ANalytical Data Interchange (ANDI)
This is for mass spectrometry and chromatography data and is described by the ASTM Standards E1947 – Mass Spectrometry (http://www.astm.org/Standards/E1947.htm) and E1947 – Chromatography (http://www.astm.org/Standards/E1948.htm). ANDI uses the Network Common Data Form (NetCDF) self describing data format that is generically defined to store array-oriented scientific data (http://www.unidata.ucar.edu/software/netcdf/).
Crystallographic Information Framework (CIF)
Used for crystallographic data only. This format is less traditional in its data organization than other formats because it not only contains x-ray crystallography patterns but a large, and very well organized, array of metadata about the system under study. This text format is a fundamental standard in the discipline of crystallography and authors of new crystallographic work must provide data in this format to get their work published. (http://www.iucr.org/resources/cif)
The Joint Committee on Atomic and Molecular Physical Data – Data Exchange format (JCAMP-DX)
For UV/Visible Spectrophotometry, Infrared, Mass Spectrometry, Nuclear Magnetic Resonance, and Electron Spin Resonance data. JCAMP-DX is currently administered by the International Union of Pure and Applied Chemistry (IUPAC) at http://www.jcamp-dx.org, although the format has not been actively updated in almost 10 years. Although JCAMP-DX has not formally been standardized, it is currently the de facto standard for sharing spectral data and all the major databases store their data in the format.
JCAMP-DX is a file specification consisting of a number of LABELLED-DATA-RECORDs or LDRs. These are defined to allow reporting of spectral metadata and raw/processed instrument data. The table below outlines some of the main LDRs in JCAMP-DX.
An example JCAMP-DX file is shown below.
##TITLE= Cholesterol (pktab1.jdx)
##JCAMP-DX= 5 $$home made
##DATA TYPE= MASS SPECTRUM
##DATA CLASS= PEAK TABLE
##ORIGIN= Dept of Chem, UWI, Mona, JAMAICA
##OWNER= public domain
##$URL= http://wwwchem.uwimona.edu.jm:1104/spectra/testdata/index.html
##SPECTROMETER/DATA SYSTEM= Finnigan
##INSTRUMENTAL PARAMETERS= LOW RESOLUTION
##.SPECTROMETER TYPE= TRAP
##.INLET= GC
##.IONIZATION MODE= EI+
##XUNITS= m/z
##YUNITS= relative abundance
##XFACTOR= 1
##YFACTOR= 1
##FIRSTX= 0
##LASTX= 386
##NPOINTS= 46
##FIRSTY= 0
##PEAK TABLE= (XY..XY)
0,0
41,520 43,1000 55,630 67,417 69,404 79,544 81,906 91,685 95,772 105,801
107,685 119,439 121,515 133,468 135,461 145,760 147,571 159,529 161,386
173,249 185,150 187,149 199,216 213,454 215,130 228,122 229,187 231,150
247,568 255,378 260,165 261,106 275,306 287,24 297,22 301,207 314,67
325,65 328,20 339,19 353,354 368,791 369,262 371,140 386,324
##END=
Note the LDRs for XFACTOR and YFACTOR. Although in the above example these LDRs are both 1 (because the raw data is already integers), these factors are commonly used to represent numeric values with a large number of decimal places as integers so that only one or two numbers (the factors) have to be stored as decimal values. This means that storage of rounded numbers is minimized and any error incurred because of rounding is applied evenly to all the data points. An example of data in this format is below.
##YFACTOR= 9.5367E-7
…
##XYDATA= (X++(Y..Y))
4400 68068800 68092800 68145600 68100800 68140800 68232000
4394 68304000 68316800 68195200 68152000 68182400 68176000
4388 68240000 68252800 68156800 68156800 68236800 68292800
4382 68302400 68265600 68233600 68214400 68224000 68284800
4376 68353600 68334400 68219200 68230400 68315200 68276800
4370 68259200 68264000 68257600 68316800 68292800 68339200
The data in a JCAMP-DX file can be all be compressed using a number of different human readable compression formats. This capability was added in order that spectral files were not to large for the storage media available (see above). There are four formats for compression, called ASCII Squeezed Difference Format (ASDF) outlined in one of the original articles on the JCAMP-DX format (http://old.iupac.org/jcamp/protocols/dxir01.pdf). These are listed below and use the letters as pseudo-digits.
|
Pseudo-digits for ASDF Formats |
|
|
1. ASCII digits |
0 1 2 3 4 5 6 7 8 9 |
|
2. Positive SQZ digits |
@ A B C D E F G H I |
|
3. Negative SQZ digits |
a b c d e f g h i |
|
4. Positive DIF digits |
% J K L M N O P Q R |
|
5. Negative DIF digits |
j k I m n o p q r |
|
6. Positive DUP digits |
S T U V W X Y Z s |
Note: The above characters replace the leading digit, sign, and preceding space for SQZ, DIF, and DUP forms.
The remaining digits of a multi-digit number are standard ASCII.
|
Simple Examples of ASDF Formats |
|
|
FIX Form: (22 chars) |
1 2 3 3 2 1 0 -1 -2 -3 |
|
PAC Form: (19 chars) or: |
1+2+3+3+2+1+0-1-2-3 1 2 3 3 2 1 0-1-2-3 |
|
SQZ Form: (10 chars) |
1BCCBA@abc |
|
DIF Form: (10 chars) |
1JJ%jjjjjj |
|
DIFDUP Form: (7 chars) |
1JT%jX |
As can be seen, the best compression is obtained by using the DIFDUP format which results in a 22/7 or 3.14x compression of the data. However, this is not always the amount of compression that will get because it depends on how variable the data points are in the file and it is important to choose the right format to get the best compression.
In addition to these existing formats there are a couple of newer formats that will eventually replace the JCAMP-DX specification.
Analytical Information Markup Language (AnIML)
For all spectral data. AnIML is an eXtensible Markup Language (XML) format for storing instrument data and metadata under development since 2004 (http://animl.sourceforge.net). The specification is being coordinated under the American Standards and Testing of Materials (ASTM) E13.15 committee on analytical data.
It was recognized in 2004 that there needed to be a successor to JCAMP-DX because of i) advances in technology, ii) a recognized need to represent data from all analytical techniques, and iii) issues with variants of JCAMP-DX that made interoperability of the files difficult. AnIML files consist of up to four data sections; SampleSet, ExperimentStepSet, AuditTrailEntrySet, and SignatureSet. By design very little data/metadata is required so that legacy data, which may not have much or any metadata to describe it, can be stored in the AnIML format. An example ‘minimum’ AnIML file is shown below:
Allotrope Document Format
For all analytical data. The Allotrope Foundation was formed in 2012 from seven (now twelve) pharma companies around the notion of changing instrument data standards so that they were uniform across all instrument vendors. Three years later, Allotrope has just made available the first version of the Allotrope Document Format (ADF) that is based on the HD5 format for managing and storing data (https://www.hdfgroup.org/HDF5/). The ADF format stores data and metadata from the entire laboratory process (not just the instrument data) and is arrange in layers for metadata (using controlled vocabularies and ontologies), data, and linked files. The specification is so new that it has not yet been released to the community, but should be out by the end of the year. See http://www.allotrope.org.
Other spectral file formats
Lists of file formats for the following analytical techniques can be found at