tsgxr: Loading HyLogger TSG data into xarray
For those working with HyLogger data in TSG format, I've put together an open source Python package tsgxr which allows reading these relatively arcane formats directly into Xarray. tsgxr is available via GitHub and also directly from PyPI.
Over the past few weeks I've restructured the package and added the ability to lazily-load data, meaning that even the largest files should now be able to be opened (if not loaded, necessarily) on even modest hardware.
Note:
tsgxrleans on another package for some of the basic structure/loaders,pytsg, now managed by the Geological Survey of Western Australia but originally written by Ben @ FractalGeoAnalytics and Andrew Rodger (ex-CSIRO). The dependence onpytsghas lessened after writing thexarrayloaders, and some bugfixes intsgxrare yet to make it back topytsg, but they're on the way.
HyLogger Data
HyLogger instruments are used principally for hyperspectral line-scanning of drillcore (typically at about 1cm resolution), from which spectral mineralogy is derived - allowing identification and unmixing of mineral signatures downhole, and later domaining along the drillhole trace ($\pm$ in 3D). The instrument generates a composite dataset including spectral images (for which there are typically two or three hyperspectral cubes - near-infrared, thermal-infrared and with the latest HyLogger instruments also mid-infrared; NIR, TIR and MIR for short), a lidar trace along the centre of the drillcore, and high resolution truecolor imagery of the samples.
The datasets are provided in TSG formats (partly due to the fact that they're processed through TSG), which include one .bip image per hyperspectral sensor, each paried with a .tsg file containing metadata/headers, a lidar trace in .dat and a .bip file containing a concatenated series of compressed JPEGs (i.e., a small JPEG bomb, if you're short on RAM). This latter item has caused issues previously, particularly for longer drillholes (e.g., for one dataset, 6GB of compressed JPEGs would expand to ~220GB), and as such was something I was looking to handle more cleanly.
HyLogger data is typically scanned by state geological surveys across Australia, and a large amoutn of this is provided openly via the National Virtual Core Library (NVCL), supported by AuScope and also available via the AuScope portal.
Recent tsgxr Updates
In the interest of keeping this brief and minimially outdated over time I'll just mention a few things which I think are useful below, and save the rest for the README or to be updated in the package directly.
Using an xarray.DataTree Structure
Newer versions of tsgxr use a xarray.DataTree structure, which enables each of the varied parts of the dataset to be represented (and stored/serialized if desired) within a single heirarchical dataset. Being in xarray, this also allows labelling the relevant physical/semantic dimensions of the dataset, assignment of the respective coordinates, and the inclusion of metadata as attributes. A short overview for an example (the STAVELY_17 hole as provided by the NVCL) is given below:
DT : xarray.DataTree = open_tsg(
"./07e4dcac-5216-44a6-9a6b-0c4c1f7ce7d", index_coord="depth",
)
DT<xarray.DataTree>
Group: /
├── Group: /NIR
│ Dimensions: (depth: 19754, wavelength: 531, feature: 25)
│ Coordinates:
│ * depth (depth) float32 79kB 0.004111 0.004112 ... 156.0
│ sample (depth) uint64 158kB 1 4 2 ... 23233 23276 23251
│ tray (depth) uint16 40kB ...
│ section (depth) uint8 20kB ...
│ hole (depth) object 158kB ...
│ * wavelength (wavelength) float64 4kB 380.0 384.0 ... 2.5e+03
│ band (wavelength) int64 4kB ...
│ * feature (feature) uint8 25B 0 1 2 3 4 ... 20 21 22 23 24
│ Data variables: (12/54)
│ Spectra (depth, wavelength) float32 42MB ...
│ Centres (depth, feature) float32 2MB ...
│ ...
│ Attributes: (12/34)
│ core_qual: ['Void', 'Rubble', 'Crack', 'Core']
│ TSA704_S Minerals: ['Opal', 'Dickite', 'Kaolinite-PX', 'Kaolinite...
│ ...
├── Group: /TIR
│ Dimensions: (depth: 19754, wavelength: 341, feature: 25)
│ Coordinates:
│ * depth (depth) float32 79kB 0.004111 ... 156.0
│ sample (depth) uint64 158kB 1 4 2 ... 23276 23251
│ tray (depth) uint16 40kB ...
│ section (depth) uint8 20kB ...
│ hole (depth) object 158kB ...
│ * wavelength (wavelength) float64 3kB 6e+03 ... 1.45e+04
│ band (wavelength) int64 3kB ...
│ * feature (feature) uint8 25B 0 1 2 3 4 ... 21 22 23 24
│ Data variables: (12/47)
│ Spectra (depth, wavelength) float32 27MB ...
│ Centres (depth, feature) float32 2MB ...
│ ...
│ Attributes: (12/28)
│ TSA703_T Groups: ['SILICA', 'K-FELDSPAR', 'PLAGIOCLASE', 'GARNE...
│ ...
├── Group: /Lidar
│ Dimensions: (depth: 19754)
│ Coordinates:
│ * depth (depth) float32 79kB 0.004111 0.004112 0.004114 ... 156.0 156.0
│ sample (depth) uint64 158kB 1 4 2 5 0 3 ... 23204 23205 23233 23276 23251
│ tray (depth) uint16 40kB ...
│ section (depth) uint8 20kB ...
│ hole (depth) object 158kB ...
│ Data variables:
│ Lidar (depth) float32 79kB 92.44 75.37 75.98 77.04 ... 2.398 0.9316 58.43
└── Group: /Image
Dimensions: (depth: 2898500, channel: 3, width: 926)
Coordinates:
* depth (depth) float32 12MB 0.004111 0.004177 0.004243 ... 156.0 156.0
section (depth) uint16 6MB 0 0 0 0 0 0 0 0 0 ... 49 49 49 49 49 49 49 49 49
tray (depth) uint16 6MB 0 0 0 0 0 0 0 0 ... 186 186 186 186 186 186 186
* channel (channel) int64 24B 0 1 2
* width (width) float64 7kB -0.03054 -0.03047 -0.0304 ... 0.03047 0.03054
Data variables:
Image (depth, width, channel) uint8 8GB ...Lazy-Loading
tsgxr now includes a series of backends for xarray which allow opening of components of TSG datasets without reading all of the contained data (i.e. only the metadata and some of the smaller components are read at open-time). Notably, when loading data, some limitations of indexing and lazy loading mean that more data than necesary might be loaded, as the depth-slicing is used as the principal axis (this is how the files are arranged; e.g., if slicing across the drillcore width for imagery, or across wavelengths for spectra - all of the data for that depth slice will be loaded, then subset). Some improvements to this may come in the future.
import xarray
ds : xarray.Dataset = xarray.open_dataset(tsgfile, engine="tsg") # loading spectral .bip file lazily
ds : xarray.Dataset = xarray.open_dataset(crasfile, engine="lazycras") # loading truecolor image .bip lazyilyThese interfaces are used internally within tsgxr, such that the key entry point open_tsg() now by-default lazy-loads the components for which this is an option (spectra, truecolor imagery).
Simpler Spectral Products
TSG has some limitations in how it runs its unmixing algorithms, such that for any given pixel along the line scan, only a few mineral species are unmixed by TSA (typically 2 or 3, depending on which spectral sensor is considered). These are provided in long-form in the TSG dataset, where paired records under e.g.(Min1, Grp1, Wt1), (Min2, Grp2, Wt2), represent series of values along the line scan identifying i) the mineral species unmixed (Min1 would be dominant), ii) the group assigned to that mineral (the more general classification, for which we can typically have more confidence) and iii) the unmixing fraction assigned to that mineral/group. tsgxr provides some options for turning this into a nice wide-form (but sparse) table more amenable to statistics and visualization.
Where collapse_products=True is used in the open_tsg() call, the spectral products/scalars will be reorganized into more useful tabluar forms (here e.g sTSAS_Grp, sTSAS_Min, ...) indexed by whichever coordinate used for spectra and the group/mineral as labelled under the respective systems used:
DT : xarray.DataTree = open_tsg(
"./07e4dcac-5216-44a6-9a6b-0c4c1f7ce7d", index_coord="depth", collapse_products=True
)
DT['NIR'].ds : xarray.Dataset<xarray.DatasetView> Size: 57MB
Dimensions: (depth: 19754, wavelength: 531, sTSASgroup: 7,
sTSASmineral: 15, sTSAVgroup: 2,
sTSAVmineral: 2, uTSASgroup: 7,
uTSASmineral: 15, uTSAVgroup: 2,
uTSAVmineral: 2, feature: 25)
Coordinates: (12/16)
* depth (depth) float32 79kB 0.004111 0.004112 ... 156.0
sample (depth) uint64 158kB 1 4 2 ... 23233 23276 23251
tray (depth) uint16 40kB ...
section (depth) uint8 20kB ...
hole (depth) object 158kB ...
* wavelength (wavelength) float64 4kB 380.0 384.0 ... 2.5e+03
... ...
* sTSAVmineral (sTSAVmineral) object 16B 'Goethite' 'Galvanis...
* uTSASgroup (uTSASgroup) object 56B 'KAOLIN' ... 'CARBONATE'
* uTSASmineral (uTSASmineral) object 120B 'Kaolinite-PX' ... ...
* uTSAVgroup (uTSAVgroup) object 16B 'OXIDE' 'NOTAROK'
* uTSAVmineral (uTSAVmineral) object 16B 'Goethite' 'Galvanis...
* feature (feature) uint8 25B 0 1 2 3 4 ... 20 21 22 23 24
Data variables: (12/22)
Spectra (depth, wavelength) float32 42MB ...
sTSAS_Grp (depth, sTSASgroup) float64 1MB ...
sTSAS_Min (depth, sTSASmineral) float64 2MB ...
sTSAV_Grp (depth, sTSAVgroup) float64 316kB ...
sTSAV_Min (depth, sTSAVmineral) float64 316kB ...
uTSAS_Grp (depth, uTSASgroup) float64 1MB ...
...
Attributes: (12/34)
core_qual: ['Void', 'Rubble', 'Crack', 'Core']
TSA704_S Minerals: ['Opal', 'Dickite', 'Kaolinite-PX', 'Kaolinite...
TSA704_V Groups: ['MISC-SILICATE', 'CARBONATE', 'SULPHATE', 'OX...
...The functions achieving this are also available for use more generally: tsgxr.products.products_to_mineral_table(), tsgxr.products.products_to_group_table(). These will take a given spectral dataset, extract the relevant unmixing system, and return a nicely formatted table. They can also perform compositing via suppling a step argument (e.g. where depth is used as the primary index, step=0.5 will return a 50x downsampling to give a 50cm composite), and will return tables where the respective colormap as used within TSG is provided on the attribtes (mapping minerals or groups to hex-labelled colours).
Simple Access to Downhole Plotting
These nice mineralogy tables are also then amenable to simple downhole plotting, and tsgxr.products.plot_product_downhole() will produce stacked bar plots for a specified unmixing system, at a specific level (mineral or group), while optionally providing compositing as desired.