Share Email Print

Spie Press Book

Optical Satellite Data Compression and Implementation
Author(s): Shen-En Qian
Format Member Price Non-Member Price

Book Description

This book provides a global review of optical satellite image and data compression theories, algorithms, and system implementations. Consisting of nine chapters, it describes a variety of lossless and near-lossless data-compression techniques and three international satellite-data-compression standards. The author shares his firsthand experience and research results in developing novel satellite-data-compression techniques for both onboard and on-ground use, user assessments of the impact that data compression has on satellite data applications, building hardware compression systems, and optimizing and deploying systems. Written with both postgraduate students and advanced professionals in mind, this handbook addresses important issues of satellite data compression and implementation, and it presents an end-to-end treatment of data compression technology.

Book Details

Date Published: 13 November 2013
Pages: 416
ISBN: 9780819497871
Volume: PM241

Table of Contents
SHOW Table of Contents | HIDE Table of Contents

List of Terms and Acronyms

1 Needs for Data Compression and Image Quality Metrics
1.1 Needs for Satellite Data Compression
1.2 Quality Metrics of Satellite Images
1.3 Full-Reference Metrics
     1.3.1 Conventional full-reference metrics
     1.3.2 Perceived-visual-quality-based full-reference metrics
1.4 Reduced-Reference Metrics
     1.4.1 Four RR metrics for spatial-resolution-enhanced images
     1.4.2 RR metric using the wavelet-domain natural-image statistic model
1.5 No-Reference Metrics
     1.5.1 Statistic-based methods
     1.5.2 NR metric for compressed images using JPEG
     1.5.3 NR metric for pan-sharpened multispectral image

2 Lossless Satellite Data Compression
2.1 Introduction
2.2 Review of Lossless Satellite Data Compression
     2.2.1 Prediction-based methods
     2.2.2 Transform-based methods
2.3 Entropy Encoders
     2.3.1 Adaptive arithmetic coding
     2.3.2 Golomb coding
     2.3.3 Exponential-Golomb coding
     2.3.4 Golomb power-of-two coding
2.4 Predictors for Hyperspectral Datacubes
     2.4.1 1D nearest-neighboring predictor
     2.4.2 2D/3D predictors
     2.4.3 Predictors within a focal plane image
     2.4.4 Adaptive selection of predictor
     2.4.5 Experimental results of the predictors
2.5 Lookup-Table-Based Prediction Methods
     2.5.1 Single-lookup-table prediction
     2.5.2 Locally averaged, interband-scaling LUT prediction
     2.5.3 Quantized-index LUT prediction
     2.5.4 Multiband LUT prediction
2.6 Vector-Quantization-Based Prediction Methods
     2.6.1 Linear prediction
     2.6.2 Grouping based on bit-length
     2.6.3 Vector quantization with precomputed codebooks
     2.6.4 Optimal bit allocation
     2.6.5 Entropy coding
2.7 Band Reordering
2.8 Transform-Based Lossless Compression Using the KLT and DCT
2.9 Wavelet-Transform-Based Methods
     2.9.1 Wavelet decomposition structure
     2.9.2 Lossy-to-lossless compression: 3D set-partitioning embedded block
     2.9.3 Lossy-to-lossless compression: 3D embedded zeroblock coding

3 International Standards for Spacecraft Data Compression
3.1 CCSDS and Three Data Compression Standards
3.2 Lossless Data Compression Standard
     3.2.1 Preprocessor
     3.2.2 Adaptive entropy encoder
     3.2.3 Performance evaluation
3.3 Image Data Compression Standard
     3.3.1 Features of the standard
     3.3.2 IDC compressor
     3.3.3 Selection of compression options and parameters
     3.3.4 Performance evaluation
3.4 Lossless Multispectral/Hyperspectral Compression Standard
     3.4.1 Compressor composition
     3.4.2 Adaptive linear predictor
     3.4.3 Encoder
     3.4.4 Performance evaluation

4 Vector Quantization Data Compression
4.1 Concept of Vector Quantization Compression
4.2 Review of Conventional Fast Vector Quantization Algorithms
4.3 Fast Vector-Quantization Algorithm Based on Improved Distance to MDP
     4.3.1 Analysis of the generalized Lloyd algorithm for fast training
     4.3.2 Fast training based on improved distance to MDP
     4.3.3 Experimental results
     4.3.4 Assessment of preservation of spectral information
4.4 Fast Vector Quantization Based on Searching Nearest Partition Sets
     4.4.1 Nearest partition sets
     4.4.2 Upper-triangle matrix of distances
     4.4.3 p-least sorting
     4.4.4 Determination of NPS sizes
     4.4.5 Two fast VQ search algorithms based on NPSs
     4.4.6 Experimental results
     4.4.7 Comparison with published fast search methods
4.5 3D VQ Compression Using Spectral-Feature-Based Binary Code
     4.5.1 Spectral-feature-based binary coding
     4.5.2 Fast 3D VQ using the SFBBC
     4.5.3 Experimental results of the SFBBC-based VQ compression algorithm
4.6 Correlation Vector Quantization
     4.6.1 Process of CVQ
     4.6.2 Performance of CVQ
4.7 Training a New Codebook for a Dataset to Be Compressed
4.8 Multiple-Subcodebook Algorithm Using Spectral Index
     4.8.1 Spectral indices and scene segmentation
     4.8.2 Methodology of MSCA
     4.8.3 Improvement in processing time
     4.8.4 Experimental results of the MSCA
     4.8.5 MSCA with training set subsampling
     4.8.6 MSCA with training set subsampling plus SFBBC codebook training
     4.8.7 MSCA with training set subsampling plus SFBBC for both codebook training and coding
4.9 Successive Approximation Multistage Vector Quantization
     4.9.1 Compression procedure
     4.9.2 Features
     4.9.3 Test results
4.10 Hierarchical Self-Organizing Cluster Vector Quantization
     4.10.1 Compression procedure
     4.10.2 Features

5 Onboard Near-Lossless Data Compression Techniques
5.1 Near-Lossless Satellite Data Compression
5.2 Cluster SAMVQ
     5.2.1 Organizing continuous data flow into regional datacubes
     5.2.2 Solution for overcoming the blocking effect
     5.2.3 Removing the boundary between adjacent regions
     5.2.4 Attaining a fully redundant regional datacube for preventing data loss in the downlink channel
     5.2.5 Compression performance comparison between SAMVQ and cluster SAMVQ
5.3 Recursive HSOCVQ
     5.3.1 Reuse of codevectors of the previous region to attain a seamless conjunction between regions
     5.3.2 Training codevectors for a current frame and applying them to subsequent frames
     5.3.3 Two schemes of carrying forward reused codevectors trained in the previous region
     5.3.4 Compression performance comparison between baseline and recursive HSOCVQ
5.4 Evaluation of Near-Lossless Performance of SAMVQ and HSOCVQ
     5.4.1 Evaluation method and test dataset
     5.4.2 Evaluation of a single spectrum
     5.4.3 Evaluation of an entire datacube
5.5 Evaluation of SAMVQ with Regard to the Development of International Standards of Spacecraft Data Compression
     5.5.1 CCSDS test datasets
     5.5.2 Test results of hyperspectral datasets
     5.5.3 Compression of multispectral datasets using SAMVQ

6 Optimizing the Performance of Onboard Data Compression
6.1 Introduction
6.2 The Effect of Raw Data Anomalies on Compression Performance
     6.2.1 Anomalies in the raw hyperspectral data
     6.2.2 Effect of spikes on compression performance
     6.2.3 Effect of saturation on compression performance
     6.2.4 Summary of anomaly effects
6.3 The Effect of Preprocessing and Radiometric Conversion on Compression Performance
     6.3.1 Artifacts introduced in preprocessing and radiometric conversion
     6.3.2 Evaluation using crop leaf area index in agriculture applications
     6.3.3 Evaluation using target detection
6.4 The Effect of Radiance-Data Random Noise on Compression Performance
     6.4.1 Data processing procedure
     6.4.2 Evaluation results using statistical measures
     6.4.3 Evaluation results using target detection
6.5 Effect of Keystone and Smile on Compression Performance
6.6 Enhancing the Resilience of Compressed Data to Bit Errors in the Downlink Channel
     6.6.1 Triple-module redundancy used in the header of the codebook and index map
     6.6.2 Convolutional codes
     6.6.3 Viterbi algorithm
     6.6.4 Simulation results

7 Data Compression Engines aboard a Satellite
7.1 Top-Level Topology of Onboard Data Compressors
7.2 Vector Distance Calculators
     7.2.1 Along-spectral-bands vector distance calculator
     7.2.2 Across-spectral-bands vector distance calculator
7.3 Codevector Trainers
     7.3.1 Along-spectral-bands codevector trainer
     7.3.2 Across-spectral-bands codevector trainer
7.4 Vector Quantization Data Compression Engines
7.5 Real-time Onboard Compressor
     7.5.1 Configuration
     7.5.2 Network switch
7.6 Hardware Implementation Process of SAMVQ and HSOCVQ
     7.6.1 Codevector training
     7.6.2 SAMVQ
     7.6.3 HSOCVQ
7.7 Scenario Builder: A Real-Time Data Compression Emulator
     7.7.1 Scenario Builder overview
     7.7.2 Scenario Builder applications
     7.7.3 Architecture and data flow of Scenario Builder
     7.7.4 SORTER engine and cluster SAMVQ compression engine
     7.7.5 Recursive HSOCVQ compression engine
     7.7.6 Scenario Builder products
     7.7.7 Scenario simulation user interface
7.8 Using Scenario Builder to Optimally Design Onboard Data Compressor Architecture
     7.8.1 Parameters of the design
     7.8.2 SORTER as the front-end compressor
     7.8.3 Second-level compressor
     7.8.4 Proposed system

8 User Acceptability Study of Satellite Data Compression
8.1 User Assessment of Compressed Satellite Data
8.2 Double-Blind Test
8.3 Evaluation Criteria
8.4 Evaluation Procedure
8.5 Multidisciplinary Evaluation
     8.5.1 Precision agriculture
     8.5.2 Forest regeneration
     8.5.3 Geology
     8.5.4 Military target detection
     8.5.5 Mineral exploration 1
     8.5.6 Ocean ship and wake detection
     8.5.7 Mineral exploration 2
     8.5.8 Mineral exploration 3
     8.5.9 Civilian target detection
     8.5.10 Forest species classification
     8.5.11 Endmember extraction in mineral exploration
8.6 Overall Assessment Result and Ranking
8.7 Effect of Lossy Data Compression on Retrieval of Red-Edge Indices
     8.7.1 Test datacubes
     8.7.2 Red-edge indices
     8.7.3 Evaluation using red-edge products
     8.7.4 Evaluation results and analysis
     8.7.5 Summary of the evaluation

9 Hyperspectral Image Browser for Online Satellite Data Analysis and Distribution
9.1 Motivation for Web-Based Hyperspectral Image Analysis
9.2 Web-Based Hyperspectral Image Browser and Analysis
9.3 HIBR Functions and Data Flow
     9.3.1 Hyperspectral data compressor
     9.3.2 Hyperspectral catalog web server
     9.3.3 Overall data flow
9.4 User Scenarios
9.5 Hyperspectral Image Browser Operations
     9.5.1 HIBR visualization
     9.5.2 User product search
     9.5.3 User product generation
     9.5.4 HIBR graphical user interface
9.6 Summary


Satellite data compression has been an important subject since the beginning of satellites in orbit, and it has become an even more active research topic. Following technological advancements, the trend of new satellites has led to an increase in spatial, spectral, and radiometric resolution, an extension in wavelength range, and a widening of ground swath to better serve the needs of the user community and decision makers. Data compression is often used as a sound solution to overcome the challenges of handling a tremendous amount of data. I have been working in this area since I was pursing my Ph. D. thesis almost 30 years ago.

Over the last two decades, I - as a senior research scientist and technical authority with the Canadian Space Agency - have led and carried out research and development of innovative data compression technology for optical satellites in collaboration with my colleagues at the agency, other government departments, my postdoctoral visiting fellows, internship students, and engineers at Canadian space industry. I invented and patented two series of near-lossless satellite data compression techniques and led the Canadian industry teams who implemented the techniques and built the onboard near-lossless compressors. I also led a multidisciplinary user team to assess the impact of the near-lossless compression techniques on ultimate satellite data applications. As the representative of Canada, I am an active member of the CCSDS working group for developing international data-compression standards for satellite data systems. Three international satellite data compression standards have been developed by the working group and published by the International Organization for Standardization (ISO). In collaborating with experts in this area in the world, I have co-chaired a SPIE conference on satellite data compression, communication, and signal processing since 2005. I have published over sixty papers and currently hold six U. S. patents, two European patents, and several pending patents in the subjects of satellite data compression and implementation. I feel that I have acquired sufficient knowledge and accumulated plenty experience in this area, and it is worth the effort to systematically organize them and put them into a book.

This book is my attempt to provide an end-to-end treatment of optical satellite data compression and implementation based on 30 years of first-hand experience and research outcomes. (It is a companion text to my book Optical Satellite Signal Processing and Enhancement, published by SPIE Press.) The contents of the book consist of nine chapters that cover a wide range of topics in this field. It serves as an introduction for readers who are willing to learn the basics and the evolution of data compression, and a guide for those working on onboard and ground satellite data compression, data handling and manipulation, and deployment of data-compression subsystems. The material is written to provide clear definitions and precise descriptions for advanced researchers and expert practitioners as well as for beginners. Chapters open with a brief introduction of the subject matter, followed by a review of previous approaches and their shortcomings, a presentation of recent techniques with improved performance, and finally a report on experimental results in order to assess their effectiveness and to provide conclusions.

Chapter 1 is the introduction to the book that describes the rationale and needs for satellite data compression and introduces a set of image quality metrics for assessing compressed satellite images. Chapter 2 presents a review of satellite lossless-data-compression techniques, considering both prediction-based and transform-based methods. Chapter 3 summarizes three international satellite-data-compression standards developed by CCSDS from the perspective of applying the standards. Chapter 4 describes vector quantization (VQ) based data-compression techniques that I have developed for compressing hyperspectral data. The focus of the research was to significantly reduce the computational complexity of conventional VQ algorithms in order for them to effectively compress hyperspectral datacubes. Many innovative yet practical solutions have been developed, including two of my granted patents: Successive Approximation Multistage Vector Quantization (SAMVQ) and Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ). Chapter 5 describes how both of these techniques solve the blocking effect when applied to compressing continuous data flow generated aboard satellites and how they restrict the compression error to a level lower than that of the intrinsic noise of the original data to achieve socalled near-lossless compression. Chapter 6 addresses the optimization and implementation aspects of onboard data compression; aspects include the effect of anomalies of input data on compression performance, the location in the onboard data-processing chain where the compressor should be deployed, and the techniques to enhance error resilience in the data downlink transmission channel. Chapter 7 describes the hardware implementation of compression engines and onboard compressors that are based on SAMVQ and HSOCVQ. Chapter 8 reports a multidisciplinary user-acceptance study that assessed the impact of the compression techniques on various hyperspectral data applications to address the users' concern about possible information loss due to the lossy compression nature of SAMVQ and HSOCVQ. Chapter 9 describes the Hyperspectral Image Browser (HIBR) system, which is capable of remotely displaying large hyperspectral datacubes via the Internet and of quickly processing the datacubes directly on the compressed form for users to identify the interested data, whose richness comes mostly from the spectral information.

There are many people I would like to thank for their contributions to the works included in this book. I would like to thank the Canadian Space Agency, where I have been working for the last 20 years; my colleagues Allan Hollinger, Martin Bergeron, Michael Maszkiewicz, Ian Cunningham, and Davinder Manak for their participation in data compression projects; my postdoctoral visiting fellows Pirouz Zarrinkhat and Charles Serele; and over forty intern students who have each left their mark. I would like to thank Robert Neville (retired), Karl Staenz (now at the University of Lethbridge), and Lixin Sun at the Canada Centre for Remote Sensing for collaborating on the Canadian hyperspectral program; Jos�e L�vesque and Jean-Pierre Ardouin at the Defence Research and Development Canada for their collaboration on assessing the impact of data compression. I thank David Goodenough at the Pacific Forestry Centre; John Miller and Baoxin Hu at York University for providing datasets and for actively collaborating on the data-compression user acceptability study; and Bormin Huang of the Cooperative Institute for Meteorological Satellite Studies at the University of Wisconsin-Madison for his discussion on satellite data compression.

I would also like to thank the participants in the user acceptability study: Andrew Dyk at the Pacific Forestry Centre; Jing Chen at the University of Toronto; Harold Zwick, Dan Williams, Chris Nadeau, and Gordon Jolly at MacDonald Dettwiler Associates; and Benoit Rivard and Jilu Feng at the University of Alberta. I thank Luc Gagnon, William Harvey, Bob Barrette, and Colin Black at MacDonald Dettwiler Associates (former EMS Technologies) for the development and fabrication of onboard compressor prototypes; and Melanie Dutkiewicz and Herbal Tsang for the development of a hyperspectral browser. I thank Valec Szwarc and Mario Caron at the Communication Research Centre (Canada) for discussions on enhancing resilience to bit errors produced by compression techniques; and Peter Oswald and Ron Buckingham for their discussion on onboard data compression. I would also like to thank Penshu Yeh at the NASA Goddard Space Flight Center, Aaron Kiely at the Jet Propulsion Laboratory, Carole Thiebaut and Gilles Moury at the French Space Agency (CNES), and Raffaele Vitulli at the European Space Agency for the collaboration within the CCSDS in developing international spacecraft-data standards and for their contributions to the CCSDS work included in this book.

I would also like to thank the three anonymous manuscript reviewers for their tireless work and strong endorsement of this book, their careful and meticulous chapter-by-chapter review on behalf of SPIE Press, and their detailed comments leading to the improvement and final results of the book in its current form. Many thanks as well to Tim Lamkins, Scott McNeill, and Dara Burrows at SPIE Press for turning my manuscript into this book.

Finally, I would like to thank my wife Nancy and daughter Cynthia for their help and support. They provided great encouragement and assistance during the period I wrote this book. The credit of this book should go to them.

Shen-En Qian
Senior Scientist, Canadian Space Agency
Montreal, Canada

© SPIE. Terms of Use
Back to Top