Cambridge University researchers have accelerated the analysis of imaging data using utility computing and data warehousing technology.
The Cambridge Astronomical Survey Unit (CASU), part of Cambridge University's Institute of Astronomy is speeding the processing and classification of large volumes of research data using Kognitio’s WX2 analytical database alongside utility computing services from HP.
By the end of 2007 CASU will be collecting up to 500GB of telescope imaging data every night. These images will be grouped into catalogues that must be uploaded, cross-referenced and heavily indexed to enable research astronomers to ask useful and complex data mining queries. HP Flexible Computing Services allows CASU to tap into HP’s computing power on a utility basis to keep hardware costs under control.
As each catalogue contains billions of objects this process can take months if carried out using traditional technologies. The scientific trend to make all astronomical databases available on the internet to allow researchers to mine high-quality data, places additional pressure on the catalogue and archiving facilities.
“It is analysis of this type of data that has led to some of the most significant astronomical discoveries but since the speed of research is closely tied to the speed the data can be catalogued and mined we were not proceeding at a satisfactory pace. Gathering, classifying and cross-referencing very large quantities of data is a highly demanding process and it requires a powerful yet flexible technology,” said Jim Lewis, senior research associate at CASU.
He said the implementation had added a new dimension to the unit’s data analysis approach as well as accelerating its productivity. “The data warehouse is no longer a necessary burden but an invaluable asset for us now,” said Lewis.