User Manual available online

Finally, the very first version of user manual is available online now under the following link.

eResearch Australasia 2013 presentations

At the eResearch Australasia 2013 conference we carried out half a day workshop on Image Analysis and Processing in the Clouds using Scalable eResearch Workflow-Based System. 

Our presentations are now available online:

• Introduction, here
• Distributed Scientific Computing in Clouds, here
• Galaxy Introduction, here
• Cellular Imaging Tools on NeCTAR Cloud, here
• Cellular Imaging: Neuronal Complexity Workflow, here
• Medical Imaging, here
• CT Reconstruction Workflow, here

We are officially online

The Cloud-based Image Analysis and Processing Toolbox project provides access to existing biomedical image processing and analysis tools via remote user‐interface using the NeCTAR cloud.

If you are interested to use our tools, please contact us.

Input Validation in Galaxy

Input Validation in Galaxy

Just wanted to share, Galaxy has nice input validation features, i.e. the following code generates the following error:

And if more complicated validation is required, you can use validation hook :

And the code file should have the following function:

def validate_input( trans, error_map, param_values, page_param_map ):

To display an error message for particular parameter, set:

error_map['param_name'] = "Some message"

And if you print anything in validate_input(), you get the “Log messages” window in Galaxy UI:

We are at the ICCR 2013

Meet us at the International Conference on the Use of Computers in Radiation Therapy (ICCR2013) which is held in Melbourne Convention and Exhibition Centre (6-9 May 2013). The conference has many interesting themes, including: dose calculation methods, analysis/evaluation of 3D distributions, Monte Carlo modelling, verification imaging including CBCT, segmentation, data mining, image registration, imaging for planning including functional, etc. We display our poster entitled Cloud-Based Workflows Tools Supporting Medical Imaging in the Segmentation section, and here is the link. We are currently in the phase of accepting new pilot users. If you are interested please send us e-mail.

X-TRACT - CT and Imaging tools

X-TRACT - a software for advanced X-ray image analysis and Computed Tomography currently in use on the MASSIVE cluster at the Australian Synchrotron, ANU and at the Shanghai Synchrotron in China. X-TRACT implements a large number of conventional and advanced algorithms for 2D and 3D X-ray image reconstruction and simulation.

Major X-TRACT functionality is now available as part of Cloud-Based Image Analysis and Processing Toolbox. The following features are implemented:

FUNCTION	DESCRIPTION FROM END-USERS PERSPECTIVE
Sinogram creation	X-ray projection data must first be converted into sinograms before CTreconstruction can be carried out. Each sinogram contains data from a single row of detector pixels for each illuminating angles. This data is sufficient for the reconstruction of a single axial slice (at least, in parallel-beam geometry).
Ring artefact removal	Ring artefacts are caused by imperfect detector pixel elements as well as by defects or impurities in the scintillator crystals. Ring artefacts can be reduced by applying various image processing techniques on sinograms or reconstructed images.
Dark current subtraction	Dark current subtraction compensates for the readout noise, ADC offset, and dark current in the detector. The dark current images are collected before and/or after CT measurements with no radiation applied and with the same integration time as the one used during the measurements. The dark current image is subtracted from each CT projection.
Flat field correction	Flat-field images are obtained under the same conditions as the actual CT projections, but without the sample in the beam. They allow one to correct the CT projections for the unevenness of the X-ray illumination.
Positional drift correction	The function is used for correction of transverse drift between related experimental images.  Image drift is assessed by cross-correlating pairs of images.
Data normalisation	Data normalisation
TIE-based phase extraction	The TIE algorithm allows the recovery of the optical phase of an electromagnetic wave (e.g. an X-ray beam) from a single near-field in-line image by solving the Transport of Intensity equation under the assumption that the phase shift and absorption distributions are proportional to each other. This method is usually applied in propagation-based in-line CT imaging (PCI-CT).
FBP CT reconstruction	Filtered back-projection (FBP) parallel-beam CT reconstruction.
Gridrec CT reconstruction	High speed CT reconstruction algorithm.
Centre of rotation	Automated calculation of the centre of sample rotation in a CT scan from experimental X-ray projections, sinograms or reconstructed axial slices.
CT Reconstruction Filters	The choice of available CT reconstruction filters will include at least the Liner-Ramp, Shepp-Logan, Cosine, Hamming and Hann filters.
ROI reconstruction	This option enables the user to select a subset of axial slices to be reconstructed and/or limit the reconstruction area to a user-defined rectangular subarea of the axial slice. The option reduces the reconstruction time and the size of the output data.

 And here's a short video showing the basic usage of X-TRACT in Galaxy cloud:

HCA-Vision Components in Cloud-based Image Analysis and Processing Toolbox Ready to Test

HCA-Vision components in Cloud-based Image Analysis and Processing Toolbox are ready to test. Here is a video clip showing an example of how to build a workflow using some of the tools in the toolbox:

Enjoy using the toolbox and look forward to its release in the near future!

NeCTAR Workshop on Cloud-based Computational Frameworks

Yesterday on the 22nd of April, together with NeCTAR, we have organised the NeCTAR Workshop on Cloud-based Computational Frameworks. Around 30 people arrived to Sydney to share their knowledge, expertise and also to discuss common problems across the NeCTAR projects.

The workshop was held in an anti-conference way, with the agenda prepared interactively. Discussed topics included:

• Demonstrations, Galaxy + CloudMan
• Galaxy for image analysis
• High Throughput Computing
• Storage
• AAA
• Programming Models + Patterns
• Deployments
• Orchestration

During the workshop, we also had also the Marshmallow Challange to enhance collaborative thinking.

Photographs from the workshop can be found under the following link.

MILXView components updated with ITK 4.3.1

The MILXView components of the toolbox: Image registration, segmentation, Partial volume correction (PVC), atlas normalisation (SUVR), cortrical thickness estimation (CTE) and CTE surface have now all be updated to use the latest Insight toolkit (ITK 4.3.1)

Feed Hadoop with a Large Amount of Images for Parallel Processing

We are investigating how to use Hadoop to process a large amount of images in parallel using our image Toolbox in the NeCTAR cloud. One technical requirement is to find out how to feed Hadoop with a set of binary image files?

Hadoop was originally developed for text mining. As a result, the whole design is based on <key, value> pairs as input and output. It is straightforward to use it for text mining for rich software harnesses, examples etc., but not for binary data, such as images. This blog discusses two approaches to this requrement as follows:

1. Using Hadoop SequenceFile

A sequenceFile is a flat file consisting of binary <k, v> pair, which can be directly used as Hadoop's inputs.

Here is an example for generating a SequenceFile from a set of image files in Java:

Here is the execution of the above code:

Then, the generated file can be used to feed Hadoop as <k, v> pair as input for paralel processing.

2. Using HDFS (Hadoop Distributed File System)'s URL:

Instead of feeding Hadoop with data contents directly, we can feed Hadoop a file list, where lists all files HDFS' URI as v of <k, v>. Each mapper uses the assigned <k, v> to load the corresponding data contents (e.g. images) to process as shown as follows:

Here is a simple comparsion between the above two approaches:

Clusters and computational frameworks in the NeCTAR cloud

Many NeCTAR Virtual Laboratory and eResearch Tool projects are working on deploying Clusters in the Cloud (such as CloudMan and StarCluster). Some are also investigating computational frameworks (such as Hadoop) in the cloud.
At NeCTAR's Software Projects Collaboration Workshop (Dec 2012) some projects expressed a desire to share knowledge about such cloud-based computational frameworks.

Aim

This workshop aims to kick-start a NeCTAR Interest Group for sharing knowledge about deploying and using clusters and computational frameworks in the NeCTAR cloud.
Target audience
Software engineers, software architects and technical project managers deploying or using clusters or computational frameworks on the NeCTAR Research Cloud are encouraged to attend.  NeCTAR funded projects working on clusters in the cloud are particularly encouraged to attend.

Agenda

We'll run this workshop as an Unconference/ Open-space where participants will set the agenda and run sessions on the day. Please come along prepared to share examples of how you are building clusters and computational frameworks in the NeCTAR Cloud. As per BarCamp rules there will be NO SPECTATORS, ONLY PARTICIPANTS!

Register HERE

Us at the 9th Annual e-Health Research Colloquium in Brisbane

The Australian e-Health Research centre (AeHRC) develops and deploys leading edge ICT innovations in the healthcare domain. It hosted its 9th Annual e-Health Research Colloquium in Brisbane, on Wednesday 27th March 2013. We presented our poster on MILXCloud – a faster, smarter way to process imaging data in the cloud – at the event.

Galaxy on the Cloud with CloudMan

CloudMan is a tool that facilitates Galaxy deployments in cloud environments. Quoting the brief description from the official web site:

“CloudMan is a cloud manager that orchestrates the steps required to provision and manage compute clusters on a cloud infrastructure, all through a web browser. In minutes, you can use a vanilla SGE cluster or a complete bioinformatics analysis environment.”

As such it was an obvious choice for us to deploy our Galaxy based Image Processing and Analysis Toolkit on the NeCTAR cloud. Although spawning new clusters when the CloudMan is already setup on a cloud it’s quite easy, there is a fair amount of work required to get to this stage, which includes among others:

• creating images and volume snapshot with our customized Galaxy
• setting up the Cloudman/Galaxy configuration files in object storage
• … and contributing to the CloudMan project itself to fix some issues with OpenStack compatibility

The good new is that we have managed to deploy CloudMan and our customized Galaxy with image processing tools  on an OpenStack cloud.  We can now build on-demand Galaxy/SGE (Sun Grid Engine) clusters with up to 20 nodes.

Galaxy in this configuration uses DRMAA specification to submit computational task to SGE to be scheduled for execution on the cluster. That (given access to sufficient cloud resource allocation) let’s us scale the application with increasing number of users.

We have also managed to successfully enable MPI support in the cloud SGE, which we can now use to execute MPI based tools (e.g. some of the X-TRACT components) utilizing the cluster resources for speeding up parallelizable computation.

The CloudMan relies heavily on the use of volumes (cloud block storage) and volume snapshots that are not currently publicly available in the NeCTAR cloud.  The experimental support has been around for while and it seems that production version may be available soon and then we can migrate out deployment to NeCTAR.

Alternatively we are considering modifying CloudMan to work without volumes. 

Neuro-imaging pipelines

We have implemented a number of pipelines as part of our NeCTAR RT035 project. Brief descriptions of the pipelines implemented are described below.

The SUVR tool provides intensity normalisation of PET images for quantitative purposes.

The Registration tool allows the user to perform affine or rigid transforms when registering two images together.

The segmentation tool allows the user to segment a brain for a given MRI image.

Alzheimer’s disease and other neuro degenerative diseases are associated with the loss of Grey matter in the cortex. It is therefore necessary to try and quantify this loss. We use the cortical thickness estimation (CTE) tool to provide us with this analysis.

Overview of main functions of CTE implemented:

• Atlas registration
• Align an atlas image to a target image
• Segmentation
• Segment the MRI into Grey matter (GM), white matter (WM) and cerebrospinal fluid
• Bias Field Correction
• Estimate and remove the noise from the image
• Partial Volume Estimation
• Quantify the amount of partial voluming inside each voxel
• Topology Correction
• Create the topology of the brain to ensure that it is genus zero
• Thickness Estimation
• Compute the thickness of the cortex for each grey matter voxel

Outputs from the CTE (above) are used as inputs to the CTE surface pipeline. The CTE surface transfers all cortical thickness values, generated by the CTE pipeline, to a common template mesh.

Overview of main functions implemented for CTE Surface:

• Cortical Surface Extraction
• Extract a 3D mesh from the brain segmentation
• Topological Correction
• Removes holes and handles from the mesh
• Biomarker mapping on cortical surface
• Mapping of various values on a mesh i.e. Thickness, PET values, MR Intensity
• Surface registration
• Align the meshes of any given subject to a template to obtain a correspondence across subjects
• Transfer of biomarkers on template surface
• Map all values from all subjects to a common space where they
• Can be compared

Galaxy allows us to create a workflow by joining two or more pipelines together. We use workflows to connect the CTE with CTE surface pipelines as shown in the picture:

Galaxy based workflows for CTE

The partial volume effect is the loss of apparent activity in small objects or regions because of the limited resolution of the imaging system.  It occurs in medical imaging such as positron emission tomography (PET). The method to correct for the partial volume effect is referred to as the partial volume correction.

Overview of main functions implemented for PET PVC:

• PVC registration
• Registration of the PET Image to its corresponding MRI
• Segmentation
• Segmentation of the MRI into GM, WM and CSF
• Partial Volume Correction (PVC)
• Correction for spill in and spill over of the PET image using the MRI segmentation

Currently, all pipelines described above use version 3.20.1 of the Insight toolkit (ITK). We are migrating these to the latest version of ITK (4.3).

Imaging workflows in Galaxy

We use Galaxy to provide a user-friendly access to our imaging tools. Galaxy allows you to create processing workflows by linking different tools to one another in a simple graphical way.

Constructing workflows in Galaxy is very easy. 

As you notice it works from a Web browser. The tool are on the left pane. This example uses Cellular Imaging tools (I'll give you a tour about them in my next post).
To keep you interested, soon we are going to post a series of video tutorials on how to use Galaxy to do image analysis.