\documentclass[11pt,twoside]{article}  % Leave intact
\usepackage{adassconf}

%\documentstyle[11pt,twoside,adassconf]{article}

\begin{document}   % Leave intact

\paperID{P1-25}
%%%% ID=P1-25

\title{The ALMA Prototype Science Pipeline}

\author{Lindsey\ E.\ Davis, Brian  E.\ Glendenning, and Doug Tody}
\affil{National Radio Astronomy Observatory, PO Box O,
       1003 Lopezville Road, Socorro, NM 87801}

\contact{Lindsey Davis}
\email{ldavis@nrao.edu}

\paindex{Davis, L. E.}
\aindex{Glendenning, B. E.}
\aindex{Tody, D.}

\authormark{Davis, Glendenning, \& Tody}

\keywords{astronomy: radio, ALMA, pipelines, AIPS++, computing: distributed}

\begin{abstract}          % Leave intact
In this paper we describe the ALMA Prototype Pipeline Project, a joint ALMA
Computing IPT / NRAO Interferometry Software Division initiative to develop
a Python based pipeline processing capability for ALMA.
\end{abstract}

\section{Introduction}

The primary function of the ALMA science pipeline is to automatically
calibrate and image ALMA interferometry data and store the reduced data
in the ALMA science archive. The ALMA pipeline system will provide
the processing heuristics, in the form of Python scripts, and the software
infrastructure required to execute the pipeline in the ALMA computing
environment. The data reduction modules will be provided by the AIPS++ group.
The ALMA prototype pipeline project is a joint ALMA Computing IPT / NRAO
Interferometry Software Division initiative to develop a Python based pipeline
processing capability for ALMA using the ALMA Common Software (ACS)
framework and AIPS++ data processing modules.

\section{The Science Pipeline Requirements}

\paragraph{The pipeline software infrastructure must support:}
\begin{itemize}
\item reuse of legacy science software
\item an average data rate of 6 MB / second
\item development, maintenance, and testing by scientific staff
\item deployment at multiple sites with varied computing resources
\end{itemize}
\paragraph {To meet these requirements the pipeline architecture must:}
\begin{itemize}
\item be open, distributed, and component based
\item support a powerful scripting language
\item provide robust error handling and logging systems
\item be portable, scalable, and efficient
\item handle resource allocation transparently
\item be compatible with the offline reduction system
\end{itemize}

\section{The Science Pipeline Architecture: The Logical View}

Figure 1 shows the logical view of the ALMA science pipeline. The ALMA
scheduler issues a pipeline processing request. The heuristics lookup module
selects the appropriate pipeline application script. The pipeline application
script is queued for execution by the pipeline executor. The pipeline
application script retrieves the required input data from the ALMA archive,
calls the AIPS++ components to do the actual processing, and writes the pipeline
results to the ALMA archive. When processing is complete the pipeline executor
issues a pipeline processing complete event to the ALMA scheduler. The
pipeline control module and GUI provide mechanisms whereby the pipeline
operator may intervene in pipeline execution. The CORBA based ACS system
manages communications between components of the pipeline, and provides
basic lifecycle management, event notification, logging, and error
handling services.

\begin{figure}
\epsscale{1.00}
\plotone{P1-25_1.eps}
\caption{The logical view of the science pipeline architecture.} 
\end{figure}

\section{The Science Pipeline Architecture: The Physical View}

Figure 2 shows the physical view of the ALMA science prototype pipeline.
The interfaces to the ALMA scheduler and archive are not part of the
prototype pipeline project and have been omitted from the figure. The
heuristics lookup, pipeline executor, pipeline control, and pipeline
application modules from Figure 1 will be implemented in Python. The
pipeline application modules may be executed as Python commands (Python
CLI) or wrapped as ACS components and run inside an ACS Python container
(Python script as servant). The two AIPS++ C++ modules may be
executed as Python commands (the Python CLI) or called from within Python
pipeline application scripts. The pipeline control GUI will be written
in Java (Java GUI). ACS will manage communications between the pipeline
objects and provide basic services and tools (e.g. Object Explorer). The
prototype pipeline architecture is open and will support importing visualization
tools and data reduction components from other systems in the future.

\begin{figure}
\epsscale{1.00}
\plotone{P1-25_2.eps}
\caption{The physical view of the science pipeline architecture.} 
\end{figure}

\section{The Prototype Pipeline Development Plan}

The primary goal of the ALMA prototype pipeline project is to develop a
Python based pipeline processing capability for ALMA using ACS to
execute AIPS++ data processing components. A simple VLA 8 GHz GRB
survey data pipeline will be used to drive the prototype pipeline
development effort, help focus AIPS++ component development, and
demonstrate pipeline processing capability. The final prototype
pipeline will be scripted in Python using AIPS++ C++ components. The
project development plan is divided into three phases A, B, C. The
goals and current status of each phase are summarized below.

\begin{itemize}
\item Phase A Goals
\begin{itemize}
\item Demonstrate basic task execution including messaging.
\item Connect some key AIPS++ components to ACS.
\end{itemize}
\item Phase A Status: Complete
\begin{itemize}
\item AIPS++ vlafiller, constants, quanta, and imager ported to ACS.
\end{itemize}
\end{itemize}

\begin{itemize}
\item Phase B Goals
\begin{itemize}
\item Add remaining data processing components required for GRB use case.
\item Demonstrate a basic Python scripting capability for the AIPS++ components.
\item Demonstrate a Java client capability for AIPS++ components.
\end{itemize}
\item Phase B Status: In progress
\begin{itemize}
\item Python test scripts written for vlafiller, constants, quanta, and imager.
\item Ports of remaining major AIPS++ components underway.
\end{itemize}
\end{itemize}

\begin{itemize}
\item Phase C Goals
\begin{itemize}
\item Implement and test prototype VLA GRB pipeline in Python.
\item Run the pipeline as a servant.
\end{itemize}
\item Phase C Status: In progress
\begin{itemize}
\item ACS team have implemented Python as servant capability IN ACS.
\item GRB use case development completed.
\end{itemize}
\end{itemize}

\section{Current Status and Future Plans}

The prototype pipeline project is on track to meet its primary goal
of providing a Python based pipeline processing capability for ALMA. The
prototype pipeline design already satisfies the first three requirements
outlined in section 2. The processing efficiency issue is being successfully
addressed by the AIPS++ group. A secondary goal of the prototype pipeline
project is to evaluate ACS for use as a future analysis framework. This effort
will address longer term issues of portability, scalability, resource
allocation, and compatibility with the off-line system.

More information about the ALMA ACS, pipeline, and offline systems (AIPS++)
can be found at the ALMA TWiki site at:

http://almasw.hq.eso.org/almasw/bin/view/Main/WebHome

\acknowledgments

We are grateful to T. Cornwell for proposing the migration of AIPS++ to ACS
and developing the migration plan. We would also like to acknowledge the
hard work of the protopipeline team members Dongshan Guo, Darrell Schiebel,
Raymond Rusk, Wes Young. Finally we thank the ALMA ACS team and Dave Fugate
in particular for rapid development of the ACS Python capabilities.


\end{document}  % Leave intact
