\section{Numerical Results}

We have evaluated performance of this initial implementation
on a variety of machines.  The data
in Tables \ref{flops} - \ref{preconditioners}
were generated from tests run on the IBM SP at Argonne National
Laboratory.  The processors' frequency is 120MHz with a peak
flop rate of 480MFlops.  Each processor has 256 MB RAM, 128 KB cache
for data, and a 32 KB cache for instructions.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{|rlr|rrccc|}
\hline
 n & Ecc & Proc & Faces & PG Its & Seconds  & \%CG & Flops/Proc \\
\hline
640,000 & 0.1 & 1 & 27 & 281 &1.7485e+03 & 81 & 2.28e+07 \\
640,000 & 0.1 & 2 & 27 & 246 & 9.8243e+02 & 82 & 2.11e+07 \\
640,000 & 0.1 & 4 & 27 & 252 & 5.7280e+02 & 83 & 1.87e+07 \\
640,000 & 0.1 & 8 & 27 & 255 & 2.8476e+02 & 84 & 1.87e+07 \\
640,000 & 0.1 & 16 & 27 & 255 & 1.6695e+02 & 84 & 1.68e+07 \\
640,000 & 0.1 & 32 & 27 & 252 & 9.8769e+01 & 85 & 1.43e+07 \\
640,000 & 0.1 & 64 & 27 & 251 & 6.2460e+01 & 87 & 1.14e+07 \\
2,560,000  & 0.1 & 8  & 46 & 502 & 3.3419e+03 & 89 & 1.85e+07  \\
2,560,000  & 0.1 & 16 & 45 & 486 & 1.7810e+03 & 89 & 1.78e+07 \\
2,560,000  & 0.1 & 32 & 46 & 480 & 9.2834e+02  & 89 & 1.72e+07 \\
2,560,000  & 0.1 & 64 & 46 & 504 & 5.2248e+02 & 89 & 1.57e+07 \\
\hline
640,000 & 0.9 & 1 & 24 & 55 & 7.0721e+02  & 88 & 2.02e+07 \\
640,000 & 0.9 & 2 & 24 & 59 & 3.8811e+02 & 88 & 1.87e+07 \\
640,000 & 0.9 & 4 & 21 & 54 & 2.4083e+02 & 91 & 1.38e+07 \\
640,000 & 0.9 &  8 & 22 & 53& 1.2018e+02 & 90 & 1.46e+07 \\
640,000 & 0.9 & 16 & 24 & 56 & 7.7393e+01 & 91 & 1.30e+07 \\
640,000 & 0.9 & 32 & 23 & 54 & 4.4023e+01 & 88 & 1.11e+07 \\
640,000 & 0.9 & 64 & 24 & 58 & 3.5922e+01 & 92 & 8.14e+06 \\
2,560,000  & 0.9 & 8 & 37 & 102 & 1.4075e+03 & 94 & 1.21e+07  \\
2,560,000  & 0.9 & 16 & 40 & 100 & 8.1645e+02 & 94 & 1.20e+07 \\
2,560,000  & 0.9 & 32 & 41 & 100 & 3.9223e+02 & 93 &  1.30e+07 \\
2,560,000  & 0.9 & 64 & 40 & 102 & 2.1718e+02 & 94 & 1.20e+07   \\
\hline
640,000 & 0.99 & 1 &  9 & 18 & 4.2458e+02  & 93 & 1.87e+07 \\
640,000 & 0.99 & 2 &  9 & 18 & 2.0925e+02 & 93 & 2.11e+07 \\
640,000 & 0.99 & 4 &  9 & 18 & 1.2800e+02  & 94 & 1.56e+07 \\
640,000 & 0.99 & 8 &  9 & 18 & 5.9173e+01 & 93 & 1.60e+07 \\
640,000 & 0.99 & 16 & 9 & 18 & 3.4309e+01  & 94 & 1.46e+07 \\
640,000 & 0.99 & 32 & 9 & 18 & 1.9975e+01  & 94 & 1.26e+07 \\
640,000 & 0.99 & 64 & 9 & 18 & 1.9300e+01 & 97 & 7.10e+06 \\
2,560,000  & 0.99 & 8 & 19 & 38 & 8.1565e+02 & 96 & 1.26e+07 \\
2,560,000  & 0.99 & 16 & 19 & 38 & 4.1195e+02  & 96 & 1.31e+07 \\
2,560,000  & 0.99 & 32 & 19 & 38 & 1.9730e+02 & 95 & 1.38e+07 \\
2,560,000  & 0.99 & 64 & 19 & 38 & 1.1258e+02 &95 & 1.29e+07   \\
\hline
\end{tabular}
\caption{Performance of the  GPCG algorithm on the Journal Bearing Problem.}
\label{flops}
\end{center}
\end{table}

In each of the tests, the initial point $x$ was set to the
lower bounds $l$.  We also set the constants $\eta_1  = 0.1$ and
used the conjugate gradient stopping criteria
(\ref{cgstop2}) with $\eta_2=0.01$.  The program terminated with
an approximate solution satisfying (\ref{bqp_approximate_sol})
with $\epsilon=10^{-6}$.  These
solutions also satisfied the absolute tolerance of
$\| \nabla_{\Omega} q(x) \| < 10^{-8} $.
The conjugate gradient method with a block Jacobi preconditioner was
used to solve the reduced linear systems.
Each block of the preconditioner used a zero fill ILU factorization.

Table \ref{flops} lists the results for a representative sampling of the
problems solved by our implementation.
It shows the number of variables (n) and the eccentricity (ecc)
of the problem, the number of processors (Proc) and
wall clock time used to solve the problem, the number of GPCG iterates (Faces)
the total number of projected gradient
iterations (PG Its), the percentage of the algorithm time spent in the
conjugate
gradient method, and the rate
of floating point operations per processor.
The time in the conjugate gradient method includes extracting the submatrix,
computing the preconditioner, approximately solving the linear system,
and the projected line search.

These results are very encouraging.  For the problem with
$640,000$ variables and eccentricity parameter of $0.1$, the
16 processor calculated a solution $10.47$ times faster
than $1$ processor could. The speedup for 32 and 64 processors
was $17.7$ and $28.0$ respectively.
This statistic gives an overall
efficiency of $65.5\%$ for 16 processors and $43.7\% $ for 64 processors.
For eccentricity rates of $0.9$ and $0.99$,
the efficiencies were $57.1\%$ and $77.4\%$ respectively
for 16 processors and $30.8\%$ and $34.4\% $ for 64 processors.
These modest speedups reflect that this relatively small problem
has an insufficient computation to communication ratios
for large numbers of processors.
We achieve much better efficiencies for
problems of dimension $2,560,000$, whose overall efficiency
of 64 processors, relative to 8 processors, is
$80.0\%$, $81.0\%$, and $90.6\%$ for eccentricity parameters
of $0.1, 0.9$, and $0.99$ respectively.


\begin{table}[h]
\small
\begin{center}
\begin{tabular}{|c|cccccc|cc|}
\hline
\multicolumn{1}{|c|}{} &
\multicolumn{6}{c|}{Percentage of Time}&
\multicolumn{2}{c|}{Total MFlops} \\
\hline
\multicolumn{1}{|c|}{Number}&
\multicolumn{1}{c}{Extract}&
\multicolumn{1}{c}{Mat-Vec}&
\multicolumn{1}{c}{Linear}&
\multicolumn{1}{c}{Vec} &
\multicolumn{1}{c}{Vec} &
\multicolumn{1}{c|}{Gradient} &
\multicolumn{1}{c}{Mat-Vec}&
\multicolumn{1}{c|}{Linear} \\

\multicolumn{1}{|c|}{Proc.}&
\multicolumn{1}{c}{Submatrix}&
\multicolumn{1}{c}{Multiply}&
\multicolumn{1}{c}{Solve}&
\multicolumn{1}{c}{Local} &
\multicolumn{1}{c}{Red} &
\multicolumn{1}{c|}{Proj} &
\multicolumn{1}{c}{Multiply}&
\multicolumn{1}{c|}{Solve} \\

\hline
1 & 6 & 27 & 67 & 13 & 9 & 19 & 28 & 28  \\
2 & 5 & 31 & 68 & 11 & 8 & 18 & 45 & 52   \\
4 & 5 & 28 & 62 & 11 & 9 & 17 & 88 & 102  \\
8 & 6 & 29 & 64 & 10 & 10 & 17 & 174 & 198 \\
16 & 5 & 26 & 62 & 9 & 15 & 16 & 337 & 387  \\
32 & 6 & 23 & 62 & 8 & 22 & 15 & 630 & 625  \\
64 & 6 & 20 & 61 & 5 & 30 & 13 & 1131 & 1008  \\
\hline
\end{tabular}
\caption{Scalability of GPCG Subroutines, ( n=640,000, ecc=0.1 ) }
\label{routines}
\end{center}
\end{table}


For the journal bearing problem with $640,000$ variables and
an eccentricity parameter of $0.1$,
Table \ref{routines} indicates how well many of the functions within
the GPCG algorithm scale.
The percentage of time spent in the projected gradient (Grad Prog) portion of
the algorithm slightly decreases as the number of processor increases.
This part of the algorithm has slightly better scalability than
than the conjugate gradient part because
much of its work comes from projecting vectors back into the
feasible space and searching for variables that equal one of
their bounds.
These routines require no communication between processors.
On the other hand, these perfectly scalable routines
do not involve much arithmetic, which makes the flop rate
for the projected gradient method lower for the conjugate gradient method.
The percentage of time doing vector reductions (Vec Red) such as dot products
and norms remains fairly constant about $9\%$ of the total computation
time for $1-8$ processors.  The efficiency of the algorithm declined
more quickly thereafter as the percentage of time doing vector reductions
increased to $30\%$ on $64$ processors.  The percentage of time required
for local vector (Vec Local) operations like Axpy's and projections declined as
the processors increased.
The percentage of time
required to extract the submatrix remained the nearly constant
at $5-6\%$ of the total computation time, indicating relative efficiency for this
phase of computation.
On the other hand, the creation of a second large matrix requires
additional storage.  For the large problems with over two and a half million
variables, the matrices exceeded the memory capacity
offers by one and two  processors.

The journal bearing problem with eccentricity parameter
$ecc = 0.99$, lost almost all its scalability
when the number of processors increased from 32 to
64.  For this problem, only about $51\%$ of the variables
lie between the bounds at the solution.  For this problem, the
reduced linear systems tended to be smaller than the
systems created for other problems.  Table
\ref{problem-differences} shows percentage of variables that
lie between their bounds at the solution (Free V $\%$).
For example, when $ecc=0.1$,
about $68 \%$ of the variables lie between the bounds at the solution.
When $ecc=0.99$, only $51\%$ of the solution variables are free, which
tends to create smaller
systems that are less scalable than larger ones.

\begin{table}[h]
\small
\begin{center}
\begin{tabular}{|lc|ccc|ccc|}
\hline
\multicolumn{1}{|c}{ecc} &
\multicolumn{1}{c|}{Free Var.} &
\multicolumn{3}{c|}{Projected Gradient} &
\multicolumn{3}{c|}{Conjugate Gradient} \\
& \% &
Its/GPCG It & \% Time &Flops/Pr. &
Its/GPCG It & \% Time &Flops/Pr. \\
\hline
0.1 & 67.8 & 9.4 & 5 & 1.44e+07 & 119 & 86 & 2.86e+07 \\
0.9 & 53.6 & 2.3 & 3 & 1.42e+07 & 57 &  78 & 2.81e+07 \\
0.99 & 51.1 & 2.0 & 2 & 1.48e+07 & 82 &  79 & 2.81e+07  \\
\hline
\end{tabular}
\label{problem-differences}
\caption{Comparison of the Journal Bearing Problem with Different Parameters}
\end{center}
\end{table}


\begin{figure}
\caption{Overall Efficiency of GPCG on the Journal Bearing Problem (n=640,000, ecc=0.1)}
\begin{center}
\begin{barenv}
\setwidth{50}
\setstretch{2}
\setxname{ \ \ \ \ \ \  Processors}
\setyaxis{0}{100}{10}\setyname{\%}
\setnumberpos{empty}
\sethspace{0.2}
\bar{89}{1}[\texttt{2}]
\bar{76}{1}[\texttt{4}]
\bar {77}{1}[\texttt{8}]
\bar{65}{1}[\texttt{16}]
\bar{55}{1}[\texttt{32}]
\bar{44}{1}[\texttt{64}]
\end{barenv}
\end{center}
\end{figure}

Because the allocation of time for various phases of computation
varies from one problem to another,
the flop rates for the various problems also vary.
Table \ref{problem-differences} shows some
breakdown of the time spent in the gradient projection
and conjugate gradient methods and their respective flop
rates for $n=640,000$ on 16 processors.   When $ecc = 0.1$,
about $86\%$ of the time is spent using the conjugate gradient
method, which has the highest rate of computation.
Another $5\%$ of the time was spent in the projected gradient
iterations, which leaves less than $9\%$ of the time for
tasks like extracting the submatrix, extracting the subvector,
and creating the linear solver, which require very few arithmetic
operations. For this problem,
the average number of conjugate gradient iterations per
GPCG iterate was about $119$.  For the other problems with higher eccentricities,
the average number of conjugate gradient iterations per
GPCG iteration was significantly less.
With fewer conjugate gradient operations over which to
amortize the cost of extracting the reduced system,
the percentage of time spent solving the linear systems decreases.
For  $ecc=0.1$, the percentage of time spent on low flops tasks
and the projected line search is almost half the percentage of time
required for the other two problems for these tasks.  Therefore, the
higher flop rate of this problem is expected.

Our problems achieved a per processor
flop rate of about 20 MFlops, or approximately $4\%$, of the peak rate
of 480 MFlops offered by the IBM SP.  For comparison purposes, we
solved of a solid fuel ignition model, also called
the `Bratu problem' \cite{minpack2},
using an inexact Newton nonlinear equation solver within PETSc.
This problem also derives from the discretization of an elliptic
PDE, with one degree of freedom per node, defined
over a rectangular grid.
The Jacobian of the function has the same nonzero pattern
as the journal bearing problem, so the linear solves used in each
problem are very similar.
When run on the IBM SP with two processors on a $100 \times 100$ grid,
the Bratu problem achieved 42 MFlops per processor.
This rate is higher than the rate achieved by the GPCG algorithm,
but much of the difference can be accounted toward differences in
the computational requirements of the two optimization algorithms.
Whereas, the Newton method can sustain higher flop rate, the
GPCG algorithm, as previously mentioned, spends some time on tasks
with no arithmetic operations.  The extraction of the submatrix
uses about $6\%$ of the time, creating the reduced linear
solvers uses about $3\%$ of the time, and partitioning variables
according to whether they equal upper bound, equal the lower bound,
or lie somewhere in between, requires about $7\%$ of the
total time.  Hence, it is unlikely the GPCG algorithm can
achieve an overall computation rate as high as a Newton algorithm.

The results presented here for both the journal bearing and solid fuel
ignition models use a general sparse AIJ format (also called a
compressed, sparse row format) for matrix data.  Alternatives storage
schemes that exploit the structured sparsity of these problems would
achieve higher flop rates for matrix operations by alleviating
unnecessary memory references.  Likewise, block sparse storage
variants for problems with multiple unknowns per grid point would
achieve higher flop rates \cite{gkmt98}.  Since our optimization
algorithms use a data-structure-neutral interface to matrix and vector
operations, we can easily experiment with such alternatives without
altering any of the optimization code.

We also developed an implementation of the GPCG algorithm
using traditional programming techniques of Fortran and C.
This code runs on only a single processor and does not employ
any the object-oriented structure of the Optimization Toolkit
described in Section \ref{design}.  The input matrix must contain the
lower triangular half of the matrix $A$ in sparse row format
with the diagonal stored separately, and the
vectors must be simple arrays of double precision variables.
The conjugate gradient method uses the variable fill
incomplete Cholesky factorization of Mor\'e and Lin \cite{CJL99a}.

To compare the performance of these two implementations of the GPCG
algorithm, we tested the version with the same set of parameters.
Algorithmically, the two implementations are identical except for
the preconditioner used in the conjugate gradient method.  Whereas
the PETSc version used a zero fill incomplete LU factorization, the
other code used a Cholesky factorization with some additional fill.
The implementation  of the linear algebra in the two versions is very
different.
The PETSc library of linear algebra has
been fine tuned to achieve high performance.  The other
code uses simple implementation of vector and matrix operations.
It is careful to avoid memory striding, but
it does not explicitly account for cache size
or utilize loop unrolling and other techniques required for
high performance.

A comparison of the running times on the IBM SP can be found
in Table \ref{legacy}.
The results show that the PETSc code runs faster.
The overhead costs that result from an object-oriented approach
were more than compensated for by the savings achieved by
external libraries with a high performance implementation
of linear algebra routines.

\begin{table}[h]
\begin{center}
\hspace{.5cm}

\begin{tabular}{|lccc|}
\hline
ecc & n & Fortran/C  & PETSc \\
 & & sec.  & sec. \\
\hline
0.1 & 40,000 & 30 & 19.2\\
0.9 & 40,000&  19.4 & 7.4\\
0.99 & 40,000 & 7.1 & 4.2\\
0.1 & 160,000 & 314 & 162 \\
0.9 & 160,000 & 107 & 65 \\
0.99 & 160,000 & 54.6 & 34 \\
\hline
\end{tabular}
\label{legacy}
\caption{Time for Optimization Toolkit vs. Traditional Fortran/C}
\end{center}
\end{table}

One virtue of object-oriented design is the capability for easy algorithm
experimentation and comparison.  For example,
we tested several of the preconditioners
offered by PETSc.
One preconditioner tested was the block Jacobi preconditioner with
a zero fill ILU preconditioner used in each block.
Another preconditioner tested was the diagonal Jacobi preconditioner.
A third preconditioner we tried was a zero fill parallel incomplete Cholesky
factorization implemented by the BlockSolve package\cite{bs-user-ref}.
These switches were runtime choices that required no change in interface.

Our experiments show that the block Jacobi
performed better than the other two.
The statistics summarized in Table \ref{preconditioners}
include the problem dimension (n), eccentricity parameter (ecc),
and the number of processors used to solve the problem.  For
each of the three preconditioners, the table lists the number
of iterations of the GPCG algorithm (Its), the seconds  required
to solve the problem, the number of total number of conjugate
gradient iterations
and number of conjugate gradient iterations in the last
GPCG iteration (CG Its),
and the percentage of time spent in the conjugate gradient method (\%).
This percentage includes the time for extracting the matrix, creating
the linear solver, creating the preconditioner, and solving the system.

\begin{table}[htbp]
\begin{center} \begin{tiny}
\begin{tabular}{|rlr|rrr|rrr|rrr|}
\hline
\multicolumn{3}{|c|}{Problem}&
\multicolumn{3}{c|}{Block Jacobi, ILU(0)}&
\multicolumn{3}{c|}{Diagonal}&
\multicolumn{3}{c|}{BlockSolve ICC(0)}\\
 n & ecc & proc &
Its & Seconds & CG Its &
Its & Seconds & CG Its &
Its & Seconds & CG Its \\
\hline
640,000 & 0.1 & 1 & 27 & 1.749e+03 & 2806/118 & 27 & 3.896e+03 & 13539/932 &  &&  \\
640,000 & 0.1 & 4 & 27 & 5.728e+02 & 3060/157 & 27 & 1.185e+03 & 13513/906 & 27 & 3.222e+03 & 5236/227  \\
640,000 & 0.1 & 16 & 27 & 1.669e+02 & 3225/165 & 27 & 3.414e+02 & 13511/904 & 27 & 8.870e+02 & 5291/282 \\
2,560,000 & 0.1 & 8 & 46 & 3.342e+03 & 9524/230 & 45 & 7.200e+03 & 43813/1157 & 46 & 1.8342e+04 & 17326/606 \\
2,560,000 & 0.1 & 16 & 45 & 1.781e+03 & 9766/250 & 46 & 3.961e+03 & 44786/1593 & 46 & 9.396e+03 & 17361/545 \\
\hline
640,000 & 0.9 & 1 & 24 & 7.072e+02 & 1205/182 & 22 &  1.6946e+03 & 7014/904&  &  & \\
640,000 & 0.9 & 4 & 21 & 2.408e+02 & 1120/190 & 22 & 5.576e+02 & 7014/904  & 26 & 1.403e+03 & 1419/314  \\
640,000 & 0.9 & 16 & 24 & 7.739e+01 & 1379/203 & 22 & 1.584e+02 & 7014/904 & 27 & 3.843e+02 & 1461/333  \\
2,560,000 & 0.9 & 8 & 37 & 1.408e+03 & 3036/352 & 39 & 3.220e+03 & 19626/2447 & 40 & 5.627e+03 & 3431/742 \\
2,560,000 & 0.9 & 16 & 40 & 8.164e+02 & 3549/425 & 37 & 1.714e+03 & 20262/2579 & 44 & 3.052e+03 & 3460/636 \\
\hline
640,000 & 0.99 & 1 &  9 & 4.246e+02 &  730/208 &  9 & 8.607e+02 & 3655/879 &  & & \\
640,000 & 0.99 & 4 &  9 & 1.280e+02 &  737/193 &  9 & 2.737e+02 & 3655/879 &  9 & 6.0291e+02 &  880/578 \\
640,000 & 0.99 & 16 &  9 & 3.431e+01 &  738/243 &  9 & 7.921e+01 & 3655/879 &  9 & 1.6569e+02 &  893/585  \\
2,560,000 & 0.99 & 8  & 19 & 8.157e+02 & 1952/156 & 19 & 1.796e+03 & 11439/966 & 19 & 2.9597e+03 & 2033/1057  \\
2,560,000 & 0.99 & 16 & 19 & 4.120e+02 & 2055/160 & 19 & 9.239e+02 & 11413/940 & 19 & 1.4823e+03 & 2074/1097 \\
\hline
\end{tabular}
\label{preconditioners}
\caption{Comparison of Preconditioners}
\end{tiny}\end{center}
\end{table}

As expected, the ICC and Block Jacobi methods required fewer
iterations than the diagonal Jacobi.  Also not surprising, was
that the diagonal and block Jacobi used less time than
the ICC preconditioner.
The Jacobi methods are very scalable, whereas
the ICC preconditioner is inherently sequential to create
and apply.  Somewhat surprising, however, was that in some problems, the ICC
required more conjugate gradient iterations than the
block Jacobi despite having more nonzeros in the preconditioner.
Currently, the ICC preconditioner has only been implemented in
BlockSolve format, so that for each linear solve, the matrix has to
be converted into the BlockSolve format.  This conversion takes
time, but this time amounted to less that $2\%$ of the total computation time.
This difference was certainly not enough to compensate for
the dramatic time difference, and hence, warrant an implementation
of the ICC preconditioner in other formats.
Solving problems using the block Jacobi preconditioner generally used
about half the time required using diagonal preconditioning, and
about a fourth the time required when the ICC preconditioner was used.

\section{Observations and Future Work}

This object-oriented implementation of GPCG is
consistent with the software interoperability
goals of the DOE2000 Initiative.  The problem data can be
entered independent of its representation.  The GPCG program
tries not to do the linear algebra
itself; rather, we want it to call routines previously designed and
efficiently implemented by other
numerical packages like PETSc.
The algorithm heavily uses standard matrix and vector operations like
{\tt AXPY(), MatVecMult()}, and {\tt Norm()} which are common to
many numerical software libraries.

On several occasions, however, the program requires additional vector
functionality, including:
\begin{enumerate}

\item {\tt VecPointwiseMin(Vec V1, Vec V2, Vec VMin)}: define each component
of {\tt VMin} as the minimum of the corresponding points of {\tt V1} and
{\tt V2}.

\item {\tt VecMin(Vec V, Scalar *minimum)}: provide the
smallest value in the vector {\tt V}.  This function is already implemented
for single processors.

\item {\tt VecFindIdenticalPoints(Vec V1, Vec V2, IS* Same)}: create
an index set {\tt Same}
containing the indices of the elements of {\tt V1} and {\tt V2}
with identical values.

\item {\tt VecMaxStepLenth(Vec Step, Vec Bound, Scalar *lenth)}:
return the largest positive scalar such that {\tt *length $\times$
Step $\leq$ Bound}.
Return negative if the answer in infinity.

\end{enumerate}
The current version of the code implements these functions by accessing the data
through complementary OT implementations.

As Table \ref{flops} shows, most of the time in the
GPCG algorithm is spent solving systems of linear equations.
We found that that the linear solvers of PETSc worked very well.
Other options, however, may improve the performance of GPCG.

One option in not currently offered by PETSc is the ability
to ``continue'' the conjugate gradient method after stopping it.
Recall that the decision whether to continue with
the conjugate gradient method or return to the projected gradient
method has several parts.  In the current implementation,
if the algorithm decides to continue with the conjugate gradient
method, it uses the approximate solution already calculated as
the starting point.  However, the history of
the previous conjugate directions is lost, and the algorithm
starts anew with a gradient direction.  The GPCG algorithm
may save some conjugate gradient iterations if the conjugate gradient
history from a previous solve were available for a new solve.

Adding this feature to the linear solver of PETSc would be of
secondary importance.
As noted earlier, the time and memory spent extracting submatrices
has not significantly impeded the algorithm on our test problems.
Continuing the conjugate gradient method generally occurs on only the
last iterate anyway, and thus the ``continuation'' feature would
not be used extensively.  Furthermore, it is not clear how much
that feature would reduce the number of conjugate
gradient iterations and the overall time.

More tools for solving symmetric systems of linear equations would
also be helpful in future algorithms.  For this algorithm the
conjugate gradient method worked well, but the preconditioners,
with the exception of diagonal Jacobi, had nonsymmetric factorizations.
A block incomplete Cholesky preconditioner and other
preconditioners for symmetric systems would be of interest.  Other
optimization algorithms involve a symmetric indefinite matrix.  While
GMRES and other methods designed for general linear systems would
work, iterative methods designed specifically for symmetric systems
would could save computation time.

The TAO project is working to  develop a
new set of software
components for the solution of large-scale optimization problems.
Research within GPCG and other projects mentioned
above represents early investigations in extending the benefits of modern
software styles to classes of optimization algorithms.
Valuable lessons have been learned about
abstractions for various algorithm classes.
In addition, during recent years, we have learned much about the design
of high  performance libraries for
parallel numerical linear algebra, which can
provide the core kernels of many optimization codes.
Our experience in developing GPCG
shows that high-performance computing
and object-oriented programming can be used in numerical optimization
and prepares us
well to address the design of other optimization algorithms.

\par
