
\chapter{Adding a Solver}
\label{chapter:addsolver}
One of the strengths of both TAO and PETSc is the ability to allow 
users to extend the built-in solvers with new user-defined algorithms.
It is certainly possible to develop new optimization algorithms outside of 
TAO framework, but
Using TAO to implement a solver has many advantages,

\begin{enumerate}
\item TAO includes other optimization solvers with an identical interface, 
so application problems may
conveniently switch solvers to compare their effectiveness.

\item TAO provides support for function evaluations and
derivative information.  It allows for the direct evaluation
of this information by the application developer, 
contains limited support for finite difference approximations, and
allows the uses of matrix-free methods.
The solvers can obtain this function and derivative information
through a simple interface  while the details of its computation 
are handled within the toolkit.

\item TAO provides line searches, convergence
tests, monitoring routines, and other tools
that are helpful in an optimization algorithm.
The availability
of these tools means that the developers of the optimization
solver do not have to write these utilities.

\item PETSc offers vectors, matrices, index sets, and linear solvers
that can be used by the solver.  These objects are standard mathematical
constructions that have many different implementations.
The objects may be distributed over multiple processors, restricted to
a single processor, have a dense representation, 
use a sparse data structure, or vary in many other ways.  
TAO solvers do not need to know how
these objects are represented or how the operations defined on them
have been implemented.  Instead, the solvers apply these operations
through an abstract interface that leaves the details to PETSc
and external libraries.
This abstraction allows solvers to work seamlessly with a variety
of data structures while allowing application developers to select 
data structures tailored for their purposes.

\item PETSc provides the user a convenient
method for setting options at runtime, performance profiling, and debugging.

\end{enumerate}

\section{Header File}
TAO solver implementation files must include the TAO implementation file \texttt{taoimpl.h}:
\begin{verbatim}
   #include "petsc/private/taoimpl.h"
\end{verbatim}
This file contains data elements that are generally kept hidden from application
programmers, but may be necessary for solver implementations to access.
\noindent


\section{TAO Interface with Solvers}
TAO solvers must be written in C or C++ and include several routines with
a particular calling sequence.  Two of these routines are mandatory:
one that initializes the TAO structure with the appropriate information
and one that applies the algorithm to a problem instance.
Additional routines may be written to set options within the
solver, view the solver, setup appropriate data structures, and destroy
these data structures. In order to implement the conjugate
gradient algorithm, for example, the following structure is
useful.
\begin{verbatim}
typedef struct{

  PetscReal beta;
  PetscReal eta;
  PetscInt  ngradtseps;
  PetscInt  nresetsteps;
  Vec X_old;
  Vec G_old; 

} TAO_CG;
\end{verbatim}
This structure contains two parameters, two counters, and two work vectors.
Vectors
for the solution and gradient are not needed here because the TAO
structure has pointers to them.


\subsection{Solver Routine}
All TAO solvers have a routine that accepts a TAO structure and
computes a solution.  
TAO will call this routine when the application
program uses the routine {\tt TaoSolve()} and will pass to the solver
information
about the objective function and constraints, pointers to the
variable vector and gradient vector, and support for line searches,
linear solvers, and convergence monitoring.  As an example, consider
the following code that solves an unconstrained minimization problem
using the conjugate gradient method.

\begin{verbatim}
PetscErrorCode TaoSolve_CG(Tao tao){

  TAO_CG  *cg = (TAO_CG *) tao->data;
  Vec x = tao->solution;
  Vec g = tao->gradient;
  Vec s = tao->stepdirection;
  PetscInt     iter=0;
  PetscReal  gnormPrev,gdx,f,gnorm,steplength=0;
  TaoLineSearchConvergedReason lsflag=TAO_LINESEARCH_CONTINUE_ITERATING;
  TaoConvergedReason reason=TAO_CONTINUE_ITERATING;
  PetscErrorCode ierr;

  PetscFunctionBegin;

  ierr = TaoComputeObjectiveAndGradient(tao,x,&f,g);CHKERRQ(ierr);
  ierr = VecNorm(g,NORM_2,&gnorm);  CHKERRQ(ierr);

  ierr = VecSet(s,0); CHKERRQ(ierr); 

  cg->beta=0;
  gnormPrev = gnorm;

  /* Enter loop */
  while (1){

    /* Test for convergence */
    ierr = TaoMonitor(tao,iter,f,gnorm,0.0,step,&reason);CHKERRQ(ierr);
    if (reason!=TAO_CONTINUE_ITERATING) break;

    cg->beta=(gnorm*gnorm)/(gnormPrev*gnormPrev);
    ierr = VecScale(s,cg->beta); CHKERRQ(ierr);
    ierr = VecAXPY(s,-1.0,g); CHKERRQ(ierr);
    
    ierr = VecDot(s,g,&gdx); CHKERRQ(ierr);
    if (gdx>=0){     /* If not a descent direction, use gradient */
      ierr = VecCopy(g,s); CHKERRQ(ierr);
      ierr = VecScale(s,-1.0); CHKERRQ(ierr);
      gdx=-gnorm*gnorm;
    } 

    /* Line Search */
    gnormPrev = gnorm;  step=1.0;
    ierr = TaoLineSearchSetInitialStepLength(tao->linesearch,1.0);
    ierr = TaoLineSearchApply(tao->linesearch,x,&f,g,s,&steplength,&lsflag);
    ierr = TaoAddLineSearchCounts(tao); CHKERRQ(ierr);
    ierr = VecNorm(g,NORM_2,&gnorm);CHKERRQ(ierr);
    iter++;
  }
  
  PetscFunctionReturn(0);
}
\end{verbatim}

The first line of this routine casts the second argument to a pointer
to a {\tt TAO\_CG} data structure.  This structure contains pointers
to three vectors and a scalar that will be needed in the algorithm.

After declaring an initializing several variables, the solver lets TAO 
evaluate the function and gradient at the
current point in the using the routine 
{\tt Tao\-Compute\-Objective\-And\-Gradient()}.
Other routines may be used to evaluate the Hessian matrix or evaluate
constraints.  TAO may obtain this information using direct evaluation 
or other means, but these details do not affect our implementation
of the algorithm.

The norm of the gradient is a standard measure used
by unconstrained minimization solvers to define convergence.
This quantity is always nonnegative and equals zero at the solution.  
The solver will pass this quantity, the current
function value, the current iteration number, and a measure of
infeasibility to TAO with the routine
\begin{verbatim}
   PetscErrorCode TaoMonitor(Tao tao, PetscInt iter, PetscReal f,
                  PetscReal res, PetscReal cnorm, PetscReal steplength,
                  TaoConvergedReason *reason);
\end{verbatim}
Most optimization algorithms are iterative, and solvers should
include this command somewhere in each iteration.  This routine
records this information, and applies any monitoring routines and 
convergence tests set by default or the user.
In this routine, the second argument is the current
iteration number, and the third argument is the current function value.
The fourth argument is a nonnegative error measure associated with the
distance between the current solution and the optimal solution.  Examples
of this measure are the norm of the gradient or the square root of a duality 
gap. The fifth argument is a nonnegative error 
that usually
represents a measure of the infeasibility
such as the norm of the constraints or violation of bounds.
This number should be zero for unconstrained solvers.
The sixth argument is a nonnegative steplength, 
or the multiple of the step direction added to the previous iterate.
The results of the convergence test are returned in the last argument.
If the termination reason is {\tt TAO\_CONTINUE\_ITERATING}, the
algorithm should continue.

After this monitoring routine, the solver computes a step direction
using the conjugate gradient algorithm and computations using Vec objects.  
These methods include
adding vectors together and computing an inner product.  A full list
of these methods can be found in the manual pages.

Nonlinear conjugate gradient algorithms also require a line search.  TAO
provides several line searches and support for using them.
The routine
\begin{verbatim}
   TaoLineSearchApply(TaoLineSearch ls, Vec x, PetscReal *f, Vec g, 
                          TaoVec *s, PetscReal *steplength, 
                          TaoLineSearchConvergedReason *lsflag)
\end{verbatim}
passes the current solution, gradient, and objective value to the
line search and returns a new solution, gradient, and objective value.  More
details on line searches can be found in Section~\ref{sec:TaoLineSearch}.
The details of the line search applied are specified elsewhere, when
the line search is created.

TAO also includes support for linear solvers using PETSc KSP objects.  
Although this algorithm
does not require one, linear solvers are an important part of many
algorithms.  Details on the use of these solvers can be found in
the PETSc users manual.

\subsection{Creation Routine}
The TAO solver is initialized for a particular algorithm in a separate
routine.  This routine sets default convergence tolerances, creates a
line search or linear solver if needed, and creates structures needed
by this solver.   For example, the routine that creates the nonlinear
conjugate gradient algorithm shown above can be implemented as follows.

\begin{verbatim}
PETSC_EXTERN PetscErrorCode TaoCreate_CG(Tao tao)
{
  TAO_CG *cg = (TAO_CG*)tao->data;
  const char *morethuente_type = TAOLINESEARCH_MT;
  PetscErrorCode ierr;

  PetscFunctionBegin;

  ierr = PetscNewLog(tao,&cg); CHKERRQ(ierr);
  tao->data = (void*)cg;
  cg->eta = 0.1;
  cg->delta_min = 1e-7;
  cg->delta_max = 100;
  cg->cg_type = CG_PolakRibierePlus;

  tao->max_it = 2000;
  tao->max_funcs = 4000;

  tao->ops->setup = TaoSetUp_CG;
  tao->ops->solve = TaoSolve_CG;
  tao->ops->view = TaoView_CG;
  tao->ops->setfromoptions = TaoSetFromOptions_CG;
  tao->ops->destroy = TaoDestroy_CG;

  ierr = TaoLineSearchCreate(((PetscObject)tao)->comm, &tao->linesearch);
  CHKERRQ(ierr);
  ierr = TaoLineSearchSetType(tao->linesearch, morethuente_type); CHKERRQ(ierr);
  ierr = TaoLineSearchUseTaoRoutines(tao->linesearch, tao); CHKERRQ(ierr);

  PetscFunctionReturn(0);
}
EXTERN_C_END
\end{verbatim}

\noindent This routine declares some variables and then allocates memory for the 
{\tt TAO\_CG} data structure. Notice that the \texttt{Tao} object 
now has a pointer to this data structure (\texttt{tao->data}) so it can be 
accessed by the other functions written for this solver implementation.

This routine also sets some default parameters particular to the conjugate
gradient algorithm, sets default convergence tolerances, and creates
a particular line search.
These defaults could be specified in the routine that solves the problem,
but specifying them here gives the user the opportunity to modify these
parameters either by using direct calls setting parameters or by using options.

Finally, this solver passes to TAO the names of all the other routines
used by the solver.  

Note that the lines {\tt EXTERN\_C\_BEGIN} and {\tt EXTERN\_C\_END} surround
this routine.  These macros are required to preserve the name of this
function without any name-mangling from the C++ compiler (if used).

\subsection{Destroy Routine}
Another routine needed by most solvers destroys the data structures
created by earlier routines.  For the nonlinear conjugate gradient
method discussed earlier, the following routine destroys the two
work vectors and the {\tt TAO\_CG} structure.
\begin{verbatim}
PetscErrorCode TaoDestroy_CG(TAO_SOLVER tao)
{
  TAO_CG *cg = (TAO_CG *) tao->data;
  PetscErrorCode    ierr;

  PetscFunctionBegin;

  ierr = VecDestroy(&cg->X_old); CHKERRQ(ierr);
  ierr = VecDestroy(&cg->G_old);CHKERRQ(ierr);

  PetscFree(tao->data);
  tao->data = NULL;

  PetscFunctionReturn(0);
}
\end{verbatim}
This routine is called from within the {\tt TaoDestroy()} routine.
Only algorithm-specific data objects are destroyed in this
routine; any objects indexed by TAO ({\tt tao->linesearch}, {\tt tao->ksp}, {\tt tao->gradient}, etc.)
will be destroyed by TAO immediately after the algorithm-specific destroy 
routine completes.


\subsection{SetUp Routine}
If the SetUp routine has been set by the initialization routine, TAO
will call it during the execution of \texttt{TaoSolve()}.
While this routine is optional, it is often provided to allocate
the gradient vector, work vectors, and other data structures 
required by the solver.
It should have the following form.
\begin{verbatim}
PetscErrorCode TaoSetUp_CG(Tao tao)
{
  PetscErrorCode ierr;
  TAO_CG *cg = (TAO_CG*)tao->data;
  PetscFunctionBegin;

  ierr = VecDuplicate(tao->solution,&tao->gradient); CHKERRQ(ierr);
  ierr = VecDuplicate(tao->solution,&tao->stepdirection); CHKERRQ(ierr);
  ierr = VecDuplicate(tao->solution,&cg->X_old); CHKERRQ(ierr);
  ierr = VecDuplicate(tao->solution,&cg->G_old); CHKERRQ(ierr);

  PetscFunctionReturn(0);
}
\end{verbatim}

\subsection{SetFromOptions Routine}
The SetFromOptions routine should be used to check for any 
algorithm-specific options set 
by the user and will be called when the application makes a call
to {\tt TaoSetFromOptions()}.  It should have the following form.
\begin{verbatim}
PetscErrorCode TaoSetFromOptions_CG(Tao tao, void *solver);
{
  PetscErrorCode ierr;
  TAO_CG *cg = (TAO_CG*)solver;
  PetscFunctionBegin;
  ierr = PetscOptionsReal("-tao_cg_eta","restart tolerance","",cg->eta,
                          &cg->eta,0);  CHKERRQ(ierr);
  ierr = PetscOptionsReal("-tao_cg_delta_min","minimum delta value","",
                          cg->delta_min,&cg->delta_min,0); CHKERRQ(ierr);
  ierr = PetscOptionsReal("-tao_cg_delta_max","maximum delta value","",
                          cg->delta_max,&cg->delta_max,0); CHKERRQ(ierr);
  PetscFunctionReturn(0);
}
\end{verbatim}

\subsection{View Routine}
The View routine should be used to output any algorithm-specific information
or statistics at the end of a solve.
This routine will be called when the application makes a call to 
{\tt TaoView()} or when the command line option {\tt -tao\_view} is used.
It should have the following form.
\begin{verbatim}
PetscErrorCode TaoView_CG(Tao tao, PetscViewer viewer)
{ 
  TAO_CG *cg = (TAO_CG*)tao->data;
  PetscErrorCode ierr;

  PetscFunctionBegin;
  ierr = PetscViewerASCIIPushTab(viewer);
  ierr = PetscViewerASCIIPrintf(viewer,"Grad. steps: %d\n",cg->ngradsteps);
  ierr = PetscViewerASCIIPrintf(viewer,"Reset steps: %d\n",cg->nresetsteps);
  ierr = PetscViewerASCIIPopTab(viewer);
  PetscFunctionReturn(0);
}
\end{verbatim}


\subsection{Registering the Solver}
Once a new solver is implemented, TAO needs to know the name of
the solver and what function to use to create the solver.  To this end, one
can use the routine
\begin{verbatim}
  TaoRegister(const char *name, 
                  const char *path,
                  const char *cname, 
                  PetscErrorCode (*create) (Tao));
\end{verbatim}
\noindent
where {\tt name} is the name of the solver (i.e., {\tt tao\_blmvm}), {\tt
path} is the path to the library containing the solver, {\tt cname} is the
name of the routine that creates the solver (in our case, {\tt
  TaoCreate\_CG}), and {\tt create} is a pointer to that creation
routine.  If one is using dynamic loading, then the fourth argument will be
ignored. 

Once the solver has been registered, the new solver can be selected 
either by using the \texttt{TaoSetType()} function or by using the \texttt{-tao\_type} command line option.



