% $Id: developers.tex,v 1.18 2000/09/22 21:35:11 balay Exp $
%
% LATEX version of the PETSc users manual.
%
% manual_tex.tex is the base file for LaTeX format, while manual.tex is
% the corresponding base HTML format file.
%
\documentclass[twoside,12pt]{../sty/report_petsc}

\usepackage{makeidx,xspace}
\usepackage[bookmarksopen,colorlinks]{hyperref}
\usepackage[all]{hypcap}
\usepackage{color}
\input pdfcolor.tex

\usepackage[pdftex]{graphicx}
\usepackage{../sty/verbatim}
\usepackage{../sty/tpage}
\usepackage{../sty/here}
\usepackage{../sty/anlhelper}
\usepackage[obeyspaces]{../sty/trl}

\setlength{\textwidth}{6.5in}
\setlength{\oddsidemargin}{0.0in}
\setlength{\evensidemargin}{0.0in}
\setlength{\textheight}{9.2in}
\setlength{\topmargin}{-.8in}

\newcommand{\findex}[1]{\index{FUNCTION #1}}
\newcommand{\sindex}[1]{\index{#1}}
\newcommand{\F}{\mbox{\boldmath \(F\)}}
\newcommand{\x}{\mbox{\boldmath \(x\)}}
\newcommand{\rr}{\mbox{\boldmath \(r\)}}

\makeindex


\begin{document}

\ANLTitle{PETSc Developers Manual}{\em The PETSc Team \\
http://www.mcs.anl.gov/petsc
\vspace{0.5in} \\
{\rm This document is intended for use with PETSc 3.6}}
{}{June 2015}

\newpage

\hbox{ }

\vspace{1in}
\date{\today}


% Blank page makes double sided printout look bettter.
\newpage




\medskip \medskip


%
%   Next line temp removed
%
\noindent {\bf Abstract:}

\medskip \medskip
PETSc is
a set of extensible software libraries for scientific computation.
PETSc is designed using a object-oriented
architecture. This means that libraries consist of {\em objects} that
have certain, defined functionality. This document defines how these
objects are implemented.

This manual discusses the PETSc library design and describes
how to develop new library codes that are compatible with other PETSc components
including PETSc 3.0.
The idea is not to develop one massive library that everyone shoves code
into; rather, to develop an architecture that allows many people
to (as painlessly as possible) contribute (and maintain) their own libraries,
in a distributed fashion.

The text assumes
that you are familiar with PETSc and have access to PETSc source code and documentation
(available via \href{http://www.mcs.anl.gov/petsc}{http://www.mcs.anl.gov/petsc}.)

Before contributing code to PETSc please read Chapter 1, which contains the source code style guide.
\href{http://www.mcs.anl.gov/petsc/developers/index.html}{http://www.mcs.anl.gov/petsc/developers/index.html}
contains information on how to submit patches to PETSc.

\vspace{1cm}

Please direct all comments and questions regarding PETSc design and
development to \trl{petsc-dev@mcs.anl.gov}.  Note that all {\em
bug reports and questions regarding the use of PETSc} should continue
to be directed to \trl{petsc-maint@mcs.anl.gov}.

%
% NOTES:
%  - Be sure to place captions BEFORE labels in figures and tables!
%    Otherwise, numbering will be incorrect.  For example, use the following:
%       \caption{PETSc Vector Operations}
%       \label{fig:vectorops}
%  - Use \break to indicate a line break (needed to prevent long strings in
%    \tt mode from running of the page)
%

% Blank page makes double sided printout look bettter.
\newpage
\hbox{ }
\newpage

\tableofcontents

\chapter{Answering petsc-maint@mcs.anl.gov and petsc-users@mcs.anl.gov}

\begin{itemize}
\item Try to be polite (this is not always easy)
\item Address the person by name (when it is possible to determine their name)
\item Appologize for the problem when it is appropriate (but not otherwise)
\item Thank the person for their patience if it is more than six hours since the report came in.
\item if the person drops the petsc-maint or petsc-users from the mailing list add it back in.
\end{itemize}

Often it pays to not ask too many questions or give to many suggestions in the same email. The user often only responds to the first of them anyways.

\chapter{Style Guide}

The PETSc team uses certain conventions to make our source code consistent. Groups
developing classs compatible with PETSc are, of course, free to organize their
own source code anyway they like.

\section{Names}
Consistency of names for variables, functions, etc. is extremely
important in making the package both usable and maintainable.
We use several conventions:
\begin{itemize}
\item All function names and enum types consist of words, each of
      which is capitalized, for example KSPSolve() and
      MatGetOrdering().
\item All enum elements and macro variables are capitalized. When
      they consist of several complete words, there is an underscore between each word.
\item Functions that are private to PETSc (not callable by the
      application code) either
      \begin{itemize}
        \item have an appended \trl{_Private} (for example,
           \trl{StashValues_Private}) or
        \item have an appended \trl{_Subtype} (for example,
           \trl{MatMult_SeqAIJ}).
        \item have an appended \trl{_Default} (for example,
           \trl{KSPBuildSolutionDefault}) if used by several classes.
      \end{itemize}

      In addition, functions that are not intended for use outside
      of a particular file are declared static.
\item Function names in structures are the same as the base application
      function name without the object prefix, and all are in small letters.
      For example, MatMultTranspose() has a structure name of
      \trl{multtranspose()}.
\item Each application usable function begins with the name of the class object, followed by any subclass name,
      for example, ISInvertPermutation(), MatMult() or KSPGMRESSetRestart().
\item Functions that PETSc provides as defaults for user provideable functions end with \trl{Default} (for example, \trl{KSPMonitorDefault()} or \trl{PetscSignalHandlerDefault()})
\item Options database keys are lower case, have an underscore between words and match the function name associated with the option without the word set. For example, \trl{-ksp_gmres_restart}.
\end{itemize}

\section{Coding Conventions and Style Guide}

Within the PETSc source code, we adhere to the following guidelines
so that the code is uniform and easily maintainable:

\begin{itemize}
\item Public PETSc includes, \trl{petsc*.h}, cannot include private PETSc \trl{petsc/private/*impl.h} includes.
\item Public and private PETSc includes, cannot include include files in the PETSc source tree.
\item All PETSc function bodies are indented two characters.
\item Each additional level of loops, if statements, etc. is indented
      two more characters.
\item Wrapping lines should be avoided whenever possible.
\item Source code lines do not have a hard length limit, generally we like them less than 150 characters wide.
\item The local variable declarations should be aligned. For example,
      use the style
\begin{tabbing}
   Scalar \=a;\\
   int \>i,j;\\
\end{tabbing}
instead of
\begin{tabbing}
   int    i,j;\\
   Scalar a;
\end{tabbing}
\item Array and pointer arguments to functions where the array values are not changed should be labeled as \trl{const} arguments.
\item Scalar values passed to functions should {\bf never} be labeled as const.
\item The prototypes for functions should not include the names of the variables; for example write
\begin{tabbing}
PetscErrorCode MyFunction(PetscInt);
\end{tabbing}
not
\begin{tabbing}
PetscErrorCode MyFunction(PetscInt myvalue);
\end{tabbing}
\item All local variables of a particular type (e.g., \trl{int}) should be
      listed on the same line if possible; otherwise, they should be listed
      on adjacent lines.
\item Equal signs should be aligned in regions where possible.
\item There {\em must} be a single blank line
      between the local variable declarations and the body of the function.
\item The first line of the executable statments must be PetscFunctionBegin;
\item The following text should be before each function
\begin{tabbing}
\#undef \_\_FUNCT\_\_\\
\#define \_\_FUNCT\_\_ ``FunctionName''
\end{tabbing}
this is used by various macros (for example the error handlers) to always know
what function the process is in.
\item Use PetscFunctionReturn(returnvalue); not \trl{return(returnvalue);}
\item {\em Never} put a function call in a return statment; do not do
\begin{tabbing}
   PetscFunctionReturn( somefunction(...) );
\end{tabbing}
\item Do {\bf not} put a blank line immediately after PetscFunctionBegin; or
a blank line immediately before PetscFunctionReturn(0);.
\item Indentation for \trl{if} statements {\em must}  be done  as
as
\begin{tabbing}
   if (  ) \{\\
     ....\\
   \} else \{\\
     ....\\
   \}
\end{tabbing}
\item {\em Never}  have
\begin{tabbing}
   if\= (  ) \\
     \>a single indented line
\end{tabbing}
or
\begin{tabbing}
   fo\=r (  ) \\
     \>a single indented line
\end{tabbing}
instead use either
\begin{tabbing}
   if (  ) a single line
\end{tabbing}
or
\begin{tabbing}
   if\= (  ) \{\\
     \>a single indented line\\
   \}
\end{tabbing}
\item {\em No} tabs are allowed in {\em any} of the source code.
\item Do not use sqrt(), pow(), sin() etc directly in PETSc source code or examples, rather use PetscSqrtScalar(), PetscSqrtReal() etc depending on the context. See petscmath.h for expressions to use.
\item The open bracket \{ should be on the same line as the {\em if ()} test, {\em for ()}, etc. never on
      its own line unless followed by an \trl{else} or \trl{else if}, for example
\begin{tabbing}
   \} else \{ \\
\end{tabbing}
never
\begin{tabbing}
   \} \\
   else \{.
\end{tabbing}
 The closing bracket \} should {\bf always} be on its own line.
\item Do not leave chunks of commented out code in the source files.
\item In function declaration the open bracket \{ should be on the {\bf next} line, not on the same line as the function name and
      arguments. This is an exception to the rule above.
\item assert.h should not be included in PETSc source and assert() should not be used. It doesn't play well in the parallel MPI world.
\item The macros SETERRQ() and CHKERRQ() should be on the
      same line as the routine to be checked unless this violates the
      150 character width rule. Try to make error messages short, but
      informative.
\item {\em No} space after a ( or before a ). No space before the CHKXXX(). That is, do not write
\begin{tabbing}
   ierr = PetscMalloc1( 10,\&a ); CHKERRQ(ierr);
\end{tabbing}
instead write
\begin{tabbing}
   ierr = PetscMalloc1(10,\&a);CHKERRQ(ierr);
\end{tabbing}
\item Except in code that may be called before PETSc is fully initialized, always use PetscMalloc(), PetscNew(), and PetscFree(), not \trl{malloc()} and \trl{free()}.
\item {\em No} space after the ) in a cast, no space between the type and the * in a caste.
\item {\em No} space before or after a , in lists
That is, do not write
\begin{tabbing}
    int a, b,c;\\
    ierr = func(a, 22.0);CHKERRQ(ierr);
\end{tabbing}
instead write
\begin{tabbing}
    int a,b,c;\\
    ierr = func(a,22.0);CHKERRQ(ierr);
\end{tabbing}

\item Subroutines that would normally have void ** argument to return a pointer to some data, should actually be protyped as as void*. This prevents the caller from having to put a (void**) caste in each function call. See, for example, DMDAVecGetArray().

\item Do not use the {\em register} directive.
\item Never use a local variable counter like PetscInt flops = 0; to accumulate flops and then call PetscLogFlops() {\em always} just
      call PetscLogFlops() directly when needed.
\item Do not use {\em if (rank == 0)} or {\em if (v == NULL)} or {\em if (flg == PETSC\_TRUE)} or {\em if (flg == PETSC\_FALSE)}
instead use {\em if (!rank)} or {\em if (!v)} or {\em if (flg)} or {\em if (!flg)}.
\item Do not use {\em \#ifdef} or {\em \#ifndef} rather use {\em \#if defined(...} or {\em \#if !defined(...}
\item MPI routines and macros that are not part of the 1.0 or 1.1 standard should not be used in PETSc without appropriate ./configure checks and \#if defined() checks the code. Code should also be provided that works if the MPI feature is not available. For example,
\begin{tabbing}
\#if\= defined(PETSC\_HAVE\_MPI\_IN\_PLACE)\\
\>    ierr  = MPI_Allgatherv(\=MPI\_IN\_PLACE,0,MPI\_DATATYPE\_NULL,lens,recvcounts,\\
\> \> displs,MPIU\_INT,comm);CHKERRQ(ierr);\\
\#else\\
\>    ierr  = MPI_Allgatherv(lens,sendcount,MPIU\_INT,lens,recvcounts,\\
\> \> displs,MPIU\_INT,comm);CHKERRQ(ierr);\\
\#endif
\end{tabbing}
\item There shall be no PETSc routines introduced that provide essentially the same functionality as an available MPI routine. For example, one should not write a routine PetscGlobalSum() that takes a scalar value and performs an MPI\_Allreduce() on it. One should use MPI\_Allreduce() directly in the code.
\item XXXTypes (for example KSPType) do not have an underscore in them, unless they refer to another package that uses an underscore, for example MATSOLVERSUPERLU\_DIST.
\end{itemize}

\section{Formatted Comments for Documentation and Fortran Interfaces}

PETSc uses formatted comments and the Sowing packages to generate documentation (manual pages) and the fortran interfaces. Documentation for Sowing and the formating may be found at \href{http://www.cs.uiuc.edu/homes/wgropp/projects/software/sowing/doctext.pdf}{http://www.cs.uiuc.edu/homes/wgropp/projects/software/sowing/doctext.pdf}.

\begin{verbatim}
/*@     indicates a formatted comment of a function that will be used for both documentation and a Fortran interface.
/*@C    a formatted comment of a function that will be used only for documentation
/*E     a formatted comment of an enum used for documentation only
/*M     a formatted comment of a CPP macro used for both documentation and a Fortran interface.
/*MC    a formatted comment of a CPP macro for documentation
\end{verbatim}

Functions that take char* or function pointer arguments must have the C symbol since the Fortran interface generator cannot handle them so the Fortran interface for them must be created manually.

The Fortran interface files go into the three directories depending on how they are created: ftn-auto, ftn-custom, ftn-f90.

% -----------------------------------------------------------------------------------
\chapter{The PETSc Kernel}
\label{chapter:kernel}
PETSc provides a variety of basic services for writing scalable, component
based libraries; these are referred to as the PETSc kernel. The source
code for the kernel is in \trl{src/sys}. It contains systematic support for
\begin{itemize}
  \item PETSc types
  \item error handling
  \item memory management
  \item profiling
  \item object management
  \item file IO
  \item options database
  \item basic objects for viewing and drawing.
\end{itemize}
Each of these is discussed in a section below.

% -------------------------------------------------------------------------------
\section{PETSc Types}
For maximum flexibility, the basic data types \trl{int}, \trl{double} etc are
generally not used in source code, rather it has:
\begin{itemize}
  \item PetscScalar
  \item PetscInt
  \item PetscMPIInt
  \item PetscBLASInt
  \item PetscBool
  \item PetscBT  - bit storate of logical true and false
\end{itemize}
PetscInt can be set using ./configure (option --with-64-bit-indices) to be either \trl{int} (32 bit) or \trl{long long}
(64 bit)
to allow indexing into very large arrays. PetscMPIInt are for integers passed to MPI
as counts etc, these are always \trl{int} since that is what the MPI standard uses. Similarly
PetscBLASInt are for counts etc passed to BLAS and LAPACK routines. These are almost always
\trl{int} unless one is using a special ``64 bit integer'' BLAS/LAPACK (this is available, for
example on Solaris systems).

In addition there a special types
\begin{itemize}
  \item PetscClassId
  \item PetscErrorCode
  \item PetscLogEvent
\end{itemize}
in fact, these are currently always \trl{int} but their use clarifies the code.

\section{Implementation of Error Handling}

PETSc  uses a ``call error handler; then (depending on result) return
error code'' model when problems are detected in the running code.

The public include file for error handling is
 \href{http://www.mcs.anl.gov/petsc/petsc-master/include/petscerror.h.html}{include/petscerror.h}, the
source code for the PETSc error handling is in
\trl{src/sys/error/}.

\subsection{Simplified Interface}

The simplified C/C++ macro-based interface consists of the following two calls
\begin{itemize}
\item SETERRQ(comm,error code,''Error message'');
\item CHKERRQ(ierr);
\end{itemize}

The macro SETERRQ() is given by
\begin{tabbing}
return PetscError(\=comm,\_\_LINE\_\_,\_\_FUNCT\_\_,\_\_FILE\_\_,error code,error type,\\
\> ''Error message'');
\end{tabbing}
It calls the error handler with the current function name and location: line number,
file and directory, plus an error code and an error message. Normally \trl{comm} is PETSC_COMM_SELF, it can only be another communicator if
one is absolutely sure the same error will be generated on all processes in the communicator. This is to prevent the same error message from
being printed by many processes. The \trl{error type} is \trl{PETSC_ERROR_INITIAL} on detection of the initial error and \trl{PETSC_ERROR_REPEAT} for any additional calls. This is so that
the detailed error information is only printed once instead of for all levels of returned errors.

The macro CHKERRQ() is defined by
\begin{tabbing}
  if (ierr)  PetscError(\=PETSC\_COMM\_SELF,\_\_LINE\_\_,PETSC\_FUNC\_\_,\_\_FILE\_\_, \\
 \>ierr,PETSC\_ERROR\_REPEAT," ");
\end{tabbing}

In addition to SETERRQ() are the macros SETERRQ1(), SETERRQ2(), SETERRQ3()
and SETERRQ4() that allow one to include additional arguments that the message
string is formated. For example,
\begin{tabbing}
  SETERRQ2(comm,PETSC\_ERR,''Iteration overflow: its \%d norm \%g'',its,norm);
\end{tabbing}
The reason for the numbered format is because CPP macros cannot handle a variable number
of arguments.

\subsection{Error Handlers}
The error handling function PetscError() calls the ``current'' error handler
with the code
\begin{tabbing}
Pe\=tscErrorCode PetscError(\=MPI\_Comm,int line,const char *func,const char* file,\\
 \> \> const char *dir,error code, error type,const char *mess)\\
\{ \\
  \> PetscErrorCode ierr;\\
  \>PetscFunctionBegin;\\
  \>if (!eh)     ierr = PetscTraceBackErrorHandler(line,func,file,dir,error code,error type,mess,0);\\
  \>else         ierr = (*eh-$>$handler)(line,func,file,dir,error code,error type,mess,eh-$>$ctx);\\
  \>PetscFunctionReturn(ierr);\\
\}
\end{tabbing}
The variable \trl{eh} is the current error handler context and is defined in
 \href{http://www.mcs.anl.gov/petsc/petsc-master/src/sys/error/err.c.html}{src/sys/error/err.c} as
\begin{tabbing}
ty\=pedef struct \_EH*\= EH;\\
struct \_EH \{\\
  \>PetscErrorCode \> handler(\=MPI_Comm,int,const char*,const char*,const char *,\\
\> \> \> PetscErrorCode,PetscErrorType,const char*,void *);\\
  \>void \>  *ctx;\\
  \>EH   \>  previous;\\
\};
\end{tabbing}

One can set a new error handler with the command
\begin{tabbing}
Pe\=tscErrorCode PetscPushErrorHandler(\=PetscErrorCode (*handler)(int,const char *,const char*,\\
   \> \>                       const char*,\\
    \> \>                      PetscErrorCode,PetscErrorType,const char*,void*),\\
   \>  \>                      void *ctx )\\
\{\\
  \> PetscErrorCode ierr;\\
  \>EH neweh; \\
  \>PetscFunctionBegin;\\
  \>ierr = PetscNew(\&neweh);CHKERRQ(ierr);\\
  \>if (eh) {neweh-$>$previous = eh;} \\
  \>else    {neweh-$>$previous = 0;}\\
  \>neweh-$>$handler = handler;\\
  \>neweh-$>$ctx     = ctx;\\
  \>eh             = neweh;\\
  \>PetscFunctionReturn(0);\\
\}
\end{tabbing}
which maintains a linked list of error handlers. The most recent error handler is removed
via
\begin{tabbing}
Pe\=tscErrorCode PetscPopErrorHandler(void)\\
\{\\
 \> EH tmp;\\
\>  PetscErrorCode ierr;\\
\>  PetscFunctionBegin;\\
\>  if (!eh) PetscFunctionReturn(0);\\
\>  tmp = eh;\\
\>  eh  = eh-$>$previous;\\
\>  ierr = PetscFree(tmp);CHKERRQ(ierr);\\
\>  PetscFunctionReturn(0);\\
\}
\end{tabbing}

PETSc provides several default error handlers
\begin{itemize}
\item PetscTraceBackErrorHandler(), 
\item PetscAbortErrorHandler(), \trl{-on_error_abort},
\item PetscReturnErrorHandler(),
\item PetscEmacsClientErrorHandler(), 
\item PetscMPIAbortErrorHandler(), and
\item PetscAttachDebuggerErrorHandler(). \trl{-on_error_attach_debugger}.
\end{itemize}

\subsection{Error Codes}

The PETSc error handler takes a generic error code.
The generic error codes are defined in
\href{http://www.mcs.anl.gov/petsc/petsc-master/include/petscerror.h.html}{include/petscerror.h}, the same generic
error code would be used many times in the libraries. For example the
generic error code \trl{PETSC_ERR_MEM} is used whenever requested memory allocation
is not available.

\subsection{Detailed Error Messages}
In a modern parallel component oriented application code it does not make sense
to simply print error messages to the screen (more than likely there is no
``screen'', for example with Windows applications).
PETSc provides the replaceable function pointer
\begin{tabbing}
   (*PetscErrorPrintf)(``Format'',...);
\end{tabbing}
that, by default prints to standard out. Thus error messages should not
be printed with printf() or fprintf() rather it should be printed with
(*PetscErrorPrintf)(). One can direct all error messages to
stderr with the command line options \trl{-error_output_stderr}.


% -----------------------------------------------------------------------------------
\section{Implementation of Profiling}
\label{sec:profimpl}

This section provides details about the implementation of event
logging and profiling within the PETSc kernel.
The interface for profiling in PETSc is contained in the file
\href{http://www.mcs.anl.gov/petsc/petsc-master/include/petsclog.h.html}{include/petsclog.h}. The source code for the profile logging
is in \trl{src/sys/plog/}.

\subsection{Profiling Object Create and Destruction}

The creation of objects is profiled with the command
 \findex{PetscLogObjectCreate()}
\begin{tabbing}
   PetscLogObjectCreate(PetscObject h);
\end{tabbing}
which logs the creation of any PETSc object.
Just before an object is destroyed, it should be  logged with
with \findex{PetscLogObjectDestroy()}
\begin{tabbing}
   PetscLogObjectDestroy(PetscObject h);
\end{tabbing}
These are called automatically by PetscHeaderCreate() and
PetscHeaderDestroy() which are used in creating all objects
inherited off the basic object. Thus these logging routines should
never be called directly.

If an object has a clearly defined parent object (for instance, when
a work vector is generated for use in a Krylov solver), this information
is logged with the command, \findex{PetscLogObjectParent()}
\begin{tabbing}
   PetscLogObjectParent(PetscObject parent,PetscObject child);
\end{tabbing}
It is also useful to log information about the state of an object, as can
be done with the command \findex{PetscLogObjectState()}
\begin{tabbing}
   PetscLogObjectState(PetscObject h,const char *format,...);
\end{tabbing}

For example, for sparse matrices we usually log the matrix
dimensions and number of nonzeros.

\subsection{Profiling Events}

Events are logged using the
pair \findex{PetscLogEventBegin()}
\begin{tabbing}
   PetscLogEventBegin(PetscLogEvent event,PetscObject o1,...,PetscObject o4);\\
   PetscLogEventEnd(PetscLogEvent event,PetscObject o1,...,PetscObject o4);
\end{tabbing}
This logging is usually done in the abstract
interface file for the operations, for example, \href{http://www.mcs.anl.gov/petsc/petsc-master/src/mat/interface/matrix.c.html}{src/mat/interface/matrix.c}.

\subsection{Controling Profiling}

Several routines that control the default profiling available in PETSc are
\findex{PetscLogView()}
are \findex{PetscLogDefaultBegin()} \findex{PetscLogDump()} \findex{PetscLogAllBegin()}
\begin{tabbing}
   PetscLogDefaultBegin();\\
   PetscLogAllBegin();\\
   PetscLogDump(const char *filename);\\
   PetscLogView(PetscViewer);
\end{tabbing}
These routines are normally called by the PetscInitialize()
and PetscFinalize() routines when the option
\trl{-log\_summary} is given.


% -----------------------------------------------------------------------------------
\chapter{Basic Object Design}
\label{chapter:design}


PETSc is designed using strong data encapsulation.  Hence,
any collection of data (for instance, a sparse matrix) is stored in
a way that is completely private from the application code. The application
code can manipulate the data only through a well-defined interface, as it
does {\em not} ``know'' how the data is stored internally.

\section{Introduction}

PETSc is designed around several classes (e.g. Vec (vectors),
Mat (matrices, both dense and sparse)). These classs are each
implemented using C \trl{struct}s, that contain the data and function pointers
for operations on the data (much like virtual functions in classes in C++).
Each classs consists of three parts:
\begin{itemize}
\item a (small) common part shared by all PETSc compatible libraries.
\item another common part shared by all PETSc implementations of the class and
\item a private part used by only one particular implementation written in PETSc.
\end{itemize}
For example, all matrix (Mat) classs share a function table of operations that
may be performed on the matrix; all PETSc matrix implementations share some additional
data fields,including matrix size; while a particular matrix implementation in PETSc
(say compressed sparse row) has its own data fields for storing the actual
matrix values and sparsity pattern. This will be explained in more detail
in the following sections. People providing new class implementations {\bf must}
use the PETSc common part.


We will use \trl{<class>_<implementation>} to denote the actual source code and
data structures used for a particular implementation of an object that has the
\trl{<class>} interface.

\section{Organization of the Source Code}

Each class has
\begin{itemize}
\item Its own, application public, include file \trl{include/petsc<class>.h}
\item Its own directory, \trl{src/<class>}
\item A data structure defined in  the file
      \trl{include/petsc/private/<class>impl.h}.
      This data structure is shared by all the different PETSc implementations of the
      class. For example, for matrices it is shared by dense,
      sparse, parallel, and sequential formats.
\item An abstract interface that defines the application callable
      functions for the class. These are defined in the directory
      \trl{src/<class>/interface}. This is how polymorphism is supported with code that implements the abstract interface to the
operations on the object.  Essentially, these routines do some error
checking of arguments and logging of profiling information
and then call the function appropriate for the
particular implementation of the object. The name of the abstract
function is \trl{<class>Operation}, for instance, MatMult() or PCCreate(), while
the name of a particular implementation is
\trl{<class>Operation_<implementation>}, for instance,
\trl{MatMult_SeqAIJ()} or \trl{PCCreate_ILU()}. These naming
conventions are used to simplify code maintenance.

\item One or more actual implementations of the classs (for example,
      sparse uniprocessor and parallel matrices implemented with the AIJ storage format).
      These are each in a subdirectory of
      \trl{src/<class>/impls}. Except in rare
      circumstances data
      structures defined here should not be referenced from outside this
      directory.
\end{itemize}

Each type of object, for instance a vector, is defined in its own
public include file, by
\begin{tabbing}
   typedef \_p\_$<$class$>$* $<$class$>$; (for example, typedef \_p\_Vec* Vec;).
\end{tabbing}
  This organization
allows the compiler to perform type checking on all subroutine calls
while at the same time
completely removing the details of the implementation of {\tt
\_p\_$<$class$>$} from the application code.  This capability is extremely important
because it allows the library internals to be changed
without altering or recompiling the application code.


\section{Common Object Header}

All PETSc/PETSc objects have the following common header structures
defined in \href{http://www.mcs.anl.gov/petsc/petsc-master/include/petsc/private/petscimpl.h.html}{include/petsc/private/petscimpl.h}

\begin{tabbing}
\\
/* Function table common to all PETSc compatible classs */\\
\\
typedef struct \{ \\
   int (*getcomm)(PetscObject,MPI\_Comm *);\\
   int (*view)(PetscObject,Viewer);\\
   int (*destroy)(PetscObject);\\
   int (*query)(PetscObject,const char *,PetscObject *);\\
   int (*compose)(PetscObject,const char*,PetscObject);\\
   int (*composefunction)(PetscObject,const char *,void(*)(void));\\
   int (*queryfunction)(PetscObject,const char *,void (**)(void));\\
\} PetscOps;\\
\\
/* Data structure header common to all PETSc compatible classs */\\
\\
struct \_p\_$<$class$>$ \{\\
  PetscClassId     classid;                                  \\
  PetscOps         *bops;                                   \\
  $<$class$>$Ops   *ops;                                    \\
  MPI\_Comm         comm;                                    \\
  PetscLogDouble  flops,time,mem;                          \\
  int              id;                                      \\
  int              refct;                                   \\
  int              tag;                                     \\
  DLList           qlist;                                   \\
  OList            olist;                                   \\
  char             *type\_name;                              \\
  PetscObject      parent;                                  \\
  char             *name;                                    \\
  char             *prefix;                                 \\
  void             *cpp;\\
  void             **fortran\_func\_pointers;       \\
  ..........\\
  CLASS-SPECIFIC DATASTRUCTURES\\
\};
\end{tabbing}
Here \trl{<class>ops} is a function table (like the \trl{PetscOps} above) that
contains the function pointers for the operations specific to that class.
For example, the PETSc vector class object looks like

\begin{tabbing}
\\
/* Function table common to all PETSc compatible vector objects */\\
\\
typedef struct \_VecOps* VecOps;\\
struct \_VecOps \{\\
  PetscErrorCode  (*duplicate)(Vec,Vec*),           /* get single vector */\\
       (*duplicatevecs)(Vec,int,Vec**),  /* get array of vectors */\\
       (*destroyvecs)(Vec*,int),         /* free array of vectors */\\
       (*dot)(Vec,Vec,Scalar*),          /* z = x\^H * y */\\
       (*mdot)(int,Vec,Vec*,Scalar*),    /* z[j] = x dot y[j] */\\
       (*norm)(Vec,NormType,double*),    /* z = sqrt(x\^H * x) */\\
       (*tdot)(Vec,Vec,Scalar*),         /* x'*y */\\
       (*mtdot)(int,Vec,Vec*,Scalar*),   /* z[j] = x dot y[j] */\\
       (*scale)(Scalar*,Vec),            /* x = alpha * x   */\\
       (*copy)(Vec,Vec),                 /* y = x */\\
       (*set)(Scalar*,Vec),              /* y = alpha  */\\
       (*swap)(Vec,Vec),                 /* exchange x and y */\\
       (*axpy)(Scalar*,Vec,Vec),         /* y = y + alpha * x */\\
       (*axpby)(Scalar*,Scalar*,Vec,Vec),/* y = y + alpha * x + beta * y*/\\
       (*maxpy)(int,Scalar*,Vec,Vec*),   /* y = y + alpha[j] x[j] */\\
       (*aypx)(Scalar*,Vec,Vec),         /* y = x + alpha * y */\\
       (*waxpy)(Scalar*,Vec,Vec,Vec),    /* w = y + alpha * x */\\
       (*pointwisemult)(Vec,Vec,Vec),    /* w = x .* y */\\
       (*pointwisedivide)(Vec,Vec,Vec),  /* w = x ./ y */\\
       (*setvalues)(Vec,int,int*,Scalar*,InsertMode),\\
       (*assemblybegin)(Vec),            /* start global assembly */\\
       (*assemblyend)(Vec),              /* end global assembly */\\
       (*getarray)(Vec,Scalar**),        /* get data array */\\
       (*getsize)(Vec,int*),(*getlocalsize)(Vec,int*),\\
       (*getownershiprange)(Vec,int*,int*),\\
       (*restorearray)(Vec,Scalar**),    /* restore data array */\\
       (*max)(Vec,int*,double*),         /* z = max(x); idx=index of max(x) */\\
       (*min)(Vec,int*,double*),         /* z = min(x); idx=index of min(x) */\\
       (*setrandom)(PetscRandom,Vec),    /* set y[j] = random numbers */\\
       (*setoption)(Vec,VecOption),\\
       (*setvaluesblocked)(Vec,int,int*,Scalar*,InsertMode),\\
       (*destroy)(Vec),\\
       (*view)(Vec,Viewer);\\
\};\\
\\
/* Data structure header common to all PETSc vector classs */\\
\\
struct \_p\_Vec \{\\
  PetscClassId           classid;                                  \\
  PetscOps               *bops;                                   \\
  VecOps                 *ops;                                    \\
  MPI\_Comm               comm;\\
  PetscLogDouble         flops,time,mem;                          \\
  int                    id;                                      \\
  int                    refct;                                   \\
  int                    tag;                                     \\
  DLList                 qlist;                                   \\
  OList                  olist;                                   \\
  char                   *type\_name;                              \\
  PetscObject            parent;                                  \\
  char*                  name;                                    \\
  char                   *prefix;                                 \\
  void**                 fortran\_func\_pointers;       \\
  void                   *data;     /* implementation-specific data */\\
  PetscLayout            map;
  ISLocalToGlobalMapping mapping;   /* mapping used in VecSetValuesLocal() */\\
  ISLocalToGlobalMapping bmapping;  /* mapping used in VecSetValuesBlockedLocal() */\\
\};
\end{tabbing}

Each PETSc object begins with a PetscClassId which is used for error checking.
Each different class of objects has its value for the classid; these are used
to distinguish between classes. When a new class is created one needs to call
\begin{tabbing}
  ierr = PetscClassIdRegister(const char *classname,PetscClassId *classid);CHKERRQ(ierr);
\end{tabbing}
For example,
\begin{tabbing}
  ierr = PetscClassIdRegister(''index set'',\&IS\_CLASSID);CHKERRQ(ierr);
\end{tabbing}
{\bf Question: Why is a fundamental part of PETSc objects defined in PetscLog when PETSc Log
is something that can be "turned off"} One can verify that an object is valid of a particular
class with
\begin{tabbing}
  PetscValidHeaderSpecific(x,VEC\_CLASSID,1);
\end{tabbing}
The third argument to this macro indicates the position in the calling sequence of the
function the object was passed in. This is generate more complete error messages.

To check for an object of any type use
\begin{tabbing}
  PetscValidHeader(x,1);
\end{tabbing}

\section{Common Object Functions}

Several routines are provided for manipulating data within the header, these include the specific functions in the PETSc common function table.

\begin{itemize}
\item \trl{getcomm(PetscObject,MPI_Comm *)} obtains the MPI communicator associated
      with this object.

\item \trl{view(PetscObject,Viewer)} allows one to store or visualize the data inside
      an object. If the Viewer is null than should cause the object to print
      information on the object to standard out. PETSc provides a variety of simple
      viewers.

\item \trl{destroy(PetscObject)} causes the reference count of the object to be decreased
      by one or the object to be destroyed and all memory used by the object to be freed when
      the reference count drops to zero.
      If the object has any other objects composed with it then they are each sent a
      \trl{destroy()}, i.e. the \trl{destroy()} function is called on them also.

\item \trl{compose(PetscObject,const char *name, PetscObject)} associates the second object with
      the first object and increases the reference count of the second object. If an
      object with the
      same name was previously composed that object is dereferenced and replaced with
      the new object. If the
      second object is null and and object with the same name has already been
      composed that object is dereferenced (the \trl{destroy()} function is called on
      it, and that object is removed from the first object); i.e. this is a way to
      remove, by name, an object that was previously composed.

\item \trl{query(PetscObject,const char *name, PetscObject *)} retrieves an object that was
      previously composed with the first object. Retreives a null if no object with
      that name was previously composed.

\item \trl{composefunction(PetscObject,const char *name,const char *fname,void *func)} associates a function
      pointer to an object. If the object already had a composed function with the
      same name, the old one is replace. If the fname is null it is removed from
      the object. The string \trl{fname} is the  character string name of the function;
      it may include the path name or URL of the dynamic library where the function is located.
      The argument \trl{name} is a ``short'' name of the function to be used with the
      \trl{queryfunction()} call. On systems that support dynamic libraries the \trl{func}
      argument is ignored; otherwise \trl{func} is the actual function pointer.

      For example, \trl{fname} may be \trl{libpetscksp:PCCreate_LU} or
      \trl{http://www.mcs.anl.gov/petsc/libpetscksp:PCCreate_LU}.

\item \trl{queryfunction(PetscObject,const char *name,void **func)} retreives a function pointer that
      was associated with the object. If dynamic libraries are used the function is loaded
      into memory at this time (if it has not been previously loaded), not when the
      \trl{composefunction()} routine was called.

\end{itemize}

Since the object composition allows one to {\bf only} compose PETSc objects
with PETSc objects rather than any arbitrary pointer, PETsc provides
the convenience object PetscContainer, created with the
routine PetscContainerCreate(MPI\_Comm,PetscContainer*)
to allow one to wrap any kind of data into a PETSc object that can then be
composed with a PETSc object.

% --------------------------------------------------------------------------------------
\section{PETSc Implementation of the  Object Functions}

This sections discusses how PETSc implements the \trl{compose()}, \trl{query()}, {\tt
composefunction()}, and \trl{queryfunction()} functions for its object implementations.
Other PETSc compatible class implementations are free to manage these functions in any
manner; but generally they would use the PETSc defaults so that the library writer does
not have to ``reinvent the wheel.''

\subsection{Compose and Query}

In \href{http://www.mcs.anl.gov/petsc/petsc-master/src/objects/olist.c.html}{src/sys/objects/olist.c} PETSc defines a C struct
\begin{tabbing}
typedef struct \_PetscObjectList *PetscObjectList;\\
struct \_PetscObjectList \{\\
    char             name[128];\\
    PetscObject      obj;\\
    PetscObjectList  next;\\
\};
\end{tabbing}
from which linked lists of composed objects may be constructed. The routines
to manipulate these elementary objects are
\begin{tabbing}
  int PetscObjectListAdd(PetscObjectList *fl,const char *name,PetscObject obj );\\
  int PetscObjectListDestroy(PetscObjectList fl );\\
  int PetscObjectListFind(PetscObjectList fl, const char *name, PetscObject *obj)\\
  int PetscObjectListDuplicate(PetscObjectList fl, PetscObjectList *nl);
\end{tabbing}
The function PetscObjectListAdd() will create the initial PetscObjectList if the argument
\trl{fl} points to a null.

The PETSc object \trl{compose()} and \trl{query()} functions are then simply
(defined in \href{http://www.mcs.anl.gov/petsc/petsc-master/src/objects/inherit.c.html}{src/sys/objects/inherit.c})
\begin{tabbing}
  PetscErrorCode PetscObjectCompose\_Petsc(PetscObject obj,const char *name,PetscObject ptr)\\
  \{\\
    PetscErrorCode ierr;\\
\\
    PetscFunctionBegin;\\
    ierr = PetscObjectListAdd(\&obj-$>$olist,name,ptr);CHKERRQ(ierr);\\
    PetscFunctionReturn(0);\\
  \}\\
\\
  PetscErrorCode PetscObjectQuery\_Petsc(PetscObject obj,const char *name,PetscObject *ptr)\\
  \{\\
    PetscErrorCode ierr;\\
\\
    PetscFunctionBegin;\\
    ierr = PetscObjectListFind(obj-$>$olist,name,ptr);CHKERRQ(ierr);\\
    PetscFunctionReturn(0); \\
  \}
\end{tabbing}

\subsection{Compose and Query Function}

PETSc allows one to compose functions bying name and function pointer. In \href{http://www.mcs.anl.gov/petsc/petsc-master/src/sys/dll/reg.c.html}{src/sys/dll/reg.c},
PETSc defines the linked list structure

\begin{tabbing}
typedef struct \_PetscFunctionList *PetscFunctionList;\\
st\=ruct \_PetscFunctionList \{\\
 \>void   (*routine)(void);\\
 \>char   *name;  \\
 \>PetscFunctionList  *next;\\
\};
\end{tabbing}

Each PETSc object contains a PetscFunctionList object. The \trl{composefunction()} and
\trl{queryfunction()} are given by

\begin{tabbing}
PetscErrorCode PetscObjectComposeFunction\_Petsc(\=PetscObject obj,const char *name,\\
   \> void *ptr)\\
\{\\
  PetscErrorCode ierr;\\
\\
  PetscFunctionBegin;\\
  ierr = PetscFunctionListAdd(\&obj-$>$qlist,name,fname,ptr);CHKERRQ(ierr);\\
  PetscFunctionReturn(0);\\
\}\\
\\
PetscErrorCode PetscObjectQueryFunction\_Petsc(PetscObject obj,const char *name,void (**ptr)(void))\\
\{\\
  PetscErrorCode ierr;\\
\\
  PetscFunctionBegin;\\
  ierr = PetscFunctionListFind(obj-$>$qlist,name,ptr);CHKERRQ(ierr);\\
  PetscFunctionReturn(0);\\
\}
\end{tabbing}

In addition to using the PetscFunctionList mechanism to compose functions into PETSc objects, it is
also used to allow registration of new class implementations; for example, new
preconditioners, see Section \ref{sec:registeringnewmethods}.

\subsection{Simple PETSc Objects}

There are some simple PETSc objects that do not need PETSCHEADER - and
the associated functionality. These objects are internally named as
\_n\_$<$class$>$ as opposed to \_p\_$<$class$>$. For eg: \_n\_PetscTable
vs \_p\_Vec.

% -----------------------------------------------------------------------------------
\chapter{PetscObjects}

\section{Elementary Objects: IS, Vec, Mat}

\section{Solver Objects: PC, KSP, SNES, TS}

\subsection{Preconditioners: PC}

The base PETSc PC object is defined in the  \href{http://www.mcs.anl.gov/petsc/petsc-master/include/petsc/private/pcimpl.h.html}{include/petsc/private/pcimpl.h} include file.
A carefully commented implementation of a PC object can be found in
\href{http://www.mcs.anl.gov/petsc/petsc-master/src/ksp/pc/impls/jacobi/jacobi.c.html}{src/ksp/pc/impls/jacobi/jacobi.c}.


\subsection{Krylov Solvers: KSP}
The base PETSc KSP object is defined in the \href{http://www.mcs.anl.gov/petsc/petsc-master/include/petsc/private/kspimpl.h.html}{include/petsc/private/kspimpl.h} include file.
A carefully commented implementation of a KSP object can be found in
\href{http://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/impls/cg/cg.c.html}{src/ksp/ksp/impls/cg/cg.c}.

\subsection{Registering New Methods}
\label{sec:registeringnewmethods}

See \href{http://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/examples/tutorials/ex12.c.html}{src/ksp/examples/tutorials/ex12.c} for an example of registering a new
preconditioning (PC) method.


% -----------------------------------------------------------------------------------

% -----------------------------------------------------------------------------------
\chapter{The Various Matrix Classes}
\label{sec:matclasses}

PETSc provides a variety of matrix implementations, since no single
matrix format is appropriate for all problems.  This section first
discusses various matrix blocking strategies, and then
describes the assortment of matrix types within PETSc.

\section{Matrix Blocking Strategies}
\sindex{matrix blocking}
\sindex{blocking}

In today's computers, the time to perform an arithmetic operation is
dominated by the time to move the data into position, not the time to
compute the arithmetic result.  For example, the time to perform a
multiplication operation may be one clock cycle, while the time to
move the floating point number from memory to the arithmetic unit may
take 10 or more cycles. To help manage this difference in time scales,
most processors have at least three levels of memory: registers,
cache, and random access memory, RAM. (In addition, some processors
have external caches, and the complications of paging introduce
another level to the hierarchy.)

Thus, to achieve high performance, a code should first move data into
cache, and from there move it into registers and use it repeatedly
while it remains in the cache or registers before returning it to main
memory. If one reuses a floating point number 50 times while it is in
registers, then the ``hit'' of 10 clock cycles to bring it into the
register is not important. But if the floating point number is used
only once, the ``hit'' of 10 clock cycles becomes very noticeable,
resulting in disappointing flop rates.

Unfortunately, the compiler controls the use of the registers, and the
hardware controls the use of the cache. Since the user has essentially
no direct control, code must be written in such a way that the
compiler and hardware cache system can perform well. Good quality code
is then be said to respect the memory hierarchy.

The standard approach to improving the hardware utilization is to use
blocking. That is, rather than working with individual elements in
the matrices, one employs blocks of elements.  Since the use of
implicit methods in PDE-based simulations leads to matrices with a
naturally blocked structure (with a block size equal to the number of
degrees of freedom per cell), blocking is extremely advantageous.  The
PETSc sparse matrix representations use a variety
of techniques for blocking, including

\begin{itemize}
\item storing the matrices using a generic sparse matrix format, but
   storing additional information about adjacent rows with identical
   nonzero structure (so called I-nodes); this I-node information is
   used in the key computational routines to improve performance
   (the default for the MATSEQAIJ and MATMPIAIJ formats);
\item storing the matrices using a fixed (problem dependent) block size
   (via the MATSEQBAIJ and MATMPIBAIJ formats); and
\end{itemize}

The advantage of the first approach is that it is a minimal change
from a standard sparse matrix format and brings a large percent of the
improvement one obtains via blocking.  Using a fixed block size gives
the best performance, since the code can be hardwired with that
particular size (for example, in some problems the size may be 3, in
others 5, etc.), so that the compiler will then optimize for that
size, removing the overhead of small loops entirely.

The following table presents the floating point performance
for a basic matrix-vector product using these three approaches: a basic
compressed row storage format (using the PETSc runtime options
\trl{-mat_seqaij -mat_no_unroll)}; the same compressed row format using
I-nodes (with the option {-mat\_seqaij}); and a fixed block size code,
with a block size of three for these problems (using the option
{-mat\_seqbaij}). The rates were computed on one
node of an older IBM SP, using two test matrices.  The first matrix
(ARCO1), courtesy of Rick Dean of Arco, arises in multiphase flow
simulation; it has 1501 degrees of freedom, 26,131 matrix nonzeros
and, a natural block size of 3, and a small number of well terms. The
second matrix (CFD), arises in a three-dimensional Euler flow
simulation and has 15,360 degrees of freedom, 496,000 nonzeros, and a
natural block size of 5. In addition to displaying the flop rates for
matrix-vector products, we also display them for triangular solve
obtained from an ILU(0) factorization.

\medskip
\centerline{
\begin{tabular}{|c|c|c|c|c|c|}
\hline
Problem & Block size & Basic & I-node version & Fixed block size \\
\hline
\multicolumn{5}{c}{{\em Matrix-Vector Product (Mflop/sec)}} \\
\hline
Multiphase & 3 & 27 & 43 & 70 \\
Euler & 5 &  28 & 58 & 90 \\
\hline
\multicolumn{5}{c}{{\em Triangular Solves from ILU(0) (Mflop/sec)}}\\
\hline
Multiphase & 3 & 22 & 31 & 49 \\
Euler      & 5 & 22 & 39 & 65 \\
\hline
\end{tabular}
}
\medskip

These examples demonstrate that careful implementations of the basic
sequential kernels in PETSc can dramatically improve overall floating
point performance, and users can immediately benefit from such
enhancements without altering a single line of their application
codes.  Note that the speeds of the I-node and fixed block operations
are several times that of the basic sparse implementations.  The
disappointing rates for the variable block size code occur because
even on a sequential computer, the code performs the matrix-vector
products and triangular solves using the coloring introduced above and
thus does not utilize the cache particularly efficiently.  This is an
example of improving the parallelization capability at the expense of
using each processor less efficiently.

\subsection{Sequential AIJ Sparse Matrices}

The default matrix representation within PETSc is the general sparse
AIJ format (also called the Yale sparse matrix format or compressed
sparse row format, CSR).

\subsection{Parallel AIJ Sparse Matrices}

This matrix type, which is the
default parallel matrix format; additional implementation details are
given in \cite{petsc-efficient}.

\subsection{Sequential Block AIJ Sparse Matrices}

The sequential and parallel block AIJ formats, which are extensions of
the AIJ formats described above, are intended especially for use with
multiclass PDEs.  The block variants store matrix elements by
fixed-sized dense \trl{nb} $\times$ \trl{nb} blocks.  The stored row
and column indices begin at zero.

The routine for creating a sequential block AIJ matrix with \trl{m}
rows, \trl{n} columns, and a block size of \trl{nb} is
\begin{tabbing}
   ierr = MatCreateSeqBAIJ(MPI\_Comm comm,int nb,int m,int n,int nz,int *nnz, Mat *A)
\end{tabbing}
\findex{MatCreateSeqBAIJ}
The arguments \trl{nz} and \trl{nnz} can be used to preallocate matrix
memory by indicating the number of {\em block} nonzeros per row.  For good
performance during matrix assembly, preallocation is crucial; however, the
user can set \trl{nz=0} and \trl{nzz=NULL} for PETSc to dynamically
allocate matrix memory as needed.  The PETSc users manual
discusses preallocation for the AIJ format; extension to the block AIJ
format is straightforward.

Note that the routine MatSetValuesBlocked()
\findex{MatSetValuesBlocked()} can be used for more efficient matrix assembly
when using the block AIJ format.

\subsection{Parallel Block AIJ Sparse Matrices}

Parallel block AIJ matrices with block size {\t nb} can be created with
the command \findex{MatCreateBAIJ()}
\begin{tabbing}
   ierr = MatCreateBAIJ(MPI\_Comm comm,int nb,int m,int n,int M,int N,int d\_nz,\\
                          int *d\_nnz, int o\_nz,int *o\_nnz,Mat *A);
\end{tabbing}
\trl{A} is the newly created matrix, while the arguments \trl{m}, \trl{n},
\trl{M}, and \trl{N}, indicate the number of local rows and columns and
the number of global rows and columns, respectively. Either the local or
global parameters can be replaced with PETSC\_DECIDE, so that
PETSc will determine \findex{PETSC_DECIDE} them.
The matrix is stored with a fixed number of rows on
each processor, given by \trl{m}, or determined by PETSc if \trl{m} is
PETSC\_DECIDE.

If PETSC\_DECIDE is not used for
\trl{m} and \trl{n} then the user must ensure that they are chosen to be
compatible with the vectors. To do this, one first considers the product
$y = A x$. The \trl{m} that one uses in MatCreateBAIJ()
must match the local size used in the VecCreateMPI() for \trl{y}.
The \trl{n} used must match that used as the local size in
VecCreateMPI() for \trl{x}.

The user must set \trl{d_nz=0}, \trl{o_nz=0}, \trl{d_nnz=NULL}, and
\trl{o_nnz=NULL} for PETSc to control dynamic allocation of matrix
memory space.  Analogous to \trl{nz} and \trl{nnz} for the routine
MatCreateSeqBAIJ(), these arguments optionally specify
block nonzero information for the diagonal (\trl{d_nz} and \trl{d_nnz}) and
off-diagonal (\trl{o_nz} and \trl{o_nnz}) parts of the matrix.
For a square global matrix, we define each processor's diagonal portion
to be its local rows and the corresponding columns (a square submatrix);
each processor's off-diagonal portion encompasses the remainder of the
local matrix (a rectangular submatrix).
The PETSc users manual gives an example of preallocation for
the parallel AIJ matrix format; extension to the block parallel AIJ case
is straightforward.

\subsection{Sequential Dense Matrices}

PETSc provides both sequential and parallel dense matrix formats,
where each processor stores its entries in a column-major array in the
usual Fortran77 style.

\subsection{Parallel Dense Matrices}

The parallel dense matrices are partitioned by rows across the
processors, so that each local rectangular submatrix is stored in the
dense format described above.

\bibliographystyle{plain}
\bibliography{../petsc}

\end{document}
