PETSc-FEM - a general purpose, parallel, multi-physics FEM program


This is PETSc-FEM, a general purpose, parallel, multi-physics FEM (Finite Element Method) program for CFD (Computational Fluid Dynamics) applications based on PETSc . PETSc-FEM comprises both a library that allows the user to develop FEM (or FEM-like, i.e. non-structured mesh oriented) programs, and a suite of application programs. It is written in the C++ language with an OOP (Object Oriented Programming) philosophy, keeping in mind the scope of efficiency. PETSc-FEM may run in parallel using the MPI standard on a variety of architectures, but currently it has been tested in Beowulf clusters only.

PETSc-FEM is GPL, for further details read the file COPYING in the root directory of the distribution.


This file belongs to the PETSc - FEM package a library and application suite oriented to the Finite Element Method based on PETSc. Copyright (C) 1999-2008, Mario Alberto Storti, Norberto Marcelo Nigro, Rodrigo Rafael Paz, Lisandro Dalcin and Ezequiel Lopez, Centro Internacional de Metodos Numericos en Ingenieria (CIMEC-Argentina), Universidad Nacional del Litoral (UNL-Argentina), Consejo Nacional de Investigaciones Cientificas y Tecnicas (CONICET-Argentina).

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.



Currently, PETSc-FEM is known to run only on Beowulf class clusters based on Intel processors, running the GNU/Linux OS. Off course, you can also install PETSc-FEM in a single GNU/Linux box, for use in sequential (i.e. non-parallel) mode or either for simulating parallel runs, by launching several processes in the same processor. (This is a feature of MPI). PETSc-FEM compiles cleanly with the GNU compiler GCC 4.1.1 that comes with the Fedora Core 5 GNU/Linux distribution.

I guess it should be relatively easy to install in other processor architectures (SPARC or Alpha) running GNU/Linux and also on other Unix flavor's.

Required Software

Before installing PETSc-FEM, please verify you have the following packages correctly installed in your system.


Message Passing Interface. We are currently using MPICH2 1.0.3 ( but I think it should be possible to use other versions of MPI like LAM or OpenMPI.


Portable, Extensible Toolkit for Scientific Computation and can be found at Current version of PETSc-FEM compiles with PETSc 2.1.6.


This is, in fact, a prerequisite to install PETSc and can be found at


This is a C library of generic containers and can be found at I have learned recently that development of Libretto has been stopped since it has been superseded by GLIB, so that I think that in a future I will replace the Libretto code by GLIB. Also, the STL library (that comes with gcc) can replace all or some of the features provided by Libretto but I found that the STL implementation (at least that one that comes with gcc) is rather inefficient.


GLIB is a library which includes support routines for C such as lists, trees, hashes, memory allocation, and many other things. The official web site is [RedHat RPM packages: GLIB, glib-devel]


This is a matrix library to be used mainly at the element routine level, i.e. for small matrices and can be found at It is not mandatory to use Newmat for all element calculations. In fact there is included a matrix class called FastMat2 that is more performing, but Newmat is still used in some internal parts and has much more functions, so that you have to install it. Also, you may prefer Newmat over FastMat2 for element calculations since it supports overloading of algebraic operations. See the documentation on FastMat2 in the PETSc-FEM documentation for reasons why to use FastMat2 instead of Newmat.


METIS is a software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. It can be found at


The Meschach Library is a numerical library of C routines for performing calculations on matrices and vectors. It is used only for some functions that can not be done with Newmat. Currently, it is used in the advective module for performing eigenvalue decomposition of non-symmetric matrices. I think that in a future this functionality could be replaced by calls to LAPACK.


If you want to use ePerl then you need to have Perl installed. [RedHat RPM package: perl].

SuperLU (optional)

A collection of ANSI C routines to solve sparse linear systems of equations. Can be found at This package is optional, install it if you think that SuperLU is faster than PETSc for direct factorization of sparse matrices.

ANN (optional)

This is the Approximate Nearest Neighbor Library and can be found at Currently it is only used in the Navier-Stokes module for calculating the damping functions in the Smagorinsky model. However, in a future it may be used more widely for some mesh manipulations. Currently it is optional.

pthreads library (optional)

This is a POSIX compliant library for threads, currently used for asynchronous communication with the IBM Data Explorer visualization tool, but it might be used for other things in a near future. It comes with gcc, so that if you use gcc, you probably have it already installed.

Simple Sockets Library (SSL) (optional)

The Simple Sockets Library (SSL) allows C programmers to use interprocess communications via Berkeley sockets simply and quickly. Most of the SSL's functions resemble C's file i/o functions, so C programmers will find the SSL easy to learn. Currently used for communication with the IBM Data Explorer visualization tool, but but it might be used for other things in a near future. [Current version: 2.11a Tue Mar 6 09:46:16 EST 2001] [Note: If you do a web search you can find it as a file COSMIC.tar.gz or ssl.tar.gz].


Once you got the petscfem-xxx.tar.gz package (xxx stands for the version), gunzip and untar it like this:

  $ gunzip petscfem-xxx.tar.gz
  $ tar xvf petscfem-xxx.tar

This will create a petscfem-xxx directory with all the source files included.

Makefile variables


The root of the PETSc-FEM package


These are variables relative to PETSc. See documentation.


Once you have all the software mentioned in Required Software installed, you have to define the variables in the CONFIGURATION VARIABLES section in file $(PETSCFEM_DIR)/Makefile.base.


cd to directory $(PETSCFEM_DIR) and do

  $ make depend
  $ make libpetscfem        # to build the library
  $ make all                # to build the library and all the modules


  $ make what

can help you on other useful targets.

If nothing goes wrong you will end with a src/libpetscfem.a library and application binaries (ending in .bin) in their respective directories, e.g. applications/ns/ns.bin is the Navier-Stokes binary.


Documentation for PETSc-FEM is included in the $(PETSCFEM_DIR)/doc directory. There is basically a reference manual written in LaTeX (doc/petscfem.tex), and embedded documentation for the routines to be used with Doc++. In the standard distrib you should find both of them converted to HTML (LaTeX is converted with latex2html) and accessible from doc/index.html. Other formats (perhaps PostScript and PDF) may be also found in the doc directory.


Many tests and examples are in the $(PETSCFEM_DIR)/test directory.


The best way to start coding an application program is to duplicate a directory (for instance applications/advdif) or (for instance applications/ns) to a directory from your own (i.e. outside the $(PETSCFEM_DIR) tree, say /your/appl/path. Configure the $(PETSCFEM_DIR) variable in the local makefile (i.e. /your/appl/path/Makefile).You should have to configure other variables as well, perhaps $(MYOBJS) (this is a list of the object (.o) files in this local directory) and $(PROG) (the name of the program to be built, without the suffix .bin).


Mario A. Storti* <>

PETSc-FEM kernel, NS and AdvDif modules.

Norberto M. Nigro* <"

NS and AdvDif modules, multi-phase flow.

Rodrigo R. Paz* <>

AdvDif module, hydrology module, compressible flow, fluid-structure interaction, preconditioners.

Lisandro Dalcin* <>

PETSc-FEM kernel, Python extension language project, linear algebra, preconditioners, multigrid.

Ezequiel Lopez* <>

Mesh relocation algorithms.

Laura Battaglia* <>

Free surface algorithms.

Gustavo Rios Rodriguez * <>

Adaptive refinement.

* CIMEC, Internacional Center for Computational Methods in Engineering, Santa Fe, Argentina.| Ordinary mail: Mario Storti, CIMEC-INTEC, Guemes 3450, 3000 Santa Fe, Argentina.