Compilation of Fortran 90 code that uses PETSc

See also questions close to this topic

• makefile rules being applied when they shouldn't

  I have a makefile (pasted below) for some C++ code, which has the annoying problem that when I do make clean the dependency files are compiled (before being removed again), which makes make clean too slow. The dependency rules are

  $(DEPENDDIR)%.d: %.cpp
      @mkdir -p $(DEPENDDIR)
      $(CXX) -M -MG -MT $(OBJECTDIR)$*.o $(CXXFLAGS) $< > $@
  

  Can anyone see what is the problem?

  I tried moving the dependencies to the part where the objects are compiled, ie:

  $(OBJECTDIR)%.o: %.cpp
      @mkdir -p $(OBJECTDIR) # $(dir $@)
      @echo " "
      $(CXX) -M -MG -MT $@ $(CXXFLAGS) $< \
          -MF $(patsubst $(OBJECTDIR)%.o, $(DEPENDDIR)%.d, $@)
  

  but then the header file version.hpp wasn't created (and compilation failed)

  KERNEL := $(shell uname -s)
  
  PROGNAME=nextsim.exec
  
  CXX = g++
  
  # setting the C++ standard according to the gcc compiler version (from gcc-5.2, the default is C++14)
  ifeq ($(shell echo `$(CXX) -dumpversion | cut -f1-2 -d.` \>= 5.2 | sed -e 's/\.//g' | bc),1)
      CXXFLAGS += -std=c++14
  else
      CXXFLAGS += -std=c++11
  endif
  
  # add g++ option flags
  CXXFLAGS += -ftemplate-depth-256 -Wno-inline \
          -fPIC -fopenmp \
          -DHAVE_CONFIG_H -D_MULTITHREADING_
  ifdef NEXTSIM_COMPILE_VERBOSE
      CXXFLAGS += -v
  endif
  
  ifdef USE_OASIS
      CXXFLAGS += -DOASIS
      CXXFLAGS += -I $(NEXTSIMDIR)/modules/oasis/include
      LDFLAGS += -lgfortran
      LDFLAGS += -L $(NEXTSIMDIR)/lib -loasis
      CHAN = MPI1
      #LIBPSMILE = $(OASIS_DIR)/lib/libpsmile.${CHAN}.a $(OASIS_DIR)/lib/libmct.a $(OASIS_DIR)/lib/libmpeu.a $(OASIS_DIR)/lib/libscrip.a
  endif
  
  ifneq (,$(strip $(filter DEBUG Debug debug PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
      #ifeq ($(NEXTSIM_BUILD_TYPE),$(filter $(NEXTSIM_BUILD_TYPE),Debug debug))
      CXXFLAGS := $(filter-out -O3 -pthread,$(CXXFLAGS))
      CXXFLAGS += -g -O0 -DNDEBUG
  ifneq (,$(strip $(filter PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
      CXXFLAGS += -DWITHGPERFTOOLS
  endif
  ifneq ($(KERNEL),Linux)
      CXXFLAGS += -Wl,-no_pie
  endif
  else
      CXXFLAGS += -O3 -pthread
  endif
  
  # add include paths
  CXXFLAGS += -I $(NEXTSIMDIR)/core/include
  CXXFLAGS += -isystem $(NEXTSIMDIR)/contrib/bamg/include # suppress annoying compilation warnings from -I
  CXXFLAGS += -isystem $(NEXTSIMDIR)/contrib/mapx/include # suppress annoying compilation warnings from -I
  # CXXFLAGS += -I $(NEXTSIMDIR)/contrib/interp/include
  
  ifdef USE_ENSEMBLE
      CXXFLAGS += -DENSEMBLE
      CXXFLAGS += -I $(NEXTSIMDIR)/modules/enkf/perturbation/include
  endif
  
  ifdef USE_AEROBULK
          CXXFLAGS += -I $(AEROBULK_DIR)/include
          CXXFLAGS += -DAEROBULK
  endif
  
  # openmpi
  CXXFLAGS += -I $(OPENMPI_INCLUDE_DIR)/
  
  # petsc
  CXXFLAGS += -I $(PETSC_DIR)/include
  
  # boost
  CXXFLAGS += -I $(BOOST_INCDIR)/ -I .
  
  # netcdf
  CXXFLAGS += -I $(NETCDF_DIR)/include
  
  # gmsh
  CXXFLAGS += -I $(GMSH_DIR)/include/gmsh
  
  CXXFLAGS += -I /opt/local/include
  
  ifeq ($(KERNEL),Linux)
      #CXXFLAGS += -std=c++0x -std=c++11 -pedantic -ftemplate-depth-256 -Wno-inline -fPIC -g -lm -pthread -v #-MMD -MP -lm -pthread -v
  
  else
  
      ifeq ($(CXX),clang)
      CXXFLAGS += -stdlib=libc++
      endif
  
      CXXFLAGS += -I /usr/local/include #-I /opt/local/include/openmpi-mp
  
      #LDFLAGS += -Wl,-rpath,/usr/local/lib #-Wl,-rpath,/opt/local/lib/openmpi-mp
      #LDFLAGS += -L /usr/local/lib #-L /opt/local/lib/openmpi-mp -lmpi_cxx -lmpi -ldl -lstdc++ -lpthread
  
      ifeq ($(CXX),clang)
      LDFLAGS += -stdlib=libc++
      endif
  
  endif
  
  LDFLAGS += -L /usr/local/lib
  
  LDFLAGS += -Wl,-rpath,$(OPENMPI_LIB_DIR)/
  ifndef MACHINE_HEXAGON
      LDFLAGS += -L $(OPENMPI_LIB_DIR)/ -lmpi_cxx -lmpi -ldl -lstdc++ #-lpthread
  else
      LDFLAGS += -L $(OPENMPI_LIB_DIR)/ -lmpichcxx -lmpich -ldl -lstdc++ #-lpthread #-lssl -luuid -lpthread -lrt
          LDFLAGS += -Wl,-rpath,$(BLAS_LAPACK_DIR)/lib
          LDFLAGS += -L $(BLAS_LAPACK_DIR)/lib -lsci_gnu_mp
  endif
  
  LDFLAGS += -Wl,-rpath,$(NETCDF_DIR)/lib -L $(NETCDF_DIR)/lib -lnetcdf_c++4
  
  LDFLAGS += -Wl,-rpath,$(BOOST_LIBDIR)
  LDFLAGS += -L $(BOOST_LIBDIR) -lboost_program_options -lboost_filesystem -lboost_system -lboost_serialization -lboost_mpi -lboost_date_time
  
  LDFLAGS += -Wl,-rpath,$(PETSC_DIR)/lib
  LDFLAGS += -L $(PETSC_DIR)/lib -lpetsc
  
  LDFLAGS += -Wl,-rpath,$(NEXTSIMDIR)/lib
  LDFLAGS += -L $(NEXTSIMDIR)/lib -lbamg
  #LDFLAGS += -L $(NEXTSIMDIR)/lib -linterp
  LDFLAGS += -L $(NEXTSIMDIR)/lib -lmapx
  #LDFLAGS += -L $(NEXTSIMDIR)/lib -loasis
  
  ifdef USE_ENSEMBLE
      LDFLAGS += -L $(NEXTSIMDIR)/lib -lpseudo2D
      LDFLAGS += -lgfortran
  endif
  
  ifdef USE_AEROBULK
          LDFLAGS += -L $(AEROBULK_DIR)/lib -laerobulk_cxx -laerobulk
          LDFLAGS += -lgfortran
  endif
  
  ifneq (,$(strip $(filter DEBUG Debug debug PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
  #ifeq ($(NEXTSIM_BUILD_TYPE),$(filter $(NEXTSIM_BUILD_TYPE),Debug debug))
      LDFLAGS += -Wl,-rpath,/opt/local/lib
  ifneq (,$(strip $(filter PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
      LDFLAGS += -L /opt/local/lib -lprofiler
  endif
  endif
  
  LDFLAGS += -L $(NEXTSIMDIR)/lib -lnextsim
  
  OBJECTDIR=$(NEXTSIMDIR)/objs/
  DEPENDDIR=$(NEXTSIMDIR)/.deps/
  BINARYDIR=bin/
  
  # C++ files
  CXXSRCDIR=.
  CXXHDRDIR=.
  CXXSRC=$(wildcard $(CXXSRCDIR)/*.cpp)
  # We must exclude the version.hpp file from the list of header files because otherwise we get a circular dependency
  CXXHDR=$(filter-out $(CXXHDRDIR)/version.hpp, $(wildcard $(CXXHDRDIR)/*.hpp))
  
  OBJS=$(CXXSRC:%.cpp=$(OBJECTDIR)%.o)
  DEPS=$(CXXSRC:%.cpp=$(DEPENDDIR)%.d)
  
  # Rules to always execute.
  .PHONY: exec clean mrproper all cleanall mrproperall
  
  # Default action.
  exec: $(PROGNAME)
  
  # Create a header file with the git version
  version.hpp: version.sh $(CXXSRC) $(CXXHDR)
      $(SHELL) -x $<
  
  # Delete the object files.
  clean:
      @echo " "
      $(RM) $(OBJS) $(DEPS)
      @echo " "
  
  mrproper: clean
      $(RM) $(BINARYDIR)$(PROGNAME)
      @echo " "
  
  # Rule for making the actual target
  lines="=========="
  Lines=$(lines)$(lines)$(lines)$(lines)$(lines)$(lines)$(lines)$(lines)
  $(PROGNAME): $(OBJS) #$(CCOBJS)
      @mkdir -p $(BINARYDIR)
      @echo " "
      @echo $(Lines)$(Lines)
      @echo "Creating executable: $(BINARYDIR)$(PROGNAME)"
      @echo $(Lines)$(Lines)
      @echo " "
      $(CXX) $(CXXFLAGS) -o $(BINARYDIR)$@ $^ $(LDFLAGS)
      @echo " "
      @echo $(Lines)$(Lines)
      @echo "Created executable: $(BINARYDIR)$(PROGNAME)"
      @echo $(Lines)$(Lines)
      @echo " "
  
  # Rules for object files from cpp files
  $(OBJECTDIR)%.o: %.cpp
      @mkdir -p $(OBJECTDIR) # $(dir $@)
      @echo " "
      $(CXX) -o $@ -c $< $(CXXFLAGS)
  
  # Make dependancy rules
  $(DEPENDDIR)%.d: %.cpp
      @mkdir -p $(DEPENDDIR)
      $(CXX) -M -MG -MT $(OBJECTDIR)$*.o $(CXXFLAGS) $< > $@
  
  # The compilation depends on this Makefile.
  $(OBJS): Makefile
  
  # Make everything
  all:
      cd ..; $(MAKE) all
  
  
  # Clean everything
  cleanall:
      cd ..; $(MAKE) clean
  
  # Properly clean everything
  mrproperall:
      cd ..; $(MAKE) mrproper
  
  # Properly clean & recompile
  fresh:
      cd ..; $(MAKE) fresh
  
  -include $(DEPS)
  
• Compile Readline for x86 system (static mode)

  I am writing a program in QT Creator, for the x86 architecture, using Debian 9 x86. The program will use GNU Readline library [8.0]. Also, my program must be statically built (QT is already statically built). For this I downloaded Readline from here. Then performed the configuration for the static assembly:

  ./configure —prefix=/home/out/readlinelibs/ —enable-static —with-curses
  

  then i performed the construction:

  make
  

  and installing libraries

  make install
  

  In QT's pro file I added the path to the libraries:

  LIBS += -L/home/out/readlinelibs/lib -static -lreadline -ltinfo
  

  When compiling the program I get the following error in QT:

  error: cannot find -ltinfo
  

  How to build readline in static mode? Thanks.

  P.S.: in Debian x64 I also built readline and everything works well.
• How to fix error: no member named 'chrono' in namespace 'std'

  My code will not complie-I receive error: no member named 'chrono' in namespace 'std'.

  Many online sources have suggested that this may be because I am using an outdated version of gcc, but I have verified that I am using gcc version 6.5.0, which should include the chrono library. I cloned all of this code off of a professor for whom I'm working's git hub, so I think the problem is with my system/compiler rather than with the code itself. For reference, I am using ubuntu on a windows computer.

  Here is the error message: ../src/cpp/sgsolver_maxminmax_3player.cpp:144:24: error: no member named 'chrono' in namespace 'std' unsigned seed = std::chrono::system_clock::now().time_since_epoch().count();

  Thanks so much for your help!
• slow down of a parallel subroutine when called inside a loop

  I have parallelized a subroutine. It have very good benchmark : speedup 4X on a quad core. I have them in two different source: serial.f and paral.f. The comparison is made running them from terminal and printing elapsed wall clock time. Inside each source code there is only call to the associate subroutine. But, when I modify the sources like this :

  serial.f :
  
  do i=1,100
      call serial
  end do 
  

  and like this

   paral.f :
  
   do i=1,100
        call paral
   end do
  

  performance goes down to 0.96 X speed: the parallel version is bad than the serial one! The code can be found in why calling many N times a serial subroutine is faster than calling N times the parallel version of the same subroutin For obtaining the serial.f just comment the block containing the call paral. For obtaining the paral.f just comment the block containing the call serial. I'm asking : is this a common problem ? How can I solve it to maintain the 4 X speedup maintaning the loop call? Please note : (1)I've tried translating to C and timing, benchmarks and problems remains all the same (2) I've tried translating to modern fortran and timing, benchmarks and problems remains all the same (3) I've tried all kind of tricks and rewriting of the code. I'm sure the problem is not how the subroutine is parallelized (I achieved 4 X ) but that it is called too many times inside a loop. Thank you.

  EDIT :: As requested, I'm posting a program written in modern fortran who esibit the same issues :

  program main
  
  
  use omp_lib
  
  implicit none
  
  integer ( kind = 4 ), parameter :: m = 5000
  integer ( kind = 4 ), parameter :: n = 5000
  integer ( kind = 4 ) i
  integer ( kind = 4 ) j
  integer ( kind = 4 ) nn
  real ( kind = 8 ) u(m,n)
  real ( kind = 8 ) w(m,n)
  real ( kind = 8 ) wtime,h
  
  call random_seed()
  
  
  do j=1,n
    do i=1,m
      call random_number(u(i,j))
    end do
  end do
  
  
  wtime = omp_get_wtime ( )
  
  do nn=1,100
  
  !$omp parallel do default(none) shared(u, w) private(i,j)
  
  do j = 2, n - 1
    do i = 2, m - 1
      w(i,j) = 0.25D+00 * ( u(i-1,j) + u(i+1,j) + u(i,j-1) + u(i,j+1) )
    end do
  end do
  
  !$omp end parallel do
  
  end do
  
  wtime = omp_get_wtime ( ) - wtime
  
  
  
  h=0.0D+00
  do j=1,n
    do i=1,m
      h=h+w(i,j)
    end do
  end do
  
  
  
  write ( *, '(a,g14.6)' ) '  Wall clock time serial= ', wtime
  write ( *, '(a,g14.6)' ) '  h ', h
  
  stop
  end
  

  In order to get serial_with_loop.f90 just comment openmp directives and the nn loop. You must obtain also with a similar method parall_with_loop.f90 and serial and parall without loop. You can compile with " gfortran -o name.out -fopenmp -O3 name.f90 " and launch from terminal with output redirection to text file "name.out > time_result.txt"
• How to pass a dynamically allocated array to fortran from julia

  I am trying to call some fortran (f90) code from julia, but I can't seem to get fortran to understand dynamically allocated arrays passed from julia. This fortran code:

  MODULE myModule
      implicit none
      public Func
  CONTAINS
  
  subroutine Func(A,B)
      integer*8, intent(inout) :: A(3)
      integer*8, intent(inout) :: B(:)
      integer :: i
  
      write(*,*) "size of A"
      write(*,*) size(A)
  
      write(*,*) "A"
      do i=1,3
          write(*,*) A(i)
      enddo
  
      write(*,*) "size of B"
      write(*,*) size(B)
  
      write(*,*) "B"
      do i=1,size(B)
          write(*,*) B(i)
      enddo
  
  end subroutine Func
  end module
  

  called by this julia code

  A = [1,2,3]
  B = [4,5,6]
  
  ccall((:__mymodule_MOD_func2,"./test.so"), Nothing ,(Ref{Int64},Ref{Int64}), A,B)
  

  produces this result

   size of A
             3
   A
                      1
                      2
                      3
   size of B
             0
   B
  

  I would expect size(B) to be 3.

  Is there any way to make this work without changing the fortran code? I know that I could rewrite the fortran code to include the dimension of the array in the function call, but I would prefer to be able to call fortran code that has already been written in a similar fashion to the example I provided.
• Using maxval or minval functions

  Problems using minval and maxval functions with allocatable vectors.

  The example is only to show the error message. The component H on Refl_Type will have as minimum a dimension 3 (for h, k and l indices).

  Any easy workaround?

  Program Check implicit none

    !> Type definitions
    Type :: Refl_Type
       integer,dimension(:), allocatable :: H            ! H
       integer                           :: Mult  =0     ! Mutiplicity
    End Type Refl_Type
  
    Type, extends(Refl_Type) :: SRefl_Type
       real :: Fo    =0.0  ! Observed Structure Factor
       real :: Fc    =0.0  ! Calculated Structure Factor
    End Type SRefl_Type
  
    Type :: RefList_Type
       integer                                     :: Nref
       class(refl_Type), dimension(:), allocatable :: Reflections
    end Type RefList_Type
  
    Type(RefList_Type) :: List
  
    Contains
  
    Subroutine Sub(List)
       !---- Argument ----!
       type (RefList_Type), intent(in out) :: List
  
       !---- Local Variables ----!
       integer            :: n
       integer            :: hmax,kmax,lmax
       integer            :: hmin,kmin,lmin
  
       n=List%Nref
  
       hmax=maxval(list%reflections(1:n)%h(1))
       kmax=maxval(list%reflections(1:n)%h(2))
       lmax=maxval(list%reflections(1:n)%h(3))
  
  
     End Subroutine Sub
  
  End Program Check
  

  I have the next error compilation ...

   hmax=maxval(list%reflections(1:n)%h(1))
                    1
  
  Error: Component to the right of a part reference with nonzero rank must not have the ALLOCATABLE attribute at (1)
  
• Petsc code has no parallel speed-up on 2990WX platform

  While I run my code on a old Intel Xeon platform(X5650@2.67GHz), the parallel efficiency seems good that 80%~95% speed-up with twice processor usage. However, when I run the same code on on AMD 2990WX platform, I cannot get any acceleration with any numbers of threads.

  I am so confused that why my new AMD platform performs so bad parallel efficiency and I can hardly to know where is the wrong settings in my code.

  I have a C code based on the PetSc library to solve a very-large sparse linear equation, the parallel part in my code is provided by PetSc which automatically involves MPI ( I just arrange the matrix construction tasks to each process and do not add any other communication routines).

  The system of the computation platform are both Centos7, the version of MPI library are both MPICH3, the version of PetSc are both 3.11. The BLAS on XEON platform is included by MKL, whihe the BLAS on AMD platform is included by BLIS library.

  While the program is running on the AMD platform, I use top to check the operation of the processor, and found that the CPU usage are actually different with different run settings:

  for 32 processes:

  /usr/lib64/mpich/bin/mpiexec -n 32 ./${BIN_DIR}/main
  

  for 64 processes:

  /usr/lib64/mpich/bin/mpiexec -n 64 ./${BIN_DIR}/main
  

  on XEON platform:

  /public/software/Petsc/bin/petscmpiexec -n 64 -f mac8 ./${BIN_DIR}/main
  

  with mac8 file:

  ic1:8
  ic2:8
  ic3:8
  ic4:8
  ic5:8
  ic6:8
  ic7:8
  ic8:8
  
• VSCode Itellisense with python C extension module (petsc4py)

  I'm currently using a python module called petsc4py (https://pypi.org/project/petsc4py/). My main issue is that none of the typical intellisense features seems to work with this module.

  I'm guessing it might have something to do with it being a C extension module, but I am not sure exactly why this happens. I initially thought that intellisense was unable to look inside ".so" files, but it seems that numpy is able to do this with the array object, which in my case is inside a file called multiarray.cpython-37m-x86_64-linux-gnu (check example below).

  Does anyone know why I see this behaviour in the petsc4py module. Is there anything that I (or the developers of petsc4py) can do to get intellisense to work?

  Example:

  import sys
  import petsc4py
  petsc4py.init(sys.argv)
  from petsc4py import PETSc
  
  x_p = PETSc.Vec().create()
  x_p.setSizes(10)
  x_p.setFromOptions()
  
  u_p = x_p.duplicate()
  
  
  import numpy as np
  
  x_n = np.array([1,2,3])
  u_n = x_n.copy()
  

  In this example, when trying to work with a Vec object from petsc4py, doing u_p.duplicate() cannot find the function and the suggestion is simply a repetition of the function immediately before. However, using an array from numpy, doing u_n.copy() works perfectly.

  
• gcc make run error No rule to make target

  I just downloaded petsc on a cluster that runs Linux. I configured it with:

      $ ./configure
      $ make all test
  

  After that I set the PETSC_DIR to the directory where petsc is. I tried to run a example program by first using "make ex5" which is successful but when I try "make runex5" like it says in a petsc introduction video, I get

      make: *** No rule to make target 'runex5' . Stop
  

  Although there is a file named ex5 that is created after the first make call.

  From what I understood this error is given if the file does not exist, but it is there, so I don't know what the problem might be. The only thing I suspect is that the 'ex5' that is created after the first make call is of type file and not .exe .