[DFTB-Plus-User] DFTB-Plus-User Digest, Vol 58, Issue 16
David Furman
sirok4 at gmail.com
Sun Jun 23 11:32:36 CEST 2019
Hi Ben,
Update: after updating to openmpi-4.0.1, recompiling dftb+ and detaching
binding of processes using --bind-to none
I could successfully run 4 instances with 100% load - as should be.
Best,
David
On Sat, Jun 22, 2019 at 8:26 PM <
dftb-plus-user-request at mailman.zfn.uni-bremen.de> wrote:
> Send DFTB-Plus-User mailing list submissions to
> dftb-plus-user at mailman.zfn.uni-bremen.de
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
>
> or, via email, send a message with subject or body 'help' to
> dftb-plus-user-request at mailman.zfn.uni-bremen.de
>
> You can reach the person managing the list at
> dftb-plus-user-owner at mailman.zfn.uni-bremen.de
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of DFTB-Plus-User digest..."
>
>
> Today's Topics:
>
> 1. Re: DFTB-Plus-User Digest, Vol 58, Issue 11 (Ben Hourahine)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 22 Jun 2019 20:25:50 +0100
> From: Ben Hourahine <benjamin.hourahine at strath.ac.uk>
> To: dftb-plus-user at mailman.zfn.uni-bremen.de
> Subject: Re: [DFTB-Plus-User] DFTB-Plus-User Digest, Vol 58, Issue 11
> Message-ID: <c57cb998-3aa3-5cde-0ece-72cde8978807 at strath.ac.uk>
> Content-Type: text/plain; charset="utf-8"
>
> Hello David,
>
> If possible use an optimized blas instead of the reference one, as this
> makes a big difference in performance.
>
> You might be better off using a serial compiled binary and 2 threads on
> each of the 4 images (if with a thread aware blas/lapack).
>
> Yes, sorry, had picked up 8 cores and mistakenly thought that was the
> number of images (is 4 enough for REMD?).
>
> Regards
>
> Ben
>
> On 22/06/2019 17:34, David Furman wrote:
> > Hi Ben,
> >
> > 1. I use libblas-dev 3.6.0 (This implementation is the Fortran 77
> > reference implementation found at netlib)
> > 2. I run each instance from the same shell, 'echo $OMP_NUM_THREADS'
> > yields: 1
> > 3. Yes it is the correct context. I try to run a Replica-Exchange MD
> > using 4 replicas. As far as I know, i-pi distributes the replicas to
> > clients (DFTB+ in this case) so that I can run 4 concurrent replicas.
> > Since I have a relatively large system in terms of number of atoms,
> > each replica takes a significant amount of time. Hence I was thinking
> > of parallelizing each replica on 2 or more MPI cores (i.e. parallelize
> > each DFTB+ instance). But, then I encounter this weird "bottleneck".
> > Regarding your suggestion: I'm not sure how running 8 serial instances
> > is of help in my case.
> >
> > I appreciate your help!
> >
> >
> >
> > On Sat, Jun 22, 2019 at 2:54 PM
> > <dftb-plus-user-request at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user-request at mailman.zfn.uni-bremen.de>> wrote:
> >
> > Send DFTB-Plus-User mailing list submissions to
> > dftb-plus-user at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user at mailman.zfn.uni-bremen.de>
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >
> >
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
> >
> > or, via email, send a message with subject or body 'help' to
> > dftb-plus-user-request at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user-request at mailman.zfn.uni-bremen.de>
> >
> > You can reach the person managing the list at
> > dftb-plus-user-owner at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user-owner at mailman.zfn.uni-bremen.de>
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of DFTB-Plus-User digest..."
> >
> >
> > Today's Topics:
> >
> > 1. Possible bug in DFTB+ with MPI (David Furman)
> > 2. Re: problem with installation (Farzad Molani)
> > 3. Re: Possible bug in DFTB+ with MPI (Ben Hourahine)
> >
> >
> >
> ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Sat, 22 Jun 2019 14:41:01 +0100
> > From: David Furman <sirok4 at gmail.com <mailto:sirok4 at gmail.com>>
> > To: dftb-plus-user at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user at mailman.zfn.uni-bremen.de>
> > Subject: [DFTB-Plus-User] Possible bug in DFTB+ with MPI
> > Message-ID:
> >
> > <CADtBuo94A729tJFsTW5ev69sEs4qVNM=qOXkjMKFTS8_f1N8pg at mail.gmail.com
> > <mailto:qOXkjMKFTS8_f1N8pg at mail.gmail.com>>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hi all,
> > I have a question concerning the running of DFTB+ with MPI. There
> > seems to
> > be a problem when running more than one DFTB+ instance (with MPI
> > parallelization) on the same system.
> >
> > When I run 4 instances (separate folders) of dftb+ with 2 cores
> > each (i.e.
> > 8 cores in total):
> > mpirun.openmpi -np 2 dftb+, each %CPU usage drops to 50%.
> >
> > This is the 'top' output:
> > ========================================================
> > 35426 user 20 0 423384 40404 18248 R 50.5 0.0 0:37.69
> > dftb+
> > 35434 user 20 0 423080 39720 18248 R 50.2 0.0 0:05.70
> > dftb+
> > 35396 user 20 0 422044 38660 17644 R 49.8 0.0 1:21.35
> > dftb+
> > 35402 user 20 0 423384 40460 18304 R 49.8 0.0 1:09.54
> > dftb+
> > 35403 user 20 0 422044 38872 17848 R 49.8 0.0 1:16.15
> > dftb+
> > 35427 user 20 0 422044 39048 18032 R 49.8 0.0 0:32.93
> > dftb+
> > 35435 user 20 0 421732 38164 17716 R 49.8 0.0 0:09.07
> > dftb+
> > 35395 user 20 0 423384 40360 18204 R 49.5 0.0 1:13.18
> > dftb+
> > =========================================================
> >
> > Whereas, when I run one instance with 8 cores, the efficiency is
> > 100% as
> > expected:
> >
> > =========================================================
> > 35837 user 20 0 441868 35196 19112 R 100.3 0.0 0:04.90
> > dftb+
> > 35828 user 20 0 442580 35676 19120 R 100.0 0.0 0:04.83
> > dftb+
> > 35829 user 20 0 442544 36024 19432 R 100.0 0.0 0:04.91
> > dftb+
> > 35830 user 20 0 441912 35356 19360 R 100.0 0.0 0:04.91
> > dftb+
> > 35831 user 20 0 441688 35048 19304 R 100.0 0.0 0:04.90
> > dftb+
> > 35833 user 20 0 441904 35656 19500 R 100.0 0.0 0:04.91
> > dftb+
> > 35841 user 20 0 441516 34824 19076 R 100.0 0.0 0:04.89
> > dftb+
> > 35843 user 20 0 441224 34456 18868 R 99.7 0.0 0:04.89
> > dftb+
> > =========================================================
> >
> > I run both cases with OMP_NUM_THREADS=1.
> > with open-mpi 1.10.2 and gcc 5.4.0.
> >
> > Could anyone give a hint about what is wrong?
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> > <
> http://mailman.zfn.uni-bremen.de/pipermail/dftb-plus-user/attachments/20190622/96b0d502/attachment-0001.html
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Sat, 22 Jun 2019 21:42:46 +0800
> > From: Farzad Molani <f.molani at gmail.com <mailto:f.molani at gmail.com>>
> > To: "User list for DFTB+ related questions"
> > <dftb-plus-user at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user at mailman.zfn.uni-bremen.de>>
> > Subject: Re: [DFTB-Plus-User] problem with installation
> > Message-ID:
> >
> > <CAJY0Q+obk-e8iUOJ_986DS4tWpWG4kAZ99a=6qNstswS0myifg at mail.gmail.com
> > <mailto:6qNstswS0myifg at mail.gmail.com>>
> > Content-Type: text/plain; charset="utf-8"
> >
> > I know, but I don't know how can I unstall arpack?
> > The package downloaded and now I need help to install it.
> > Please note that, I don't permit to login as a root.
> >
> > On Saturday, June 22, 2019, RICCARDO ROZZA
> > <rozza.riccardo at studium.unict.it
> > <mailto:rozza.riccardo at studium.unict.it>>
> > wrote:
> >
> > > Dear Farzad,
> > >
> > > It seems that you need to install the arpack library (ld: cannot
> > find
> > > -larpack).
> > > All the best,
> > >
> > > Riccardo.
> > >
> > > On 22 Jun 2019, at 10:01, Farzad Molani <f.molani at gmail.com
> > <mailto:f.molani at gmail.com>> wrote:
> > >
> > >
> > > Dear DFTB users,
> > > I want to install the newest version of DFTB+. The source and
> > executable
> > > files from http://www.dftbplus.org/download/dftb-stable/ have
> been
> > > downloaded and untarred. I try to install the software based on
> > > instructions on INSTALL.rst, but I got the following error after
> > the make.
> > > .
> > > .
> > > .
> > > .
> > > w.o dispuffdata.o inputdata.o pmlocalisation.o repcont.o
> repspline.o
> > > reppoly.o xlbomd.o extlagrangian.o linresp.o linrespgrad.o qm.o
> > > linrespcommon.o arpack.o eigensolver.o typegeometry.o simplemixer.o
> > > numderivs2.o potentials.o thirdorder.o charges.o dftbplusu.o
> > geoopt.o
> > > steepdesc.o gdiis.o orbitalequiv.o broydenmixer.o stress.o etemp.o
> > > hermite.o factorial.o intrinsicpr.o mixer.o andersonmixer.o
> > > elecconstraints.o angmomentum.o eigenvects.o parser.o hsdutils2.o
> > > unitconversion.o oldcompat.o typegeometryhsd.o oldskdata.o
> > densitymatrix.o
> > > populations.o repulsive.o spinorbit.o forces.o
> > -L/home/fmolani/DFTB+/
> > > dftbplus-18.2/_build/external/xmlf90 -lxmlf90
> > -L/home/fmolani/DFTB+/
> > > dftbplus-18.2/_build/external/fsockets -lfsockets
> > > -L/apps/intel/2015/composer_xe_2015.3.187/mkl/lib/intel64
> > > -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -larpack -liomp5
> > > ld: cannot find -larpack
> > > make[1]: *** [dftb+] Error 1
> > > make[1]: Leaving directory `/home/fmolani/DFTB+/dftbplus-
> > > 18.2/_build/prog/dftb+'
> > > make: *** [dftb+] Error 2
> > >
> > > I would appreciate to help me to solve the problem.
> > > Best regards, Farzad
> > > --
> > > Farzad Molani
> > > Postdoctoral Research Fellow
> > > School of Pharmaceutical Science and Technology
> > > Tianjin University
> > > Building 24, Room A205
> > > 92 Weijin Road, Nankai District
> > >
> > <
> https://www.google.com/maps/search/92+Weijin+Road,+Nankai+District+%0D%0A+P.+R.+China?entry=gmail&source=g
> >
> > > P. R. China
> > >
> > <
> https://www.google.com/maps/search/92+Weijin+Road,+Nankai+District+%0D%0A+P.+R.+China?entry=gmail&source=g
> >
> > > Tel.: +8617526526420
> > > Email: f.molani at gmail.com <mailto:f.molani at gmail.com> and
> > f.molani at tju.edu.cn <mailto:f.molani at tju.edu.cn>
> > > <fmolani at dena.kntu.ac.ir <mailto:fmolani at dena.kntu.ac.ir>>
> > >
> > >
> > > --
> > > Farzad Molani
> > > Postdoctoral Research Fellow
> > > School of Pharmaceutical Science and Technology
> > > Tianjin University
> > > Building 24, Room A205
> > > 92 Weijin Road, Nankai District
> > >
> > <
> https://www.google.com/maps/search/92+Weijin+Road,+Nankai+District+%0D%0A+P.+R.+China?entry=gmail&source=g
> >
> > > P. R. China
> > >
> > <
> https://www.google.com/maps/search/92+Weijin+Road,+Nankai+District+%0D%0A+P.+R.+China?entry=gmail&source=g
> >
> > > Tel.: +8617526526420
> > > Email: f.molani at gmail.com <mailto:f.molani at gmail.com> and
> > f.molani at tju.edu.cn <mailto:f.molani at tju.edu.cn>
> > > <fmolani at dena.kntu.ac.ir <mailto:fmolani at dena.kntu.ac.ir>>
> > > _______________________________________________
> > > DFTB-Plus-User mailing list
> > > DFTB-Plus-User at mailman.zfn.uni-bremen.de
> > <mailto:DFTB-Plus-User at mailman.zfn.uni-bremen.de>
> > >
> >
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
> > >
> > >
> > >
> >
> > --
> > Farzad Molani
> > Postdoctoral Research Fellow
> > School of Pharmaceutical Science and Technology
> > Tianjin University
> > Building 24, Room A205
> > 92 Weijin Road, Nankai District
> > P. R. China
> > Tel.: +8617526526420
> > Email: f.molani at gmail.com <mailto:f.molani at gmail.com> and
> > f.molani at tju.edu.cn <mailto:f.molani at tju.edu.cn>
> > <fmolani at dena.kntu.ac.ir <mailto:fmolani at dena.kntu.ac.ir>>
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> > <
> http://mailman.zfn.uni-bremen.de/pipermail/dftb-plus-user/attachments/20190622/aae9eaba/attachment-0001.html
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Sat, 22 Jun 2019 14:54:40 +0100
> > From: Ben Hourahine <benjamin.hourahine at strath.ac.uk
> > <mailto:benjamin.hourahine at strath.ac.uk>>
> > To: dftb-plus-user at mailman.zfn.uni-bremen.de
> > <mailto:dftb-plus-user at mailman.zfn.uni-bremen.de>
> > Subject: Re: [DFTB-Plus-User] Possible bug in DFTB+ with MPI
> > Message-ID: <16da7e6a-92dd-2184-7a98-067e751cdb14 at strath.ac.uk
> > <mailto:16da7e6a-92dd-2184-7a98-067e751cdb14 at strath.ac.uk>>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hello David,
> >
> > thoughts so far:
> >
> > 1) Which blas are you using? Sometimes openBlas steals extra
> > threads, is
> > your top output shown with thread reporting enabled (H)?
> >
> > 2) Are you definitely getting the OMP_NUM_THREADS shell variable
> > to each
> > of the instances?
> >
> > 3) If this is related to the i-PI mailing list question, is there any
> > reason not to run 8 serial instances with 1 thread each? The MPI
> > parallelism overhead on just one process is probably not worth it.
> >
> > Regards
> >
> > Ben
> >
> > On 22/06/2019 14:41, David Furman wrote:
> > > Hi all,
> > > I have a question concerning the running of DFTB+ with MPI. There
> > > seems to be a problem when running more than one DFTB+ instance
> > (with
> > > MPI parallelization) on the same system.
> > >
> > > When I run 4 instances (separate folders) of dftb+ with 2 cores
> each
> > > (i.e. 8 cores in total):
> > > mpirun.openmpi -np 2 dftb+, each %CPU usage drops to 50%.
> > >
> > > This is the 'top' output:
> > > ========================================================
> > > 35426 user 20 0 423384 40404 18248 R 50.5 0.0
> > 0:37.69 dftb+
> > > 35434 user 20 0 423080 39720 18248 R 50.2 0.0
> > 0:05.70 dftb+
> > > 35396 user 20 0 422044 38660 17644 R 49.8 0.0
> > 1:21.35 dftb+
> > > 35402 user 20 0 423384 40460 18304 R 49.8 0.0
> > 1:09.54 dftb+
> > > 35403 user 20 0 422044 38872 17848 R 49.8 0.0
> > 1:16.15 dftb+
> > > 35427 user 20 0 422044 39048 18032 R 49.8 0.0
> > 0:32.93 dftb+
> > > 35435 user 20 0 421732 38164 17716 R 49.8 0.0
> > 0:09.07 dftb+
> > > 35395 user 20 0 423384 40360 18204 R 49.5 0.0
> > 1:13.18 dftb+
> > > =========================================================
> > >
> > > Whereas, when I run one instance with 8 cores, the efficiency is
> > 100%
> > > as expected:
> > >
> > > =========================================================
> > > 35837 user 20 0 441868 35196 19112 R 100.3 0.0
> > 0:04.90 dftb+
> > > 35828 user 20 0 442580 35676 19120 R 100.0 0.0
> > 0:04.83 dftb+
> > > 35829 user 20 0 442544 36024 19432 R 100.0 0.0
> > 0:04.91 dftb+
> > > 35830 user 20 0 441912 35356 19360 R 100.0 0.0
> > 0:04.91 dftb+
> > > 35831 user 20 0 441688 35048 19304 R 100.0 0.0
> > 0:04.90 dftb+
> > > 35833 user 20 0 441904 35656 19500 R 100.0 0.0
> > 0:04.91 dftb+
> > > 35841 user 20 0 441516 34824 19076 R 100.0 0.0
> > 0:04.89 dftb+
> > > 35843 user 20 0 441224 34456 18868 R 99.7 0.0
> > 0:04.89 dftb+
> > > =========================================================
> > >
> > > I run both cases with OMP_NUM_THREADS=1.
> > > with open-mpi 1.10.2 and gcc 5.4.0.
> > >
> > > Could anyone give a hint about what is wrong?
> > >
> > >
> > > _______________________________________________
> > > DFTB-Plus-User mailing list
> > > DFTB-Plus-User at mailman.zfn.uni-bremen.de
> > <mailto:DFTB-Plus-User at mailman.zfn.uni-bremen.de>
> > >
> >
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
> >
> > --
> > Dr. B. Hourahine, Senior lecturer
> > SUPA, Department of Physics,
> > University of Strathclyde,
> > John Anderson Building,
> > 107 Rottenrow, Glasgow G4 0NG, UK.
> > +44 141 548 2325, benjamin.hourahine at strath.ac.uk
> > <mailto:benjamin.hourahine at strath.ac.uk>
> >
> > The Department is a partner in SUPA, the Scottish
> > Universities Physics Alliance
> >
> > The University of Strathclyde is a charitable body,
> > registered in Scotland, number SC015263
> >
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> > <
> http://mailman.zfn.uni-bremen.de/pipermail/dftb-plus-user/attachments/20190622/bf90b2fb/attachment.html
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > DFTB-Plus-User mailing list
> > DFTB-Plus-User at mailman.zfn.uni-bremen.de
> > <mailto:DFTB-Plus-User at mailman.zfn.uni-bremen.de>
> >
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
> >
> > ------------------------------
> >
> > End of DFTB-Plus-User Digest, Vol 58, Issue 11
> > **********************************************
> >
> >
> >
> > --
> > David Furman, PhD, MRSC**Herchel Smith Research Fellow Department of
> > Chemistry University of Cambridge Lensfield Road CB2 1EW
> >
> > _______________________________________________
> > DFTB-Plus-User mailing list
> > DFTB-Plus-User at mailman.zfn.uni-bremen.de
> >
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
>
> --
> Dr. B. Hourahine, Senior lecturer
> SUPA, Department of Physics,
> University of Strathclyde,
> John Anderson Building,
> 107 Rottenrow, Glasgow G4 0NG, UK.
> +44 141 548 2325, benjamin.hourahine at strath.ac.uk
>
> The Department is a partner in SUPA, the Scottish
> Universities Physics Alliance
>
> The University of Strathclyde is a charitable body,
> registered in Scotland, number SC015263
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mailman.zfn.uni-bremen.de/pipermail/dftb-plus-user/attachments/20190622/6c950d2c/attachment.html
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> DFTB-Plus-User mailing list
> DFTB-Plus-User at mailman.zfn.uni-bremen.de
> https://mailman.zfn.uni-bremen.de/cgi-bin/mailman/listinfo/dftb-plus-user
>
> ------------------------------
>
> End of DFTB-Plus-User Digest, Vol 58, Issue 16
> **********************************************
>
--
David Furman, PhD, MRSC
Herchel Smith Research Fellow
Department of Chemistry
University of Cambridge
Lensfield Road
CB2 1EW
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.zfn.uni-bremen.de/pipermail/dftb-plus-user/attachments/20190623/c96ca01e/attachment-0001.html>
More information about the DFTB-Plus-User
mailing list