exciting-developers Mailing List for EXCITING

Dear All,

I'd like to draw your attention to the following changes in the code:

1. IMPORTANT: The first-variational spinor index and the k-point index  
have been swapped in the following arrays: ngk, igkig, vgkl, vgkc,  
gkc, tpgkc, sfacgk. This was done to make coding of the density matrix  
routines easier.

2. An additional argument (tfbz) has been added to the routine  
genppts. This allows for the k-points to be mapped to the first  
Brillouin zone, which was required for the electron-phonon coupling  
matrix elements.

3. The intrinsic cpu_time, which was used for the calculating the run  
times, has been replaced with the system_clock intrinsic. Hopefully  
this will be more reliable for parallel runs.

Regards,
Kay.

Dear EXCITING developers,
after fixing the date of the meeting on July 21+22,
we want to provide you with further information.

1) REGISTRATION: We ask all participants to register
(just write a short email to us to confirm your participation)

2) PROGRAM/CALL FOR CONTRIBUTIONS:
Attached you find a sketch for the program.
We ask every participant to give a short (15 min) informal
introduction about his/her involvement in the exciting project.
Please give us a feedback concerning the contents, especially
we ask you for
- agenda items that you wish to be discussed
- contributions/talks for the Tuesday mornings session;
  please send us a title and short description if you want
  to give a talk


3) ACCOMMODATION AND TRAVELING:
Informations about accommodations and the location of our institute
are found under http://tinyurl.com/3ggdln
Just click on the items in the map to get detailed information.
Please tell us if we should book a room for you!
The closest airports are in Graz (about 75 min. from Leoben by
bus/train) and
Vienna (3h from Leoben by bus/train).
Please don't hesitate to ask us if you need any further information!
If you are looking for someone to share a double room with, you can also
tell us -
we will try to arrange it.

Unfortunately we don't have a budget for the EXCITING project.
Thus we cannot cover the travel and accommodation costs.
We are trying to find financial support for the accommodation,
but at the moment we can not promise anything.

Looking forward to welcoming you in Leoben,

Christian and Juergen

Mag. Christian Meisenbichler
Chair of Atomistic Modelling and Design of Materials, Leoben University
Franz-Josef-Straße 18    
8700 Leoben
Tel.: ++43 (0)3842 402 4411
E-Mail: chr...@un...

Dr. Jürgen Spitaler
Material Center Leoben and
Chair of Atomistic Modelling and Design of Materials, Leoben University
Franz-Josef-Straße 18    
8700 Leoben
Tel.: ++43 (0)3842 402 4404
E-Mail: jue...@un...

Dear EXCITING developers,
thank you for adding your dates at
http://www.doodle.ch/participation.html?pollId=ziu7hewtdqem266g

Fortunately, we could find a date where everybody is available:
So wethe date of the meeting will be
>> Mo. 21. July - Tu. 22. July 2008 <<
The meeting will take place in Leoben, Austria.
We will give you more details later on.

Looking forward to meeting you in July,
Juergen Spitaler

-- 
Dr. Jürgen Spitaler
Material Center Leoben and
Chair of Atomistic Modelling and Design of Materials, Leoben University
Franz-Josef-Straße 18    
8700 Leoben
Tel.: ++43 (0)3842 402 4404
E-Mail: jue...@un...

Dear EXC!TiNG developers,
8 people have already visited the doodle page for the developers meeting
and checked their available dates.
We want to fix the date this weekend.
Thus, if you want to participate, please tell us your availabilty
until Saturday checking the doodle page:
http://www.doodle.ch/participation.html?pollId=ziu7hewtdqem266g

We are looking forward to your response
best wishes
Juergen

-- 
Dr. Jürgen Spitaler
Material Center Leoben and
Chair of Atomistic Modelling and Design of Materials, Leoben University
Franz-Josef-Straße 18    
8700 Leoben
Tel.: ++43 (0)3842 402 4404
E-Mail: jue...@mu...

Dear EXC!TiNG developers,

the EXC!TiNG code has become a nice tool already, which is growing fast. 
In this context, it was already discussed some time ago a to organize a 
small meeting of people involved in the development. To persue this idea, 
we would like to initiate an effort in this direction, and to host such a 
meeting at the University of Leoben (http://www.unileoben.ac.at/amadm). 

At this occasion we could get a survey of present implementations and 
discuss future developments. We could also identify common goals and agree 
on how to proceed. And it would certainly be nice to learn to know each 
other. 

>From our point of view some important issues to discuss are bug tracking, 
source code management, road map, releases, etc. We are sure, you will 
come up with many more suggestions. 

To facilitate the organization, we have installed a doodle page at 
http://www.doodle.ch/participation.html?pollId=ziu7hewtdqem266g, where we 
propose possible dates for a meeting (either in July or August this year). 
If you wish to participate, please check the corresponding boxes to 
indicate your availability,  which will allow us to fix a date in an easy 
way. From this we will be able to find out  the 2 days, which are most 
suitable for most of us.
 
Looking forward to your response,
Claudia Ambrosch-Draxl
Jürgen Spitaler
Christian Meisenbichler
Stephan Sagmeister
------------------------------------------------------------------------
Claudia Ambrosch-Draxl
Chair of Atomistic Modelling and Design of Materials
Department Materials Physics, University of Leoben
Franz-Josef-Straße 18, A-8700 Leoben
http://www.mu-leoben.at/amadm
+43 3842 402 4400 Phone
+43 3842 402 4402 Fax

http://www.macresearch.org/structured-programming-fortran

I did some more testing of the ScaLAPACK Exciting on my problem and one of
the conclusions I am coming to is that the problem is too small to be scaled
well, at least with this algorithm. Well, the problem is _lengthy_ in a
sense that I have a lot of calculation to do in series, i.e. many SCL
iterations each of which over a number of k-points, but again this all is
done sequentially. Now, since I parallelize each k-point, which is a
relatively small block of this calculation, the order of the problem becomes
too small to be parallelized efficiently. Increasing the order of matrices
(by e.g. rgkmax) does lead to a somewhat better scaling behavior. When the
order of the problem is instead reduced, I get very poor scaling with
basically the same calculation time on one or several machines with most of
the time spent for interprocess communication. The network is Gigabit
Ethernet with high-end switches (we work with huge amounts of graphical
data). Indeed this is more like a supercomputer task! And you are right, the
only choice would probably be a more coarse-grain parallelism, e.g. over
k-points, which worked nicely for me, but is just not so, well... scalable
:-)
 
andrei   
 
 


  _____  

From: ch...@gm... [mailto:ch...@gm...] On Behalf Of Christian
Meisenbichler
Sent: Wednesday, February 27, 2008 10:31 AM
To: Andrei Los
Cc: exc...@li...
Subject: Re: [Exciting-developers] scalapack version remark


I doubt that this scenario can be tackled with scalapack exciting, I'm
afraid. There is a reason why people inwest the same money for the
interconnect as for the computers. There are only few PBLAS operations that
work on GigE with good performance. PGEMV for example. We tried to use
Scalapack with the BSE code. The direct  eigen solver was slower with
multiple processors than with one. The time evolution version using mainly
PGEMV scaled good or at least good enough even on Gigabit Ethernet.
Additionally normal offices would not deploy high end switches for the
networking which probably leads to an additional degradation of the
communication speed when the number of concurrent communications increase. 

As you may not be able to benefit from fine grain parallelism apart from
within one node. I really think you may be well off by trying to use a
hybrid of MPI  k point parallelism and smp parallelism within a node. (The
smp arpack version can be compiled with mpi k-point parallelism)

I don't think that scalapack replaces a supercomuter it more or less
requires one!



On Tue, Feb 26, 2008 at 7:09 PM, Andrei Los <and...@is...> wrote:


Christian,
 
I also think that parallel setup can be done better than I did. I did not
really have much time to spend on this. 
 
Concerning the motivation, the problem where I desperately need more
performance and to have it finely scalable is just a "usual" problem of a
semiconductor lattice with an impurity, which I need to relax. Since I need
to keep the impurity concentration reasonably low, the number of atoms and
the calculation time gets high. It's not really the memory requirement which
is critical for me, the program uses below 1GB for this calculation, but the
flops. I don't have an access to any supercomputer facility, and my cluster
is just several core duo machines. Currently I am using a modification of
Exciting which replaces OpenMP parallelism of the k-point loop by MPI
parallelism. The problem with this is that with a small number of k-points
(12 in my case) I can only use 2, 3, 4, 6, and 12 processes to have an even
CPU loading. Fine-grain parallelism would be nice, so I could use pretty
much any number of machines and they won't have to be of the same
performance. 
 
I am sure there are other problems with similar requirements.    
 
Andrei



  _____  

From: ch...@gm... [mailto:ch...@gm...] On Behalf Of Christian
Meisenbichler
Sent: Tuesday, February 26, 2008 10:25 AM
To: Andrei Los
Cc: exc...@li...
Subject: Re: [Exciting-developers] scalapack version remark


Dear Andrej

Concerning matrix setup. I'm very convinced that doing the whole setup of
the system in the parallel data structures can be done fairly elegant. The
routine hmlaa and olpaa update the matrices through a zher2 operation.
Meaning that it requires only a  order n vector  comunication to update the
matrix. and the other setup functions calculate single matrix element
contribution that may well be scoped to the distributed part. 

But really before doing all this we really have to find out if anybody would
use such a distributed memory code. Why did you start this project what was
the motivation?

Possible scenarios where a distributed memory exciting could pay off where:



*	You had free time on a blue gene 

*	or need to calculate huge clusters or surfaces that required more
memory than affordable in a single node. But I'm afraid that calculations
like that take incredibly long. 

Anyone has such a use case?



On Mon, Feb 25, 2008 at 5:41 PM, Andrei Los <and...@is...> wrote:


I have a couple more comments on the ScaLAPACK version. Indeed, it does not
seem to be very efficient judging from CPU utilization, but I did not do a
lot of testing with regard to e.g. blocking factor (only the "standard"
values of 64 and 128 were used). Intel claims though that MKL ScaLAPACK
implementation (the one I used) is not very sensitive to the blocking
factor. Other things such as network communication topology may influence
the efficiency as well.
 
In this version matrix setup is "semi-parallel". Full-size hamiltonian and
overlap matrices in packed storage form are allocated on the master node
only, in the other processes only a fraction of this, inversely proportional
to the number of processes, is needed. The pieces are then gathered into the
packed array on the master node, further calculations are applied to them,
and they are distributed in block-cyclic fashion, as required by ScaLAPACK,
with the local array dimensions again inversely proportional to the number
of processes. Memory requirements at the master process can be cut by doing
hamiltonian and overlap matrix setup serially. This will make the code
clunkier (not terribly), but will allow one to use only one full-size
packed array. This will only make sense, however, same as the whole idea of
using ScaLAPACK, only if the number of nodes is not too small. Otherwise
regular matrix storage needed by ScaLAPACK (as opposed to packed storage)
will require a lot of memory. 
 
Another thing regarding this semi-parallel matrix setup is that only the
calls to the most time-consuming routines, i.e. hmlaa and olpaa are
parallelized. This, on one hand reduces the number of array variables which
need to be stored on all nodes, and on the other hand reduces the amount of
inter-process communication. The other routines in the matrix setup which
only use elementary operations may actually suffer from parallelization as
the cost of inter-process communication or of variable storage will be
larger that the parallelization gain. 
 
Andrei
 


  _____  

From: exc...@li...
[mailto:exc...@li...] On Behalf Of
Christian Meisenbichler
Sent: Monday, February 25, 2008 3:36 PM
To: exc...@li...
Subject: [Exciting-developers] scalapack version remark


First I have to say that I'm happy about Adrei Los' Scalapack efforts. I
would have had to work trough it eventually one day.

We learned that it is not particularly scalable but I think this could be
improved by using PARPACK [1] as we have evidence that ARPACK is efficient
at least on smp machines [2].


*	Key to a Scalapack version is to get parallel matrix setup as well.
Because setup takes the same magnitude of time as the solver. 

*	It would be important to setup the matrix in place in the
distributed data structure in order to avoid a memory bottleneck in the
matrix setup (which would render the scalapack version essentially useless).
This is as far as I can tell, not the case in the current version. The
matrix is setup in packed storage form in the master node and then
distributed among the other nodes. 

I think we could reach a distributed memory version with reasonable scaling
soon. If there are indeed problems that can take advantage of such code. 
Does anybody with access to a fast interconnect cluster have a project for
that? 

[1] http://www.caam.rice.edu/~kristyn/parpack_home.html
<http://www.caam.rice.edu/%7Ekristyn/parpack_home.html> 
[2] http://exciting.wiki.sourceforge.net/IterativeSolver

Hi i fixed a bug I overlooked in testing. with commit 40156a649296
http://tinyurl.com/24ua4l

The matrix setup was wrong which could only be recognized in systems without
shift invert symmetry. thanks to Kay who pointed me there.

I doubt that this scenario can be tackled with scalapack exciting, I'm
afraid. There is a reason why people inwest the same money for the
interconnect as for the computers. There are only few PBLAS operations that
work on GigE with good performance. PGEMV for example. We tried to use
Scalapack with the BSE code. The direct  eigen solver was slower with
multiple processors than with one. The time evolution version using mainly
PGEMV scaled good or at least good enough even on Gigabit Ethernet.
Additionally normal offices would not deploy high end switches for the
networking which probably leads to an additional degradation of the
communication speed when the number of concurrent communications increase.

As you may not be able to benefit from fine grain parallelism apart from
within one node. I really think you may be well off by trying to use a
hybrid of MPI  k point parallelism and smp parallelism within a node. (The
smp arpack version can be compiled with mpi k-point parallelism)

I don't think that scalapack replaces a supercomuter it more or less
requires one!


On Tue, Feb 26, 2008 at 7:09 PM, Andrei Los <and...@is...> wrote:

>  Christian,
>
> I also think that parallel setup can be done better than I did. I did not
> really have much time to spend on this.
>
> Concerning the motivation, the problem where I desperately need
> more performance and to have it finely scalable is just a "usual" problem of
> a semiconductor lattice with an impurity, which I need to relax. Since I
> need to keep the impurity concentration reasonably low, the number of atoms
> and the calculation time gets high. It's not really the memory requirement
> which is critical for me, the program uses below 1GB for this calculation,
> but the flops. I don't have an access to any supercomputer facility, and my
> cluster is just several core duo machines. Currently I am using a
> modification of Exciting which replaces OpenMP parallelism of the k-point
> loop by MPI parallelism. The problem with this is that with a small number
> of k-points (12 in my case) I can only use 2, 3, 4, 6, and 12 processes to
> have an even CPU loading. Fine-grain parallelism would be nice, so I could
> use pretty much any number of machines and they won't have to be of the same
> performance.
>
> I am sure there are other problems with similar requirements.
>
> Andrei
>
>  ------------------------------
> *From:* ch...@gm... [mailto:ch...@gm...] *On Behalf Of *Christian
> Meisenbichler
> *Sent:* Tuesday, February 26, 2008 10:25 AM
> *To:* Andrei Los
> *Cc:* exc...@li...
> *Subject:* Re: [Exciting-developers] scalapack version remark
>
> Dear Andrej
>
> Concerning matrix setup. I'm very convinced that doing the whole setup of
> the system in the parallel data structures can be done fairly elegant. The
> routine hmlaa and olpaa update the matrices through a zher2 operation.
> Meaning that it requires only a  order n vector  comunication to update the
> matrix. and the other setup functions calculate single matrix element
> contribution that may well be scoped to the distributed part.
>
> But really before doing all this we really have to find out if anybody
> would use such a distributed memory code. Why did you start this project
> what was the motivation?
>
> Possible scenarios where a distributed memory exciting could pay off
> where:
>
>    - You had free time on a blue gene
>    - or need to calculate huge clusters or surfaces that required more
>    memory than affordable in a single node. But I'm afraid that calculations
>    like that take incredibly long.
>
> Anyone has such a use case?
>
>
> On Mon, Feb 25, 2008 at 5:41 PM, Andrei Los <and...@is...> wrote:
>
> >  I have a couple more comments on the ScaLAPACK version. Indeed, it does
> > not seem to be very efficient judging from CPU utilization, but I did not do
> > a lot of testing with regard to e.g. blocking factor (only the
> > "standard" values of 64 and 128 were used). Intel claims though that MKL
> > ScaLAPACK implementation (the one I used) is not very sensitive to the
> > blocking factor. Other things such as network communication topology may
> > influence the efficiency as well.
> >
> > In this version matrix setup is "semi-parallel". Full-size hamiltonian
> > and overlap matrices in packed storage form are allocated on the master node
> > only, in the other processes only a fraction of this, inversely proportional
> > to the number of processes, is needed. The pieces are then gathered into the
> > packed array on the master node, further calculations are applied to them,
> > and they are distributed in block-cyclic fashion, as required by ScaLAPACK,
> > with the local array dimensions again inversely proportional to the number
> > of processes. Memory requirements at the master process can be cut by
> > doing hamiltonian and overlap matrix setup serially. This will make the code
> > clunkier (not terribly), but will allow one to use only one full-size
> >  packed array. This will only make sense, however, same as the whole idea of
> > using ScaLAPACK, only if the number of nodes is not too small. Otherwise
> > regular matrix storage needed by ScaLAPACK (as opposed to packed storage)
> > will require a lot of memory.
> >
> > Another thing regarding this semi-parallel matrix setup is that only the
> > calls to the most time-consuming routines, i.e. hmlaa and olpaa are
> > parallelized. This, on one hand reduces the number of array variables which
> > need to be stored on all nodes, and on the other hand reduces the amount of
> > inter-process communication. The other routines in the matrix setup which
> > only use elementary operations may actually suffer from parallelization as
> > the cost of inter-process communication or of variable storage will be
> > larger that the parallelization gain.
> >
> > Andrei
> >
> >
> >  ------------------------------
> > *From:* exc...@li... [mailto:
> > exc...@li...] *On Behalf Of *Christian
> > Meisenbichler
> > *Sent:* Monday, February 25, 2008 3:36 PM
> > *To:* exc...@li...
> > *Subject:* [Exciting-developers] scalapack version remark
> >
> >   First I have to say that I'm happy about Adrei Los' Scalapack efforts.
> > I would have had to work trough it eventually one day.
> >
> > We learned that it is not particularly scalable but I think this could
> > be improved by using PARPACK [1] as we have evidence that ARPACK is
> > efficient at least on smp machines [2].
> >
> >    - Key to a Scalapack version is to get parallel matrix setup as
> >    well. Because setup takes the same magnitude of time as the solver.
> >    - It would be important to setup the matrix in place in the
> >    distributed data structure in order to avoid a memory bottleneck in the
> >    matrix setup (which would render the scalapack version essentially useless).
> >    This is as far as I can tell, not the case in the current version. The
> >    matrix is setup in packed storage form in the master node and then
> >    distributed among the other nodes.
> >
> > I think we could reach a distributed memory version with reasonable
> > scaling soon. If there are indeed problems that can take advantage of such
> > code.
> > Does anybody with access to a fast interconnect cluster have a project
> > for that?
> >
> > [1] http://www.caam.rice.edu/~kristyn/parpack_home.html<http://www.caam.rice.edu/%7Ekristyn/parpack_home.html>
> > [2] http://exciting.wiki.sourceforge.net/IterativeSolver
> >
> >
>

Christian,
 
I also think that parallel setup can be done better than I did. I did not
really have much time to spend on this. 
 
Concerning the motivation, the problem where I desperately need more
performance and to have it finely scalable is just a "usual" problem of a
semiconductor lattice with an impurity, which I need to relax. Since I need
to keep the impurity concentration reasonably low, the number of atoms and
the calculation time gets high. It's not really the memory requirement which
is critical for me, the program uses below 1GB for this calculation, but the
flops. I don't have an access to any supercomputer facility, and my cluster
is just several core duo machines. Currently I am using a modification of
Exciting which replaces OpenMP parallelism of the k-point loop by MPI
parallelism. The problem with this is that with a small number of k-points
(12 in my case) I can only use 2, 3, 4, 6, and 12 processes to have an even
CPU loading. Fine-grain parallelism would be nice, so I could use pretty
much any number of machines and they won't have to be of the same
performance. 
 
I am sure there are other problems with similar requirements.    
 
Andrei



  _____  

From: ch...@gm... [mailto:ch...@gm...] On Behalf Of Christian
Meisenbichler
Sent: Tuesday, February 26, 2008 10:25 AM
To: Andrei Los
Cc: exc...@li...
Subject: Re: [Exciting-developers] scalapack version remark


Dear Andrej

Concerning matrix setup. I'm very convinced that doing the whole setup of
the system in the parallel data structures can be done fairly elegant. The
routine hmlaa and olpaa update the matrices through a zher2 operation.
Meaning that it requires only a  order n vector  comunication to update the
matrix. and the other setup functions calculate single matrix element
contribution that may well be scoped to the distributed part. 

But really before doing all this we really have to find out if anybody would
use such a distributed memory code. Why did you start this project what was
the motivation?

Possible scenarios where a distributed memory exciting could pay off where:



*	You had free time on a blue gene 

*	or need to calculate huge clusters or surfaces that required more
memory than affordable in a single node. But I'm afraid that calculations
like that take incredibly long. 

Anyone has such a use case?



On Mon, Feb 25, 2008 at 5:41 PM, Andrei Los <and...@is...> wrote:


I have a couple more comments on the ScaLAPACK version. Indeed, it does not
seem to be very efficient judging from CPU utilization, but I did not do a
lot of testing with regard to e.g. blocking factor (only the "standard"
values of 64 and 128 were used). Intel claims though that MKL ScaLAPACK
implementation (the one I used) is not very sensitive to the blocking
factor. Other things such as network communication topology may influence
the efficiency as well.
 
In this version matrix setup is "semi-parallel". Full-size hamiltonian and
overlap matrices in packed storage form are allocated on the master node
only, in the other processes only a fraction of this, inversely proportional
to the number of processes, is needed. The pieces are then gathered into the
packed array on the master node, further calculations are applied to them,
and they are distributed in block-cyclic fashion, as required by ScaLAPACK,
with the local array dimensions again inversely proportional to the number
of processes. Memory requirements at the master process can be cut by doing
hamiltonian and overlap matrix setup serially. This will make the code
clunkier (not terribly), but will allow one to use only one full-size
packed array. This will only make sense, however, same as the whole idea of
using ScaLAPACK, only if the number of nodes is not too small. Otherwise
regular matrix storage needed by ScaLAPACK (as opposed to packed storage)
will require a lot of memory. 
 
Another thing regarding this semi-parallel matrix setup is that only the
calls to the most time-consuming routines, i.e. hmlaa and olpaa are
parallelized. This, on one hand reduces the number of array variables which
need to be stored on all nodes, and on the other hand reduces the amount of
inter-process communication. The other routines in the matrix setup which
only use elementary operations may actually suffer from parallelization as
the cost of inter-process communication or of variable storage will be
larger that the parallelization gain. 
 
Andrei
 


  _____  

From: exc...@li...
[mailto:exc...@li...] On Behalf Of
Christian Meisenbichler
Sent: Monday, February 25, 2008 3:36 PM
To: exc...@li...
Subject: [Exciting-developers] scalapack version remark


First I have to say that I'm happy about Adrei Los' Scalapack efforts. I
would have had to work trough it eventually one day.

We learned that it is not particularly scalable but I think this could be
improved by using PARPACK [1] as we have evidence that ARPACK is efficient
at least on smp machines [2].


*	Key to a Scalapack version is to get parallel matrix setup as well.
Because setup takes the same magnitude of time as the solver. 

*	It would be important to setup the matrix in place in the
distributed data structure in order to avoid a memory bottleneck in the
matrix setup (which would render the scalapack version essentially useless).
This is as far as I can tell, not the case in the current version. The
matrix is setup in packed storage form in the master node and then
distributed among the other nodes. 

I think we could reach a distributed memory version with reasonable scaling
soon. If there are indeed problems that can take advantage of such code. 
Does anybody with access to a fast interconnect cluster have a project for
that? 

[1] http://www.caam.rice.edu/~kristyn/parpack_home.html
<http://www.caam.rice.edu/%7Ekristyn/parpack_home.html> 
[2] http://exciting.wiki.sourceforge.net/IterativeSolver

Dear Andrej

Concerning matrix setup. I'm very convinced that doing the whole setup of
the system in the parallel data structures can be done fairly elegant. The
routine hmlaa and olpaa update the matrices through a zher2 operation.
Meaning that it requires only a  order n vector  comunication to update the
matrix. and the other setup functions calculate single matrix element
contribution that may well be scoped to the distributed part.

But really before doing all this we really have to find out if anybody would
use such a distributed memory code. Why did you start this project what was
the motivation?

Possible scenarios where a distributed memory exciting could pay off
where:

   - You had free time on a blue gene
   - or need to calculate huge clusters or surfaces that required more
   memory than affordable in a single node. But I'm afraid that calculations
   like that take incredibly long.

Anyone has such a use case?


On Mon, Feb 25, 2008 at 5:41 PM, Andrei Los <and...@is...> wrote:

>  I have a couple more comments on the ScaLAPACK version. Indeed, it does
> not seem to be very efficient judging from CPU utilization, but I did not do
> a lot of testing with regard to e.g. blocking factor (only the "standard"
> values of 64 and 128 were used). Intel claims though that MKL ScaLAPACK
> implementation (the one I used) is not very sensitive to the blocking
> factor. Other things such as network communication topology may influence
> the efficiency as well.
>
> In this version matrix setup is "semi-parallel". Full-size hamiltonian and
> overlap matrices in packed storage form are allocated on the master node
> only, in the other processes only a fraction of this, inversely proportional
> to the number of processes, is needed. The pieces are then gathered into the
> packed array on the master node, further calculations are applied to them,
> and they are distributed in block-cyclic fashion, as required by ScaLAPACK,
> with the local array dimensions again inversely proportional to the number
> of processes. Memory requirements at the master process can be cut by
> doing hamiltonian and overlap matrix setup serially. This will make the code
> clunkier (not terribly), but will allow one to use only one full-size
>  packed array. This will only make sense, however, same as the whole idea of
> using ScaLAPACK, only if the number of nodes is not too small. Otherwise
> regular matrix storage needed by ScaLAPACK (as opposed to packed storage)
> will require a lot of memory.
>
> Another thing regarding this semi-parallel matrix setup is that only the
> calls to the most time-consuming routines, i.e. hmlaa and olpaa are
> parallelized. This, on one hand reduces the number of array variables which
> need to be stored on all nodes, and on the other hand reduces the amount of
> inter-process communication. The other routines in the matrix setup which
> only use elementary operations may actually suffer from parallelization as
> the cost of inter-process communication or of variable storage will be
> larger that the parallelization gain.
>
> Andrei
>
>
>  ------------------------------
> *From:* exc...@li... [mailto:
> exc...@li...] *On Behalf Of *Christian
> Meisenbichler
> *Sent:* Monday, February 25, 2008 3:36 PM
> *To:* exc...@li...
> *Subject:* [Exciting-developers] scalapack version remark
>
> First I have to say that I'm happy about Adrei Los' Scalapack efforts. I
> would have had to work trough it eventually one day.
>
> We learned that it is not particularly scalable but I think this could be
> improved by using PARPACK [1] as we have evidence that ARPACK is efficient
> at least on smp machines [2].
>
>    - Key to a Scalapack version is to get parallel matrix setup as
>    well. Because setup takes the same magnitude of time as the solver.
>    - It would be important to setup the matrix in place in the
>    distributed data structure in order to avoid a memory bottleneck in the
>    matrix setup (which would render the scalapack version essentially useless).
>    This is as far as I can tell, not the case in the current version. The
>    matrix is setup in packed storage form in the master node and then
>    distributed among the other nodes.
>
> I think we could reach a distributed memory version with reasonable
> scaling soon. If there are indeed problems that can take advantage of such
> code.
> Does anybody with access to a fast interconnect cluster have a project for
> that?
>
> [1] http://www.caam.rice.edu/~kristyn/parpack_home.html<http://www.caam.rice.edu/%7Ekristyn/parpack_home.html>
> [2] http://exciting.wiki.sourceforge.net/IterativeSolver
>
>

I have a couple more comments on the ScaLAPACK version. Indeed, it does not
seem to be very efficient judging from CPU utilization, but I did not do a
lot of testing with regard to e.g. blocking factor (only the "standard"
values of 64 and 128 were used). Intel claims though that MKL ScaLAPACK
implementation (the one I used) is not very sensitive to the blocking
factor. Other things such as network communication topology may influence
the efficiency as well.
 
In this version matrix setup is "semi-parallel". Full-size hamiltonian and
overlap matrices in packed storage form are allocated on the master node
only, in the other processes only a fraction of this, inversely proportional
to the number of processes, is needed. The pieces are then gathered into the
packed array on the master node, further calculations are applied to them,
and they are distributed in block-cyclic fashion, as required by ScaLAPACK,
with the local array dimensions again inversely proportional to the number
of processes. Memory requirements at the master process can be cut by doing
hamiltonian and overlap matrix setup serially. This will make the code
clunkier (not terribly), but will allow one to use only one full-size
packed array. This will only make sense, however, same as the whole idea of
using ScaLAPACK, only if the number of nodes is not too small. Otherwise
regular matrix storage needed by ScaLAPACK (as opposed to packed storage)
will require a lot of memory. 
 
Another thing regarding this semi-parallel matrix setup is that only the
calls to the most time-consuming routines, i.e. hmlaa and olpaa are
parallelized. This, on one hand reduces the number of array variables which
need to be stored on all nodes, and on the other hand reduces the amount of
inter-process communication. The other routines in the matrix setup which
only use elementary operations may actually suffer from parallelization as
the cost of inter-process communication or of variable storage will be
larger that the parallelization gain. 
 
Andrei
 


  _____  

From: exc...@li...
[mailto:exc...@li...] On Behalf Of
Christian Meisenbichler
Sent: Monday, February 25, 2008 3:36 PM
To: exc...@li...
Subject: [Exciting-developers] scalapack version remark


First I have to say that I'm happy about Adrei Los' Scalapack efforts. I
would have had to work trough it eventually one day.

We learned that it is not particularly scalable but I think this could be
improved by using PARPACK [1] as we have evidence that ARPACK is efficient
at least on smp machines [2].


*	Key to a Scalapack version is to get parallel matrix setup as well.
Because setup takes the same magnitude of time as the solver. 

*	It would be important to setup the matrix in place in the
distributed data structure in order to avoid a memory bottleneck in the
matrix setup (which would render the scalapack version essentially useless).
This is as far as I can tell, not the case in the current version. The
matrix is setup in packed storage form in the master node and then
distributed among the other nodes. 

I think we could reach a distributed memory version with reasonable scaling
soon. If there are indeed problems that can take advantage of such code. 
Does anybody with access to a fast interconnect cluster have a project for
that? 

[1] http://www.caam.rice.edu/~kristyn/parpack_home.html
[2] http://exciting.wiki.sourceforge.net/IterativeSolver

First I have to say that I'm happy about Adrei Los' Scalapack efforts. I
would have had to work trough it eventually one day.

We learned that it is not particularly scalable but I think this could be
improved by using PARPACK [1] as we have evidence that ARPACK is efficient
at least on smp machines [2].

   - Key to a Scalapack version is to get parallel matrix setup as well.
   Because setup takes the same magnitude of time as the solver.
   - It would be important to setup the matrix in place in the
   distributed data structure in order to avoid a memory bottleneck in the
   matrix setup (which would render the scalapack version essentially useless).
   This is as far as I can tell, not the case in the current version. The
   matrix is setup in packed storage form in the master node and then
   distributed among the other nodes.

I think we could reach a distributed memory version with reasonable scaling
soon. If there are indeed problems that can take advantage of such code.
Does anybody with access to a fast interconnect cluster have a project for
that?

[1] http://www.caam.rice.edu/~kristyn/parpack_home.html
[2] http://exciting.wiki.sourceforge.net/IterativeSolver

here is Andrei's patch

http://repo.or.cz/w/exciting.git?a=shortlog;h=scalapack

---------- Forwarded message ----------
From: <mpe...@an...>
Date: Feb 20, 2008 3:22 PM
Subject: Re: [Exciting-developers] iterative diagonalisation
To: Christian Meisenbichler <chr...@un...>


SLEPC is a collection of methods, and one of them results from interfacing
to
ARPACK,
in which case you gain nothing. It does interface to other 4 o 5 libraries,
for instance
LOPEX, which is a really efficient preconditioned conjugate gradient library
for eigenproblems
(there is an original webpage for LOPEX, but it is part of the HYPRE
library).

I think that on top of those libraries, they have own methods. Certainly the
libraries are large, mainly PETSC, the basic numerical template on which
SLEPC
is built.

Manolo

PD i can dig out all webpages for you, in case you want to have a look..



> http://www.grycap.upv.es/slepc/description.htm
> says that it interfaces to arpack :-)
>
> On Feb 20, 2008 11:54 AM, <mpe...@an...> wrote:
>
>> Dear all,
>>
>> i´ve just checked the wiki regarding the arpack efforts and have a
comment
>>
>> there is a relatively recent development on iterative diagonalisation
>> called
>> SLEPC. I have not tried it myself.
>>
>> in case anyone wants to try it out, all the information (and the library
>> itself) is in:
>>
>> http://www.grycap.upv.es/slepc/
>>
>> regards
>>
>> Manolo
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Exciting-developers mailing list
>> Exc...@li...
>> https://lists.sourceforge.net/lists/listinfo/exciting-developers
>>
>

http://www.grycap.upv.es/slepc/description.htm
says that it interfaces to arpack :-)

On Feb 20, 2008 11:54 AM, <mpe...@an...> wrote:

> Dear all,
>
> i´ve just checked the wiki regarding the arpack efforts and have a comment
>
> there is a relatively recent development on iterative diagonalisation
> called
> SLEPC. I have not tried it myself.
>
> in case anyone wants to try it out, all the information (and the library
> itself) is in:
>
> http://www.grycap.upv.es/slepc/
>
> regards
>
> Manolo
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Exciting-developers mailing list
> Exc...@li...
> https://lists.sourceforge.net/lists/listinfo/exciting-developers
>

Dear all,

i´ve just checked the wiki regarding the arpack efforts and have a comment

there is a relatively recent development on iterative diagonalisation called
SLEPC. I have not tried it myself.

in case anyone wants to try it out, all the information (and the library
itself) is in:

http://www.grycap.upv.es/slepc/

regards

Manolo

In my experience the direct solver routines in scalapack do not scale well.
It really has something to do with the structure of the tridiagonalization
routines that are required in the QR decomposition of the matrix. This is
only useful if you really need all the eigenvectors and, memory -- not time
is the bottleneck.

Therefore all codes use some kind of iterative scheme. Wien2k apparently
uses some sort of jacobi davidson. which is a way to solve a projected
system of order of the number of bands. The difficult part however is to get
good approximations for the eigenvectors in order to get a fast convergence,
and this is not easy for lapw. Wien2k has sorted it out but still has to
fall back to scalapack direct solver every few iterations.

This article is interesting because numerically the gauss people have some
of the same challenges as we have

Article (Rayson2008) Rayson, M. & Briddon, P. Rapid iterative method for
electronic-structure eigenproblems using localised basis functions Computer
Physics Communications, 2008, 178, 128-134

On Feb 19, 2008 3:57 PM, Andrei Los <and...@is...> wrote
>
>
> Andrei
>  ------------------------------
>  Christian and Kay,
>
> I am also more of a novice in parallel programming, with some experience
> in ScaLAPACK only so far. The idea of using ScaLAPACK was to make a parallel
> version of Exciting with minimum modifications to the code. Besides,
> ScaLAPACK is now included in Intel MKL, which in general provides a very
> well optimized multithreaded shared-memory implementation of several
> standard math libraries. Intel of course advertises their implementation of
> ScaLAPACK as also very efficient, but as I wrote yesterday, so far my
> experience with it on real problems was disappointing. I asked a question
> regarding the poor performance in Intel support forum, but did not receive a
> reply. Maybe Christian is right in saying that the direct solver algorithm
> such as PZHEGVX just does not scale well.
>
> As you see, I also tried to use the "black box" approach as much as
> possible. What was needed on top of a call to a parallel version of the
> eigenvalue solver (PZHEGVX) is at least communicating loop-control variables
> and distributing matrices over the processes. With my limited experience, I
> can say that these will likely be required regardless of the parallel
> routine used instead of ZHPGVX. Some additional variable broadcasting was
> required when it came to parallelizing matrix setup. However, as I
> discovered from the testing, interprocess communication (via Intel BLACS)
> was not a big performance-eater in the parts of the code over which I had
> control (i.e. except PZHEGVX), so e.g. matrix setup parallelized quite
> efficiently (although I had to try several ways of doing this).
>
> Overall, I am now back to my old version of Exciting parallelized over
> k-points using MPI, which of course scales very poorly with the number of
> nodes. Christian, I am attaching the ScaLAPACK-related routines. See my
> comments on usage in my email to Kay below and email me if you have
> questions. Also, I will be interested in checking out the  PARPACK version
> you are working on as soon as you are ready to share it.
>
> Andrei
>
> p.s. Does anybody know how they parallelize Wien2k?
>
>
>  ------------------------------
> *From:* Christian Meisenbichler [mailto:ch...@gm...] *On Behalf Of *Christian
> Meisenbichler
> *Sent:* Tuesday, February 19, 2008 4:04 PM
>
> *To:* Andrei Los
> *Subject:* Re: multithreaded exciting
>
> no attachments sorry. but try this adress where im writing from now
>  Am 19.02.2008 um 14:52 schrieb Andrei Los:
>
>  Christian,
>  Did you receive my email with the attachment this morning as I was
> getting some delivery errors from The University of Leoben?
>  al
>
>  ------------------------------
> *From:* ch...@gm... [mailto:ch...@gm... <ch...@gm...>] *On
> Behalf Of *Christian Meisenbichler
> *Sent:* Tuesday, February 19, 2008 2:40 PM
> *To:* Andrei Los
> *Subject:* Re: multithreaded exciting
>
> Hi Dear Andrei
>
> Maybe you know that we have a exciting developer mailing list. if not (
> https://lists.sourceforge.net/lists/listinfo/exciting-developers)
> can I forward this discussion to the list or would you rather like to post
> yourself? I think your observations and code are interesting to most of us.
>
> Yours Christian
>
>
>
> On Feb 18, 2008 9:42 PM, J. K. Dewhurst <kay...@ed...> wrote:
>
> > Dear Christian and Andrei,
> >
> > I'm replying to both your mails jointly because the issues are the same.
> >
> > It would be great to get some fine-grained parallelism into EXCITING.
> > Unfortunately this is not something I know very much about, so you have
> > a considerable advantage here.
> >
> > Would it be possible to produce a "black-box" subroutine which I could
> > use in place of zhpgvx, which would then call whatever diagonalizer is
> > best for the system? We had some discussion about this in Uppsala, and
> > it seems to be feasible. Something along the lines of "zfftifc", but
> > where I don't have to insert compiler directives or special subroutine
> > calls elsewhere in the code.
> >
> > Let me know what you think.
> >
> > Cheers,
> > Kay.
> >
> > PS. You can get the latest internal release (with RDMFT) from:
> > http://exciting.sourceforge.net/exciting-0.9.148.tgz
> >
> > Christian Meisenbichler wrote:
> > > Dear Kay
> > >
> > > I am working on alternative eigenstate solver algorithms as you may
> > > know. The motivation is of course the n^3 scaling and the fact that
> > > direct eigen solvers are hardly to get parallel algorithms for. One of
> > > the things I tried out was the ARPACK library. It was faster in some
> > > cases than the direct solver but comparable fast or slightly slower in
> > > other cases, still the behavior was nice as it converged always to
> > > the same set of eigenvectors than the direct solvers. I recently
> > > discovert that with arpack I could replace all the time consuming blas
> > > and lapack calls by multi threaded versions of those. And it worked
> > > really nice.
> > >
> > > In fact only a tiny fraction of the time in a moderately sized
> > > calculation are spent in user routines the rest all in library calls.
> > > The parallel speedup is such that you take 3.5 times less time fore a
> > > ground state calculation if you take a 4 core cpu, instead of only one
> > > core. (I have written most of it in the wiki at sf).
> > >
> > > But there where some changes I had to do in the code in order to make
> > > this work. Mainly concerned with the matrix setup. All the profiles
> > > showed that a big chunk of the calculation was spent in zmatinp. So I
> > > replaced it wit zhpr2 which gives a dramatic gain in speed there. But
> > > this also means that the code that truncated small contributions was
> > > thrown out. I did not encounter any problems but I thought there might
> > > have been a reason for that, so I ask.
> > >
> > > The code is in the public repo and is described in the wiki.
> > > http://exciting.wiki.sourceforge.net/IterativeSolver
> > >
> > > I will announce it on the mailing list soon and hope I will find
> > > people that help to test it on other platforms and with real research
> > > systems.
> > >
> > >
> > Andrei Los wrote:
> > > Hi Kay,
> > >
> > > As a follow-up to my last email, I have done some testing of ScaLAPACK
> > > verison of Exciting on a real problem (24-atom cell) and so far the
> > > performance obtained is inferior to what I had with bone-headed MPI
> > > parallelization over k-points. The average CPU load while in the call
> > to the
> > > parallel eigenvalue solver routine PZHEGVX is way below 100%. Besides,
> > > scaling over node number is also not too good. When testing on small
> > > problems I attributed poor performance to large amount of
> > inter-process
> > > communication compared to the node-local computation amount. Now, I am
> > not
> > > sure if this poor performance is due to the particular ScaLAPACK
> > > implementation or other things. It can also be network-related,
> > although I
> > > don't see a lot of network activity while in the ScaLAPACK routine
> > call.
> > > Anyway, it would be interesting to hear from you if you have any
> > results
> > > testing this version.
> > >
> > > Best,
> > > Andrei
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: Andrei Los [mailto:and...@is...]
> > >> Sent: Thursday, January 31, 2008 1:11 PM
> > >> To: 'kay...@ed...'
> > >> Subject: RE: [exciting - EXCITING Users] RE: ScaLAPACK/BLACS
> > >> (MPI) version of Exciting
> > >>
> > >> Hi Kay,
> > >>
> > >> See the attached zip file. Some of the exciting source files
> > >> were modified (e.g. main, gndstate, seceqn). These just need
> > >> to be replaced in the original package. Several other files
> > >> have been added. I did not modify the makefile (I don't use
> > >> it), but see the enclosed batch file for a compile/link
> > >> example. I used Intel fortran 10.1.011, MKL 10.0.1.015, and
> > >> MPICH2 1.0.6p1 under Windows to compile the code. At runtime,
> > >> an additional .in file, grid.in, is needed at the "master"
> > >> node. This file defines the numbers of rows and columns in
> > >> the process grid (the product of which must be equal to the
> > >> total number of processes). I tried to optimize the code so
> > >> that minimum memory is used, so that several processes can be
> > >> run on each machine (may be useful on multicore machines).
> > >> The code tested okay on some examples and my test cases, and
> > >> I am testing it now on a large "real world" problem.
> > >>
> > >> Let me know if you have any questions.
> > >>
> > >> Best,
> > >> Andrei
> > >>
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: Nobody [mailto:no...@pr...] On Behalf Of
> > >>> SourceForge.net
> > >>> Sent: Wednesday, January 30, 2008 7:16 PM
> > >>> To: no...@so...
> > >>> Subject: [exciting - EXCITING Users] RE: ScaLAPACK/BLACS
> > >>> (MPI) version of Exciting
> > >>>
> > >>>
> > >>> Read and respond to this message at:
> > >>> https://sourceforge.net/forum/message.php?msg_id=4750840
> > >>> By: exciting
> > >>>
> > >>> Hi Andrei,
> > >>>
> > >>> Thanks for your efforts -- I'd like to see if we can
> > >>>
> > >> include your code
> > >>
> > >>> in the main release. Could you email me at kay...@ed....
> > >>>
> > >>> Regards,
> > >>> Kay.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > ______________________________________________________________________
> > >>
> > >>> You are receiving this email because you elected to monitor this
> > >>> forum.
> > >>> To stop monitoring this forum, login to SourceForge.net and visit:
> > >>>
>
>
>

Hi I forward a discussion that started in private but really would be right
here.

---------- Forwarded message ----------
From: Andrei Los <and...@is...>
Date: Feb 19, 2008 3:57 PM
Subject: RE: multithreaded exciting
To: ch...@gm...


 Christian,

Gmail keeps rejecting the message with the "Illegal attachment" error even
after I renamed the file. I am inserting the message I sent this morning
below, but please tell me if I can upload the zip file with source code
somewhere (or email to a different address).

Andrei
 ------------------------------
 Christian and Kay,

I am also more of a novice in parallel programming, with some experience in
ScaLAPACK only so far. The idea of using ScaLAPACK was to make a parallel
version of Exciting with minimum modifications to the code. Besides,
ScaLAPACK is now included in Intel MKL, which in general provides a very
well optimized multithreaded shared-memory implementation of several
standard math libraries. Intel of course advertises their implementation of
ScaLAPACK as also very efficient, but as I wrote yesterday, so far my
experience with it on real problems was disappointing. I asked a question
regarding the poor performance in Intel support forum, but did not receive a
reply. Maybe Christian is right in saying that the direct solver algorithm
such as PZHEGVX just does not scale well.

As you see, I also tried to use the "black box" approach as much as
possible. What was needed on top of a call to a parallel version of the
eigenvalue solver (PZHEGVX) is at least communicating loop-control variables
and distributing matrices over the processes. With my limited experience, I
can say that these will likely be required regardless of the parallel
routine used instead of ZHPGVX. Some additional variable broadcasting was
required when it came to parallelizing matrix setup. However, as I
discovered from the testing, interprocess communication (via Intel BLACS)
was not a big performance-eater in the parts of the code over which I had
control (i.e. except PZHEGVX), so e.g. matrix setup parallelized quite
efficiently (although I had to try several ways of doing this).

Overall, I am now back to my old version of Exciting parallelized over
k-points using MPI, which of course scales very poorly with the number of
nodes. Christian, I am attaching the ScaLAPACK-related routines. See my
comments on usage in my email to Kay below and email me if you have
questions. Also, I will be interested in checking out the  PARPACK version
you are working on as soon as you are ready to share it.

Andrei

p.s. Does anybody know how they parallelize Wien2k?


 ------------------------------
*From:* Christian Meisenbichler [mailto:ch...@gm...] *On Behalf
Of *Christian
Meisenbichler
*Sent:* Tuesday, February 19, 2008 4:04 PM

*To:* Andrei Los
*Subject:* Re: multithreaded exciting

no attachments sorry. but try this adress where im writing from now
 Am 19.02.2008 um 14:52 schrieb Andrei Los:

 Christian,
 Did you receive my email with the attachment this morning as I was getting
some delivery errors from The University of Leoben?
 al

 ------------------------------
*From:* ch...@gm... [mailto:ch...@gm... <ch...@gm...>] *On
Behalf Of *Christian Meisenbichler
*Sent:* Tuesday, February 19, 2008 2:40 PM
*To:* Andrei Los
*Subject:* Re: multithreaded exciting

Hi Dear Andrei

Maybe you know that we have a exciting developer mailing list. if not (
https://lists.sourceforge.net/lists/listinfo/exciting-developers)
can I forward this discussion to the list or would you rather like to post
yourself? I think your observations and code are interesting to most of us.

Yours Christian



On Feb 18, 2008 9:42 PM, J. K. Dewhurst <kay...@ed...> wrote:

> Dear Christian and Andrei,
>
> I'm replying to both your mails jointly because the issues are the same.
>
> It would be great to get some fine-grained parallelism into EXCITING.
> Unfortunately this is not something I know very much about, so you have
> a considerable advantage here.
>
> Would it be possible to produce a "black-box" subroutine which I could
> use in place of zhpgvx, which would then call whatever diagonalizer is
> best for the system? We had some discussion about this in Uppsala, and
> it seems to be feasible. Something along the lines of "zfftifc", but
> where I don't have to insert compiler directives or special subroutine
> calls elsewhere in the code.
>
> Let me know what you think.
>
> Cheers,
> Kay.
>
> PS. You can get the latest internal release (with RDMFT) from:
> http://exciting.sourceforge.net/exciting-0.9.148.tgz
>
> Christian Meisenbichler wrote:
> > Dear Kay
> >
> > I am working on alternative eigenstate solver algorithms as you may
> > know. The motivation is of course the n^3 scaling and the fact that
> > direct eigen solvers are hardly to get parallel algorithms for. One of
> > the things I tried out was the ARPACK library. It was faster in some
> > cases than the direct solver but comparable fast or slightly slower in
> > other cases, still the behavior was nice as it converged always to
> > the same set of eigenvectors than the direct solvers. I recently
> > discovert that with arpack I could replace all the time consuming blas
> > and lapack calls by multi threaded versions of those. And it worked
> > really nice.
> >
> > In fact only a tiny fraction of the time in a moderately sized
> > calculation are spent in user routines the rest all in library calls.
> > The parallel speedup is such that you take 3.5 times less time fore a
> > ground state calculation if you take a 4 core cpu, instead of only one
> > core. (I have written most of it in the wiki at sf).
> >
> > But there where some changes I had to do in the code in order to make
> > this work. Mainly concerned with the matrix setup. All the profiles
> > showed that a big chunk of the calculation was spent in zmatinp. So I
> > replaced it wit zhpr2 which gives a dramatic gain in speed there. But
> > this also means that the code that truncated small contributions was
> > thrown out. I did not encounter any problems but I thought there might
> > have been a reason for that, so I ask.
> >
> > The code is in the public repo and is described in the wiki.
> > http://exciting.wiki.sourceforge.net/IterativeSolver
> >
> > I will announce it on the mailing list soon and hope I will find
> > people that help to test it on other platforms and with real research
> > systems.
> >
> >
> Andrei Los wrote:
> > Hi Kay,
> >
> > As a follow-up to my last email, I have done some testing of ScaLAPACK
> > verison of Exciting on a real problem (24-atom cell) and so far the
> > performance obtained is inferior to what I had with bone-headed MPI
> > parallelization over k-points. The average CPU load while in the call to
> the
> > parallel eigenvalue solver routine PZHEGVX is way below 100%. Besides,
> > scaling over node number is also not too good. When testing on small
> > problems I attributed poor performance to large amount of inter-process
> > communication compared to the node-local computation amount. Now, I am
> not
> > sure if this poor performance is due to the particular ScaLAPACK
> > implementation or other things. It can also be network-related, although
> I
> > don't see a lot of network activity while in the ScaLAPACK routine call.
> > Anyway, it would be interesting to hear from you if you have any results
> > testing this version.
> >
> > Best,
> > Andrei
> >
> >
> >
> >> -----Original Message-----
> >> From: Andrei Los [mailto:and...@is...]
> >> Sent: Thursday, January 31, 2008 1:11 PM
> >> To: 'kay...@ed...'
> >> Subject: RE: [exciting - EXCITING Users] RE: ScaLAPACK/BLACS
> >> (MPI) version of Exciting
> >>
> >> Hi Kay,
> >>
> >> See the attached zip file. Some of the exciting source files
> >> were modified (e.g. main, gndstate, seceqn). These just need
> >> to be replaced in the original package. Several other files
> >> have been added. I did not modify the makefile (I don't use
> >> it), but see the enclosed batch file for a compile/link
> >> example. I used Intel fortran 10.1.011, MKL 10.0.1.015, and
> >> MPICH2 1.0.6p1 under Windows to compile the code. At runtime,
> >> an additional .in file, grid.in, is needed at the "master"
> >> node. This file defines the numbers of rows and columns in
> >> the process grid (the product of which must be equal to the
> >> total number of processes). I tried to optimize the code so
> >> that minimum memory is used, so that several processes can be
> >> run on each machine (may be useful on multicore machines).
> >> The code tested okay on some examples and my test cases, and
> >> I am testing it now on a large "real world" problem.
> >>
> >> Let me know if you have any questions.
> >>
> >> Best,
> >> Andrei
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Nobody [mailto:no...@pr...] On Behalf Of
> >>> SourceForge.net
> >>> Sent: Wednesday, January 30, 2008 7:16 PM
> >>> To: no...@so...
> >>> Subject: [exciting - EXCITING Users] RE: ScaLAPACK/BLACS
> >>> (MPI) version of Exciting
> >>>
> >>>
> >>> Read and respond to this message at:
> >>> https://sourceforge.net/forum/message.php?msg_id=4750840
> >>> By: exciting
> >>>
> >>> Hi Andrei,
> >>>
> >>> Thanks for your efforts -- I'd like to see if we can
> >>>
> >> include your code
> >>
> >>> in the main release. Could you email me at kay...@ed....
> >>>
> >>> Regards,
> >>> Kay.
> >>>
> >>>
> >>>
> >>>
> >> ______________________________________________________________________
> >>
> >>> You are receiving this email because you elected to monitor this
> >>> forum.
> >>> To stop monitoring this forum, login to SourceForge.net and visit:
> >>>

Hi I had some impressive results with ARPACK in combination with SMP enabled
BLAS bzw LAPACK.
see
http://exciting.wiki.sourceforge.net/IterativeSolver

The code is online, please get it and test it on the platforms you get hold
of. I tested is on Power 5+ quad core cpus getting speedups of 3.5 with 4
cores.

More information in the wiki, please post any questions on this list, I will
be happy to answer

We do have 9 Members on this list! Which is great. Here every member is free
to post.

Claudia Ambrosch-Draxl
Christian Meisenbichler
francesco cricchio
Frederik Bultmark
J. Kay Dewhurst
MANUEL PEREZ JIGATO
Ricardo Gomez-Abal
Stephan Sagmeister
Juergen Spitaler

Thank you very much for participating, this is going to be fun :-)

I didn't send the application in time. So I wont be there. Sorry! The notice
that the contingent is full came in on Sunday.

On Jan 10, 2008 12:53 AM, francesco cricchio <fra...@fy...>
wrote:

> Hello everyone,
> I just wanted to say that I'm following Christian suggestion and I'm
> joining the CECAM tutorial about software development. If someone else
> from exciting developers is coming we could take the occasion to meet
> and discuss in person about Exciting development topics.
> I verified that travel and accomodation expenses for Phd students will
> be covered by CECAM .
> Best
> Francesco Cricchio
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Exciting-developers mailing list
> Exc...@li...
> https://lists.sourceforge.net/lists/listinfo/exciting-developers
>

Hello everyone,
I just wanted to say that I'm following Christian suggestion and I'm 
joining the CECAM tutorial about software development. If someone else 
from exciting developers is coming we could take the occasion to meet 
and discuss in person about Exciting development topics.
I verified that travel and accomodation expenses for Phd students will 
be covered by CECAM .
Best
Francesco Cricchio

Hello everyone,
I just wanted to say that I'm following Christian suggestion and I'm 
joining the CECAM tutorial about software development. If someone else 
from exciting developers is coming we could take the occasion to meet 
and discuss in person about Exciting development topics.
I verified that travel and accomodation expenses for Phd students will 
be covered by CECAM .
Best
Francesco Cricchio

CECAM Tutorial about software development.

http://tinyurl.com/3dggh2

<quote>
(1) Basic concepts of software maintenance (concepts [1-4] ; coding  
rules, ROBODOC [5]) ; (2) Version management tools (concepts ; bzr  
[6] and svn [7]) ; (3) Code re-use. Libraries. (concepts ; NetCDF  
[8], ETSF_IO [9], LibXC) ; (4) File formats. Conversion tools.  
(concepts ; NetCDF [8] , NQ FileFormat [10], ETSF_IO [9], XML [11],  
pspconvert [9]) ; (5) Scripting. Introduction to Python [12] ; (6)  
Testing, debugging, profiling, optimizing. (concepts - idb and gdb  
[13] ) ; (7) Build system : the autotools and platform specificities  
[14,15].
</quote>

forwarded message

 > Dear members of the psi-k community,
 >
 > Perhaps you will be interested by the CECAM tutorial "Basic  
techniques
 > and tools for the development of atomic-scale softwares", to be  
held at
 > CECAM, Lyon, France, Feb. 11 to Feb. 15, 2008. Please see

 > http://www.cecam.org/index.php?content=activities/ 
tutorial&action=details&wid=198
 > The techniques/tools presented there should be useful for generic
 > software development.
 >
 > Deadline for applications is January 17, 2008 ( note that the  
applicant
 > CV should be adequate - knowledge and practice of Unix, and either  
C or
 > Fortran).
 >
 > Happy new year 2008,
 > Xavier Gonze