16 Mar
2023
16 Mar
'23
2:30 p.m.
I can lend some insights to the `mpi4py` side of things. You can `pip
install mpi4py`, but if your machine contains multiple MPI implementations,
make sure to prepend the correct MPI wrappers to the command, e.g.:
CC=mpicc pip install mpi4py
Afterwards you can use:
import nest
from mpi4py.MPI import COMM_WORLD
...
local_data = nest.GetConnections(...)
global_data = COMM_WORLD.allgather(local_data)
`global_data` will now be a list of local_data's collected from all the
nodes. I suppose `allgather` might be a bit overkill, because all data is
gathered on all nodes. There's also `COMM_WORLD.gather` which collects the
data on just 1 node.
On Thu, 16 Mar 2023 at 14:19, Xavier Otazu Porter <xotazu(a)cvc.uab.cat>
wrote:
Hans,
--
Robin De Schepper, MSc (they/them)
Department of Brain and Behavioral Sciences
Unit of Neurophysiology
University of Pavia, Italy
Via Forlanini 6, 27100 Pavia - Italy
Tel: (+39) 038298-7607
http://www-5.unipv.it/dangelo/
We clearly need to fix the bug. Could you create
a minimal reproducer
and then create an issue
https://github.com/nest/nest-simulator/issues?
Yes I will.
As a work-around for global normalization of
weights, you could try to
use MPI4Py to compute the sum of the local weights.
I should learn MPI4Py. I am not an expert in MPI (although I am fluent
with threads programming). In addition I am clueless about what pyNEST
functions to use in order to gather all the pre-synaptic connections of a
neuron (different to GetConnections() ).
If your network sizes do not go into the
millions, maybe systems with
large numbers of threads could help, e.g., JUSUF which
has computed notes
with 128 cores. You can apply for access through FENIX (
https://fenix-ri.eu). The Human Brain Project provides relatively
easy-to-apply for resources there.
The problem is that my architecture sometimes (many times, in fact) needs
more than 128Gb RAM (just the RAM size of the nodes the cluster I am
working on). Hence, I need to work with MPI to distribute my architecture
through several nodes.
Now I am working with 1 mpi process (hence, just one node) and 32 threads.
But it implies to work with "small" architectures.
You could also issue a feature request issue on
NEST for support for
global normalization.
I will!
Thanks a lot in advance!
Xavier
Best regards,
Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser
Head, Department of Data Science
Faculty of Science and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560
Email hans.ekkehard.plesser(a)nmbu.no
Home http://arken.nmbu.no/~plesser
*From: *Xavier Otazu <xotazu(a)cvc.uab.cat>
*Date: *Thursday, 16 March 2023 at 11:57
*To: *users(a)nest-simulator.org <users(a)nest-simulator.org>
*Subject: *[NEST Users] Re: MPI error when using GetConnections
[Du mottar ikke ofte e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette
er viktig p? https://aka.ms/LearnAboutSenderIdentification ]
Hi Hans!
Commenting out the DumpLayerConnections() lines, it also crashes.
In addition, the problem disappears when I reduce the size of the number
of neurons or the connection shape (but it is not practical for me).
Hence, if it is only the normalization of the local MPI process, I
understand the only way to normalize the overall sum of weights of a
particular neuron is to not using MPI? It is impractical for me because I
use big networks (several tenths of thousands).
Cheers,
Xavier
_______________________________________________
NEST Users mailing list -- users(a)nest-simulator.org
To unsubscribe send an email to users-leave(a)nest-simulator.org
Computer Vision Center <http://www.cvc.uab.cat>
CONFIDENTIALITY WARNING <http://www.cvc.uab.es/?page_id=7475>
_______________________________________________
NEST Users mailing list -- users(a)nest-simulator.org
To unsubscribe send an email to users-leave(a)nest-simulator.org