MPI error when using GetConnections

List overview All Threads
Download

newer

older

Open NEST Developer Video...

Xavier Otazu

16 Mar 2023 16 Mar '23

11:23 a.m.

Hello, I am trying to normalize connection weights using a code similar to the one on the Documentation (see code below). I receive an error when assigning the modified weights on the last line of the for loop below (only when I use several MPI processes). That is, when using mpirun -np 1 python3 test_DumpLayerConnections.py I do not receive any error, but when using 4 MPI processes: mpirun -np 4 python3 test_DumpLayerConnections.py I receive the following error when trying to start a Simulation: Mar 16 11:13:53 NodeManager::prepare_nodes [Info]: Preparing 675 nodes for simulation. [dcccluster:747534] *** An error occurred in MPI_Allgather [dcccluster:747534] *** reported by process [2696478721,0] [dcccluster:747534] *** on communicator MPI_COMM_WORLD [dcccluster:747534] *** MPI_ERR_TRUNCATE: message truncated [dcccluster:747534] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [dcccluster:747534] *** and potentially your MPI job) I can avoid this error by commenting the last for loop assignament (the normalized weights assignment operation). I am using NEST 3.3 (compiled with MPI option) and Python 3.8.12. I guess it could be related to the DumpLayerConnections bug I told you some days ago. Thanks a lot in advance! Xavier import nest import numpy as np pos = nest.spatial.grid(shape = [30,30] ) input = nest.Create('iaf_psc_alpha', positions=pos) layer_0 = nest.Create('iaf_psc_alpha', positions=pos) layer_1 = nest.Create('iaf_psc_alpha', positions=pos) conn_neur = {'rule':'pairwise_bernoulli', 'use_on_source': True, 'mask': {'grid':{'shape':[9,9]}} } nest.CopyModel('static_synapse', 'syn_1_model') syn_0 = {'synapse_model': 'static_synapse'} syn_1 = {'synapse_model': 'syn_1_model'} nest.Connect(input, layer_0, conn_neur, syn_0) nest.Connect(input, layer_1, conn_neur, syn_1) nest.DumpLayerConnections(input, layer_0, 'static_synapse', 'conn.txt') nest.DumpLayerConnections(input, layer_1, 'syn_1_model', 'conn.txt') for neuron in layer_0: conn = nest.GetConnections(target=neuron, synapse_model='static_synapse') w = np.array(conn.weight) if (w.size>1): w_normed = w / sum(abs(w)) # L1-norm w_nparray = 2. * w_normed conn.weight = w_nparray.tolist() nest.Simulate(100)

Show replies by date

Hans Ekkehard Plesser

16 Mar 16 Mar

11:44 a.m.

Hi Xavier, A question and a comment: Does the error also occur when you comment out the DumpLayerConnections() calls? When running with multiple MPI processes, GetConnections() will only return the connections stored locally on each MPI process, so the normalization you apply in your script would be by the local sum of weights on each MPI process, not by the overall sum of weights. To normalize globally, one might be able to construct a work-around using MPI4Py (which may open new MPI issues, so one would need to be careful) or one would have to implement support for global normalization deep in the NEST kernel. Communicating all connections to all MPI processes (or all weights) would not make sense due to the bandwidth it would require. Best regards, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Xavier Otazu <xotazu(a)cvc.uab.cat> Date: Thursday, 16 March 2023 at 11:23 To: users(a)nest-simulator.org <users(a)nest-simulator.org> Subject: [NEST Users] MPI error when using GetConnections [Some people who received this message don't often get email from xotazu(a)cvc.uab.cat. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hello, I am trying to normalize connection weights using a code similar to the one on the Documentation (see code below). I receive an error when assigning the modified weights on the last line of the for loop below (only when I use several MPI processes). That is, when using mpirun -np 1 python3 test_DumpLayerConnections.py I do not receive any error, but when using 4 MPI processes: mpirun -np 4 python3 test_DumpLayerConnections.py I receive the following error when trying to start a Simulation: Mar 16 11:13:53 NodeManager::prepare_nodes [Info]: Preparing 675 nodes for simulation. [dcccluster:747534] *** An error occurred in MPI_Allgather [dcccluster:747534] *** reported by process [2696478721,0] [dcccluster:747534] *** on communicator MPI_COMM_WORLD [dcccluster:747534] *** MPI_ERR_TRUNCATE: message truncated [dcccluster:747534] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [dcccluster:747534] *** and potentially your MPI job) I can avoid this error by commenting the last for loop assignament (the normalized weights assignment operation). I am using NEST 3.3 (compiled with MPI option) and Python 3.8.12. I guess it could be related to the DumpLayerConnections bug I told you some days ago. Thanks a lot in advance! Xavier import nest import numpy as np pos = nest.spatial.grid(shape = [30,30] ) input = nest.Create('iaf_psc_alpha', positions=pos) layer_0 = nest.Create('iaf_psc_alpha', positions=pos) layer_1 = nest.Create('iaf_psc_alpha', positions=pos) conn_neur = {'rule':'pairwise_bernoulli', 'use_on_source': True, 'mask': {'grid':{'shape':[9,9]}} } nest.CopyModel('static_synapse', 'syn_1_model') syn_0 = {'synapse_model': 'static_synapse'} syn_1 = {'synapse_model': 'syn_1_model'} nest.Connect(input, layer_0, conn_neur, syn_0) nest.Connect(input, layer_1, conn_neur, syn_1) nest.DumpLayerConnections(input, layer_0, 'static_synapse', 'conn.txt') nest.DumpLayerConnections(input, layer_1, 'syn_1_model', 'conn.txt') for neuron in layer_0: conn = nest.GetConnections(target=neuron, synapse_model='static_synapse') w = np.array(conn.weight) if (w.size>1): w_normed = w / sum(abs(w)) # L1-norm w_nparray = 2. * w_normed conn.weight = w_nparray.tolist() nest.Simulate(100) _______________________________________________ NEST Users mailing list -- users(a)nest-simulator.org To unsubscribe send an email to users-leave(a)nest-simulator.org

Xavier Otazu

11:57 a.m.

Hi Hans! Commenting out the DumpLayerConnections() lines, it also crashes. In addition, the problem disappears when I reduce the size of the number of neurons or the connection shape (but it is not practical for me). Hence, if it is only the normalization of the local MPI process, I understand the only way to normalize the overall sum of weights of a particular neuron is to not using MPI? It is impractical for me because I use big networks (several tenths of thousands). Cheers, Xavier

Hans Ekkehard Plesser

12:53 p.m.

Hi Xavier, We clearly need to fix the bug. Could you create a minimal reproducer and then create an issue https://github.com/nest/nest-simulator/issues? As a work-around for global normalization of weights, you could try to use MPI4Py to compute the sum of the local weights. If your network sizes do not go into the millions, maybe systems with large numbers of threads could help, e.g., JUSUF which has computed notes with 128 cores. You can apply for access through FENIX (https://fenix-ri.eu). The Human Brain Project provides relatively easy-to-apply for resources there. You could also issue a feature request issue on NEST for support for global normalization. Best regards, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Xavier Otazu <xotazu(a)cvc.uab.cat> Date: Thursday, 16 March 2023 at 11:57 To: users(a)nest-simulator.org <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections [Du mottar ikke ofte e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette er viktig p? https://aka.ms/LearnAboutSenderIdentification ] Hi Hans! Commenting out the DumpLayerConnections() lines, it also crashes. In addition, the problem disappears when I reduce the size of the number of neurons or the connection shape (but it is not practical for me). Hence, if it is only the normalization of the local MPI process, I understand the only way to normalize the overall sum of weights of a particular neuron is to not using MPI? It is impractical for me because I use big networks (several tenths of thousands). Cheers, Xavier _______________________________________________ NEST Users mailing list -- users(a)nest-simulator.org To unsubscribe send an email to users-leave(a)nest-simulator.org

Xavier Otazu Porter

2:19 p.m.

Hans,

...

We clearly need to fix the bug. Could you create a minimal reproducer and then create an issue https://github.com/nest/nest-simulator/issues?

Yes I will.

...

As a work-around for global normalization of weights, you could try to use MPI4Py to compute the sum of the local weights.

I should learn MPI4Py. I am not an expert in MPI (although I am fluent with threads programming). In addition I am clueless about what pyNEST functions to use in order to gather all the pre-synaptic connections of a neuron (different to GetConnections() ).

...

If your network sizes do not go into the millions, maybe systems with large numbers of threads could help, e.g., JUSUF which has computed notes with 128 cores. You can apply for access through FENIX (https://fenix-ri.eu). The Human Brain Project provides relatively easy-to-apply for resources there.

The problem is that my architecture sometimes (many times, in fact) needs more than 128Gb RAM (just the RAM size of the nodes the cluster I am working on). Hence, I need to work with MPI to distribute my architecture through several nodes. Now I am working with 1 mpi process (hence, just one node) and 32 threads. But it implies to work with "small" architectures.

...

You could also issue a feature request issue on NEST for support for global normalization.

Robin Gilbert De Schepper

2:30 p.m.

I can lend some insights to the `mpi4py` side of things. You can `pip install mpi4py`, but if your machine contains multiple MPI implementations, make sure to prepend the correct MPI wrappers to the command, e.g.: CC=mpicc pip install mpi4py Afterwards you can use: import nest from mpi4py.MPI import COMM_WORLD ... local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. On Thu, 16 Mar 2023 at 14:19, Xavier Otazu Porter <xotazu(a)cvc.uab.cat> wrote:

...

Hans,

We clearly need to fix the bug. Could you create a minimal reproducer

and then create an issue https://github.com/nest/nest-simulator/issues? Yes I will.

As a work-around for global normalization of weights, you could try to

use MPI4Py to compute the sum of the local weights. I should learn MPI4Py. I am not an expert in MPI (although I am fluent with threads programming). In addition I am clueless about what pyNEST functions to use in order to gather all the pre-synaptic connections of a neuron (different to GetConnections() ).

If your network sizes do not go into the millions, maybe systems with

large numbers of threads could help, e.g., JUSUF which has computed notes with 128 cores. You can apply for access through FENIX ( https://fenix-ri.eu). The Human Brain Project provides relatively easy-to-apply for resources there. The problem is that my architecture sometimes (many times, in fact) needs more than 128Gb RAM (just the RAM size of the nodes the cluster I am working on). Hence, I need to work with MPI to distribute my architecture through several nodes. Now I am working with 1 mpi process (hence, just one node) and 32 threads. But it implies to work with "small" architectures.

You could also issue a feature request issue on NEST for support for

global normalization. I will! Thanks a lot in advance! Xavier Best regards, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser(a)nmbu.no Home http://arken.nmbu.no/~plesser *From: *Xavier Otazu <xotazu(a)cvc.uab.cat> *Date: *Thursday, 16 March 2023 at 11:57 *To: *users(a)nest-simulator.org <users(a)nest-simulator.org> *Subject: *[NEST Users] Re: MPI error when using GetConnections [Du mottar ikke ofte e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette er viktig p? https://aka.ms/LearnAboutSenderIdentification ] Hi Hans! Commenting out the DumpLayerConnections() lines, it also crashes. In addition, the problem disappears when I reduce the size of the number of neurons or the connection shape (but it is not practical for me). Hence, if it is only the normalization of the local MPI process, I understand the only way to normalize the overall sum of weights of a particular neuron is to not using MPI? It is impractical for me because I use big networks (several tenths of thousands). Cheers, Xavier _______________________________________________ NEST Users mailing list -- users(a)nest-simulator.org To unsubscribe send an email to users-leave(a)nest-simulator.org Computer Vision Center <http://www.cvc.uab.cat> CONFIDENTIALITY WARNING <http://www.cvc.uab.es/?page_id=7475> _______________________________________________ NEST Users mailing list -- users(a)nest-simulator.org To unsubscribe send an email to users-leave(a)nest-simulator.org

-- Robin De Schepper, MSc (they/them) Department of Brain and Behavioral Sciences Unit of Neurophysiology University of Pavia, Italy Via Forlanini 6, 27100 Pavia - Italy Tel: (+39) 038298-7607 http://www-5.unipv.it/dangelo/

Hans Ekkehard Plesser

3:44 p.m.

Hi Robin, Thanks for pitching in—I'm not an expert in MPI4py myself ;). local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. This will not quite work and is not as efficient as possible in this case. GetConnections returns a synapse collection which can only be used to access local neurons. One needs to communicate the actual synapse properties of interest between processes. Furthermore, in this case we only need the total weight of the synapses, so we can first compute that locally and then reduce: conn = nest.GetConnections(target=neuron, synapse_model='static_synapse') w_syns = np.array( conn.weight ) w_total = COMM_WORLD.allreduce(np.abs(w_syns).sum()) if w_total > 0: w_normed = w_syns / w_total ... If you normalize several times during a longer simulation, you should call GetConnections() only the first time you need it, since it can be slow. As long as you do not add or remove connections, the conn object will remain valid. I'd be much interested in how these ideas work :). BTW, we will soon open abstract submission for this year's NEST Conference (https://nest-simulator.org/conference).<https://nest-simulator.org/confe… You could show your work there and discuss with the NEST community! Best, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Robin Gilbert De Schepper <robingilbert.deschepper(a)unipv.it> Date: Thursday, 16 March 2023 at 14:30 To: NEST User Mailing List <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections I can lend some insights to the `mpi4py` side of things. You can `pip install mpi4py`, but if your machine contains multiple MPI implementations, make sure to prepend the correct MPI wrappers to the command, e.g.: CC=mpicc pip install mpi4py Afterwards you can use: import nest from mpi4py.MPI import COMM_WORLD ... local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. On Thu, 16 Mar 2023 at 14:19, Xavier Otazu Porter <xotazu@cvc.uab.cat<mailto:xotazu@cvc.uab.cat>> wrote: Hans,

...

We clearly need to fix the bug. Could you create a minimal reproducer and then create an issue https://github.com/nest/nest-simulator/issues?

Yes I will.

...

As a work-around for global normalization of weights, you could try to use MPI4Py to compute the sum of the local weights.

...

You could also issue a feature request issue on NEST for support for global normalization.

I will! Thanks a lot in advance! Xavier Best regards, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Xavier Otazu <xotazu@cvc.uab.cat<mailto:xotazu@cvc.uab.cat>> Date: Thursday, 16 March 2023 at 11:57 To: users@nest-simulator.org<mailto:users@nest-simulator.org> <users@nest-simulator.org<mailto:users@nest-simulator.org>> Subject: [NEST Users] Re: MPI error when using GetConnections [Du mottar ikke ofte e-post fra xotazu@cvc.uab.cat<mailto:xotazu@cvc.uab.cat>. Finn ut hvorfor dette er viktig p? https://aka.ms/LearnAboutSenderIdentification ] Hi Hans! Commenting out the DumpLayerConnections() lines, it also crashes. In addition, the problem disappears when I reduce the size of the number of neurons or the connection shape (but it is not practical for me). Hence, if it is only the normalization of the local MPI process, I understand the only way to normalize the overall sum of weights of a particular neuron is to not using MPI? It is impractical for me because I use big networks (several tenths of thousands). Cheers, Xavier _______________________________________________ NEST Users mailing list -- users@nest-simulator.org<mailto:users@nest-simulator.org> To unsubscribe send an email to users-leave@nest-simulator.org<mailto:users-leave@nest-simulator.org> Computer Vision Center<http://www.cvc.uab.cat> CONFIDENTIALITY WARNING<http://www.cvc.uab.es/?page_id=7475> _______________________________________________ NEST Users mailing list -- users@nest-simulator.org<mailto:users@nest-simulator.org> To unsubscribe send an email to users-leave@nest-simulator.org<mailto:users-leave@nest-simulator.org> -- Robin De Schepper, MSc (they/them) Department of Brain and Behavioral Sciences Unit of Neurophysiology University of Pavia, Italy Via Forlanini 6, 27100 Pavia - Italy Tel: (+39) 038298-7607 http://www-5.unipv.it/dangelo/

Xavier Otazu Porter

6:10 p.m.

Hans and Robin, Thanks for the tip! 🙂 But I believe I didn't explained correctly myself. In the code you suggest, I understand the local connections are normalized using the global amount of weights. That is, the sum of all the weights in the network will be one. This is not what I am interested in. I just want to normalize the sum of the presynaptic weights to a neuron (target). That is, I am interested in the same idea you publish on your weight normalization documentation page https://nest-simulator.readthedocs.io/en/v3.3/guides/weight_normalization.h… I guessed that the example code in this documentation page would work, because it is OK for me to receive only the targets on the MPI process executing the command (as you say in the documentation) because I want to normalize for every target neuron. And it works!, but only if I do not use MPI. I am sorry if I am missing some point. Best, Xavier ________________________________ From: Hans Ekkehard Plesser <hans.ekkehard.plesser(a)nmbu.no> Sent: Thursday, March 16, 2023 3:44 PM To: NEST User Mailing List <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections Hi Robin, Thanks for pitching in—I'm not an expert in MPI4py myself ;). local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. This will not quite work and is not as efficient as possible in this case. GetConnections returns a synapse collection which can only be used to access local neurons. One needs to communicate the actual synapse properties of interest between processes. Furthermore, in this case we only need the total weight of the synapses, so we can first compute that locally and then reduce: conn = nest.GetConnections(target=neuron, synapse_model='static_synapse') w_syns = np.array( conn.weight ) w_total = COMM_WORLD.allreduce(np.abs(w_syns).sum()) if w_total > 0: w_normed = w_syns / w_total ... If you normalize several times during a longer simulation, you should call GetConnections() only the first time you need it, since it can be slow. As long as you do not add or remove connections, the conn object will remain valid. I'd be much interested in how these ideas work :). BTW, we will soon open abstract submission for this year's NEST Conference (https://nest-simulator.org/conference).<https://nest-simulator.org/confe… You could show your work there and discuss with the NEST community! Best, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Robin Gilbert De Schepper <robingilbert.deschepper(a)unipv.it> Date: Thursday, 16 March 2023 at 14:30 To: NEST User Mailing List <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections I can lend some insights to the `mpi4py` side of things. You can `pip install mpi4py`, but if your machine contains multiple MPI implementations, make sure to prepend the correct MPI wrappers to the command, e.g.: CC=mpicc pip install mpi4py Afterwards you can use: import nest from mpi4py.MPI import COMM_WORLD ... local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. On Thu, 16 Mar 2023 at 14:19, Xavier Otazu Porter <xotazu@cvc.uab.cat<mailto:xotazu@cvc.uab.cat>> wrote: Hans,

...

We clearly need to fix the bug. Could you create a minimal reproducer and then create an issue https://github.com/nest/nest-simulator/issues?

Yes I will.

...

As a work-around for global normalization of weights, you could try to use MPI4Py to compute the sum of the local weights.

...

You could also issue a feature request issue on NEST for support for global normalization.

Hans Ekkehard Plesser

17 Mar 17 Mar

8:09 a.m.

Hi Xavier, Assuming that "neuron" is a NodeCollection representing a single neuron, then conn = nest.GetConnections(target=neuron, synapse_model='static_synapse') will extract only those connections that have this particular neuron as target, and conn.weight will return the weights of only these connections, i.e., the weights of all incoming connections to neuron (of type static_synapse). Aren't those the weights you need? One thing I have not looked into yet is the optimal way to handle normalization of input to many neurons, where the weights into each neuron should be handled separately. The simplest way would be to just loop over the procedure I suggested, which would mean having a separate "conn" for each target neuron, but that would entail lots of GetConnections, conn.weight, and mpi allreduce calls. A more efficient way would probably be to do conn = nest.GetConnections(source=all_relevant_sources, target=all_relevant_targets, synapse_model=...) and then for each round w_local = conn.weight Now the challenge comes to group and sum the weights in w_local by target neuron, so we get an array with the total input weight to each target neuron on each rank. allreduce() can then sum up these arrays so you get arrays with the total input weight to each target neuron. The main challenges here (and I don't have time to look into this right now) are: 1. How to group locally by target neuron. As long as connections don't change, the grouping will be constant and needs to be worked out only once after calling GetConnections. 2. Efficiently do the local summation. 3. Ensure that all MPI ranks have the local sum for each target neuron in the same location in the weight array. Essentially, the weight array passed to allreduce must have entries for all target neurons, while the locally computed weights will only have the local neurons. So one needs to insert the local values in the right locations of the global array. I am sorry if this sounds a bit confusing, I'll see if I or someone else can get back to you later, but with several conferences ahead, time is short right now. Best, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Xavier Otazu Porter <xotazu(a)cvc.uab.cat> Date: Thursday, 16 March 2023 at 18:12 To: NEST User Mailing List <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections Det er ikke ofte du mottar e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette er viktig.<https://aka.ms/LearnAboutSenderIdentification> Hans and Robin, Thanks for the tip! 🙂 But I believe I didn't explained correctly myself. In the code you suggest, I understand the local connections are normalized using the global amount of weights. That is, the sum of all the weights in the network will be one. This is not what I am interested in. I just want to normalize the sum of the presynaptic weights to a neuron (target). That is, I am interested in the same idea you publish on your weight normalization documentation page https://nest-simulator.readthedocs.io/en/v3.3/guides/weight_normalization.h… I guessed that the example code in this documentation page would work, because it is OK for me to receive only the targets on the MPI process executing the command (as you say in the documentation) because I want to normalize for every target neuron. And it works!, but only if I do not use MPI. I am sorry if I am missing some point. Best, Xavier ________________________________ From: Hans Ekkehard Plesser <hans.ekkehard.plesser(a)nmbu.no> Sent: Thursday, March 16, 2023 3:44 PM To: NEST User Mailing List <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections Hi Robin, Thanks for pitching in—I'm not an expert in MPI4py myself ;). local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. This will not quite work and is not as efficient as possible in this case. GetConnections returns a synapse collection which can only be used to access local neurons. One needs to communicate the actual synapse properties of interest between processes. Furthermore, in this case we only need the total weight of the synapses, so we can first compute that locally and then reduce: conn = nest.GetConnections(target=neuron, synapse_model='static_synapse') w_syns = np.array( conn.weight ) w_total = COMM_WORLD.allreduce(np.abs(w_syns).sum()) if w_total > 0: w_normed = w_syns / w_total ... If you normalize several times during a longer simulation, you should call GetConnections() only the first time you need it, since it can be slow. As long as you do not add or remove connections, the conn object will remain valid. I'd be much interested in how these ideas work :). BTW, we will soon open abstract submission for this year's NEST Conference (https://nest-simulator.org/conference).<https://nest-simulator.org/confe… You could show your work there and discuss with the NEST community! Best, Hans Ekkehard -- Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no> Home http://arken.nmbu.no/~plesser From: Robin Gilbert De Schepper <robingilbert.deschepper(a)unipv.it> Date: Thursday, 16 March 2023 at 14:30 To: NEST User Mailing List <users(a)nest-simulator.org> Subject: [NEST Users] Re: MPI error when using GetConnections I can lend some insights to the `mpi4py` side of things. You can `pip install mpi4py`, but if your machine contains multiple MPI implementations, make sure to prepend the correct MPI wrappers to the command, e.g.: CC=mpicc pip install mpi4py Afterwards you can use: import nest from mpi4py.MPI import COMM_WORLD ... local_data = nest.GetConnections(...) global_data = COMM_WORLD.allgather(local_data) `global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node. On Thu, 16 Mar 2023 at 14:19, Xavier Otazu Porter <xotazu@cvc.uab.cat<mailto:xotazu@cvc.uab.cat>> wrote: Hans,

...

We clearly need to fix the bug. Could you create a minimal reproducer and then create an issue https://github.com/nest/nest-simulator/issues?

Yes I will.

...

As a work-around for global normalization of weights, you could try to use MPI4Py to compute the sum of the local weights.

...

You could also issue a feature request issue on NEST for support for global normalization.

645

days inactive

646

days old

users@nest-simulator.org

Manage subscription

8 comments

4 participants

tags (0)

participants (4)

Hans Ekkehard Plesser
Robin Gilbert De Schepper
Xavier Otazu
Xavier Otazu Porter