16 Mar
2023
16 Mar
'23
11:44 a.m.
Hi Xavier,
A question and a comment:
Does the error also occur when you comment out the DumpLayerConnections() calls?
When running with multiple MPI processes, GetConnections() will only return the
connections stored locally on each MPI process, so the normalization you apply in your
script would be by the local sum of weights on each MPI process, not by the overall sum of
weights. To normalize globally, one might be able to construct a work-around using MPI4Py
(which may open new MPI issues, so one would need to be careful) or one would have to
implement support for global normalization deep in the NEST kernel. Communicating all
connections to all MPI processes (or all weights) would not make sense due to the
bandwidth it would require.
Best regards,
Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser
Head, Department of Data Science
Faculty of Science and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560
Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no>
Home http://arken.nmbu.no/~plesser
From: Xavier Otazu <xotazu(a)cvc.uab.cat>
Date: Thursday, 16 March 2023 at 11:23
To: users(a)nest-simulator.org <users(a)nest-simulator.org>
Subject: [NEST Users] MPI error when using GetConnections
[Some people who received this message don't often get email from xotazu(a)cvc.uab.cat.
Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Hello,
I am trying to normalize connection weights using a code similar to the one on the
Documentation (see code below). I receive an error when assigning the modified weights on
the last line of the for loop below (only when I use several MPI processes). That is, when
using
mpirun -np 1 python3 test_DumpLayerConnections.py
I do not receive any error, but when using 4 MPI processes:
mpirun -np 4 python3 test_DumpLayerConnections.py
I receive the following error when trying to start a Simulation:
Mar 16 11:13:53 NodeManager::prepare_nodes [Info]:
Preparing 675 nodes for simulation.
[dcccluster:747534] *** An error occurred in MPI_Allgather
[dcccluster:747534] *** reported by process [2696478721,0]
[dcccluster:747534] *** on communicator MPI_COMM_WORLD
[dcccluster:747534] *** MPI_ERR_TRUNCATE: message truncated
[dcccluster:747534] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now
abort,
[dcccluster:747534] *** and potentially your MPI job)
I can avoid this error by commenting the last for loop assignament (the normalized weights
assignment operation). I am using NEST 3.3 (compiled with MPI option) and Python 3.8.12. I
guess it could be related to the DumpLayerConnections bug I told you some days ago.
Thanks a lot in advance!
Xavier
import nest
import numpy as np
pos = nest.spatial.grid(shape = [30,30] )
input = nest.Create('iaf_psc_alpha', positions=pos)
layer_0 = nest.Create('iaf_psc_alpha', positions=pos)
layer_1 = nest.Create('iaf_psc_alpha', positions=pos)
conn_neur = {'rule':'pairwise_bernoulli', 'use_on_source': True,
'mask': {'grid':{'shape':[9,9]}} }
nest.CopyModel('static_synapse', 'syn_1_model')
syn_0 = {'synapse_model': 'static_synapse'}
syn_1 = {'synapse_model': 'syn_1_model'}
nest.Connect(input, layer_0, conn_neur, syn_0)
nest.Connect(input, layer_1, conn_neur, syn_1)
nest.DumpLayerConnections(input, layer_0, 'static_synapse', 'conn.txt')
nest.DumpLayerConnections(input, layer_1, 'syn_1_model', 'conn.txt')
for neuron in layer_0:
conn = nest.GetConnections(target=neuron, synapse_model='static_synapse')
w = np.array(conn.weight)
if (w.size>1):
w_normed = w / sum(abs(w)) # L1-norm
w_nparray = 2. * w_normed
conn.weight = w_nparray.tolist()
nest.Simulate(100)
_______________________________________________
NEST Users mailing list -- users(a)nest-simulator.org
To unsubscribe send an email to users-leave(a)nest-simulator.org