local_num_threads, multiprocessing and topology.ConnectLayers - NEST Users

28 Apr 2020

Hi all,

I'm trying to understand some inner workings of Nest. Rigth now I'm running
simulations with close half millons elements, using mpirun in a cluster
with 25 nodes. The problem I am having is that the "setup" (layer creation
and connections) phase takes close to 8min and the simulation only takes
1min.

So I tried to use python's multiprocessing package to speed it up, with the
following code:

nest.ResetKernel()
nest.SetKernelStatus({"local_num_threads": 1})

#...

connections = [
    (layer1, layer1, conn_ee_dict, 1),
    (layer1, layer2, conn_ee_dict, 2),
    (layer2, layer2, conn_ee_dict, 3),
    (layer2, layer1, conn_ee_dict, 4)
]

# Process the connections.
def parallel_topology_connect(parameters):
    [pre, post, projection, number] = parameters
    print(f"Connection number: {number}")
    topology.ConnectLayers(pre, post, projection)

pool = multiprocessing.Pool(processes=4)
pool.map(parallel_topology_connect, connections)

The above example takes around 0.9s, but if the last two to lines are
changed for a sequential call, it takes 2.1s:

for [pre, post, projection, number] in connections:
    print(f"Connection number: {number}")
    topology.ConnectLayers(pre, post, projection)

So far the multiprocessing works great, the problem comes when the
"local_num_threads" parameters is changed from 1 to 2 or more, in the
cluster it could be 32. The code gets stuck in the topology.Connect without
any error, after a while I just stopped it.

Also I realised that the tolopoly.ConnectLayers just spawn one thread to
connects layers despite the local_num_threads is setted more than one.

Any idea what is going on?

Thanks in advance
Juan Manuel

PD: The full example code is attached (60 lines of code). The
local_num_threads and multiprocessing_flag variables change the behaviors
of the code.