Hi all,
I'm trying to understand some inner workings of Nest. Rigth now I'm running simulations with close half millons elements, using mpirun in a cluster with 25 nodes. The problem I am having is that the "setup" (layer creation and connections) phase takes close to 8min and the simulation only takes 1min.
So I tried to use python's multiprocessing package to speed it up, with the following code:
nest.ResetKernel() nest.SetKernelStatus({"local_num_threads": 1})
#...
connections = [ (layer1, layer1, conn_ee_dict, 1), (layer1, layer2, conn_ee_dict, 2), (layer2, layer2, conn_ee_dict, 3), (layer2, layer1, conn_ee_dict, 4) ]
# Process the connections. def parallel_topology_connect(parameters): [pre, post, projection, number] = parameters print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
pool = multiprocessing.Pool(processes=4) pool.map(parallel_topology_connect, connections)
The above example takes around 0.9s, but if the last two to lines are changed for a sequential call, it takes 2.1s:
for [pre, post, projection, number] in connections: print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
So far the multiprocessing works great, the problem comes when the "local_num_threads" parameters is changed from 1 to 2 or more, in the cluster it could be 32. The code gets stuck in the topology.Connect without any error, after a while I just stopped it.
Also I realised that the tolopoly.ConnectLayers just spawn one thread to connects layers despite the local_num_threads is setted more than one.
Any idea what is going on?
Thanks in advance Juan Manuel
PD: The full example code is attached (60 lines of code). The local_num_threads and multiprocessing_flag variables change the behaviors of the code.
Hi Juan,
Creating connections in complex networks can take time. Sometimes, it is possible to improve on connection times by tweaks to the way in which the network is constructed. Given that you have a quite large network, I assume you have a considerable number of layers and thus also a quite large number of calls to ConnectLayers(). In that case, the forthcoming NEST 3 will most likely reduce construction times noticeably, because layers passed to Connect in a much more efficient way. We currently also have not fully thread-parallelised connection construction for "divergent" connections, in contrast to "convergent". We could look into that if switching between "convergent" and "divergent" gives you noticeable improvements in speed.
Please DO NOT USE MULTIPROCESSING with NEST. NEST internally parallelizes network construction and maintains internal data structures in this process. Running several ConnectLayers() calls simultaneously will lead to unpredictable results.
Best, Hans Ekkehard
On 28 Apr 2020, at 20:27, Juan Manuel Vicente <juanma.v82@gmail.commailto:juanma.v82@gmail.com> wrote:
Hi all,
I'm trying to understand some inner workings of Nest. Rigth now I'm running simulations with close half millons elements, using mpirun in a cluster with 25 nodes. The problem I am having is that the "setup" (layer creation and connections) phase takes close to 8min and the simulation only takes 1min.
So I tried to use python's multiprocessing package to speed it up, with the following code:
nest.ResetKernel() nest.SetKernelStatus({"local_num_threads": 1})
#...
connections = [ (layer1, layer1, conn_ee_dict, 1), (layer1, layer2, conn_ee_dict, 2), (layer2, layer2, conn_ee_dict, 3), (layer2, layer1, conn_ee_dict, 4) ]
# Process the connections. def parallel_topology_connect(parameters): [pre, post, projection, number] = parameters print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
pool = multiprocessing.Pool(processes=4) pool.map(parallel_topology_connect, connections)
The above example takes around 0.9s, but if the last two to lines are changed for a sequential call, it takes 2.1s:
for [pre, post, projection, number] in connections: print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
So far the multiprocessing works great, the problem comes when the "local_num_threads" parameters is changed from 1 to 2 or more, in the cluster it could be 32. The code gets stuck in the topology.Connect without any error, after a while I just stopped it.
Also I realised that the tolopoly.ConnectLayers just spawn one thread to connects layers despite the local_num_threads is setted more than one.
Any idea what is going on?
Thanks in advance Juan Manuel
PD: The full example code is attached (60 lines of code). The local_num_threads and multiprocessing_flag variables change the behaviors of the code. <smalltestcase.py>_______________________________________________ NEST Users mailing list -- users@nest-simulator.orgmailto:users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.orgmailto:users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
Hans,
In what scenarios, Python's multi-processing API is okay to use in NEST? I am still using SLI as an interface to NEST, so I am just curious.
Itaru.
On Wed, Apr 29, 2020 at 2:32 PM Hans Ekkehard Plesser < hans.ekkehard.plesser@nmbu.no> wrote:
Hi Juan,
Creating connections in complex networks can take time. Sometimes, it is possible to improve on connection times by tweaks to the way in which the network is constructed. Given that you have a quite large network, I assume you have a considerable number of layers and thus also a quite large number of calls to ConnectLayers(). In that case, the forthcoming NEST 3 will most likely reduce construction times noticeably, because layers passed to Connect in a much more efficient way. We currently also have not fully thread-parallelised connection construction for "divergent" connections, in contrast to "convergent". We could look into that if switching between "convergent" and "divergent" gives you noticeable improvements in speed.
Please DO NOT USE MULTIPROCESSING with NEST. NEST internally parallelizes network construction and maintains internal data structures in this process. Running several ConnectLayers() calls simultaneously will lead to unpredictable results.
Best, Hans Ekkehard
On 28 Apr 2020, at 20:27, Juan Manuel Vicente juanma.v82@gmail.com wrote:
Hi all,
I'm trying to understand some inner workings of Nest. Rigth now I'm running simulations with close half millons elements, using mpirun in a cluster with 25 nodes. The problem I am having is that the "setup" (layer creation and connections) phase takes close to 8min and the simulation only takes 1min.
So I tried to use python's multiprocessing package to speed it up, with the following code:
nest.ResetKernel() nest.SetKernelStatus({"local_num_threads": 1})
#...
connections = [ (layer1, layer1, conn_ee_dict, 1), (layer1, layer2, conn_ee_dict, 2), (layer2, layer2, conn_ee_dict, 3), (layer2, layer1, conn_ee_dict, 4) ]
# Process the connections. def parallel_topology_connect(parameters): [pre, post, projection, number] = parameters print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
pool = multiprocessing.Pool(processes=4) pool.map(parallel_topology_connect, connections)
The above example takes around 0.9s, but if the last two to lines are changed for a sequential call, it takes 2.1s:
for [pre, post, projection, number] in connections: print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
So far the multiprocessing works great, the problem comes when the "local_num_threads" parameters is changed from 1 to 2 or more, in the cluster it could be 32. The code gets stuck in the topology.Connect without any error, after a while I just stopped it.
Also I realised that the tolopoly.ConnectLayers just spawn one thread to connects layers despite the local_num_threads is setted more than one.
Any idea what is going on?
Thanks in advance Juan Manuel
PD: The full example code is attached (60 lines of code). The local_num_threads and multiprocessing_flag variables change the behaviors of the code. <smalltestcase.py>_______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
Itaru,
Until we do a very careful assessment about interactions between NEST and multiprocessing, I would say NEVER.
We have a few test cases and maybe also examples which use mpi4py, but only to collect results from MPI-parallel simulations for analysis. When running NEST/PyNEST with MPI, it is also essential that all MPI ranks make *identical* calls to nest for NEST to work correctly.
Best, Hans Ekkehard
On 29 Apr 2020, at 07:46, Itaru Kitayama <itaru.kitayama@gmail.commailto:itaru.kitayama@gmail.com> wrote:
Hans,
In what scenarios, Python's multi-processing API is okay to use in NEST? I am still using SLI as an interface to NEST, so I am just curious.
Itaru.
On Wed, Apr 29, 2020 at 2:32 PM Hans Ekkehard Plesser <hans.ekkehard.plesser@nmbu.nomailto:hans.ekkehard.plesser@nmbu.no> wrote:
Hi Juan,
Creating connections in complex networks can take time. Sometimes, it is possible to improve on connection times by tweaks to the way in which the network is constructed. Given that you have a quite large network, I assume you have a considerable number of layers and thus also a quite large number of calls to ConnectLayers(). In that case, the forthcoming NEST 3 will most likely reduce construction times noticeably, because layers passed to Connect in a much more efficient way. We currently also have not fully thread-parallelised connection construction for "divergent" connections, in contrast to "convergent". We could look into that if switching between "convergent" and "divergent" gives you noticeable improvements in speed.
Please DO NOT USE MULTIPROCESSING with NEST. NEST internally parallelizes network construction and maintains internal data structures in this process. Running several ConnectLayers() calls simultaneously will lead to unpredictable results.
Best, Hans Ekkehard
On 28 Apr 2020, at 20:27, Juan Manuel Vicente <juanma.v82@gmail.commailto:juanma.v82@gmail.com> wrote:
Hi all,
I'm trying to understand some inner workings of Nest. Rigth now I'm running simulations with close half millons elements, using mpirun in a cluster with 25 nodes. The problem I am having is that the "setup" (layer creation and connections) phase takes close to 8min and the simulation only takes 1min.
So I tried to use python's multiprocessing package to speed it up, with the following code:
nest.ResetKernel() nest.SetKernelStatus({"local_num_threads": 1})
#...
connections = [ (layer1, layer1, conn_ee_dict, 1), (layer1, layer2, conn_ee_dict, 2), (layer2, layer2, conn_ee_dict, 3), (layer2, layer1, conn_ee_dict, 4) ]
# Process the connections. def parallel_topology_connect(parameters): [pre, post, projection, number] = parameters print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
pool = multiprocessing.Pool(processes=4) pool.map(parallel_topology_connect, connections)
The above example takes around 0.9s, but if the last two to lines are changed for a sequential call, it takes 2.1s:
for [pre, post, projection, number] in connections: print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
So far the multiprocessing works great, the problem comes when the "local_num_threads" parameters is changed from 1 to 2 or more, in the cluster it could be 32. The code gets stuck in the topology.Connect without any error, after a while I just stopped it.
Also I realised that the tolopoly.ConnectLayers just spawn one thread to connects layers despite the local_num_threads is setted more than one.
Any idea what is going on?
Thanks in advance Juan Manuel
PD: The full example code is attached (60 lines of code). The local_num_threads and multiprocessing_flag variables change the behaviors of the code. <smalltestcase.py>_______________________________________________ NEST Users mailing list -- users@nest-simulator.orgmailto:users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.orgmailto:users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.nohttp://nmbu.no/ Home http://arken.nmbu.no/~plesser
_______________________________________________ NEST Users mailing list -- users@nest-simulator.orgmailto:users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.orgmailto:users-leave@nest-simulator.org _______________________________________________ NEST Users mailing list -- users@nest-simulator.orgmailto:users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.orgmailto:users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
Then, this should be documented clearly for the Python interface users, otherwise, people tend to fall in this trap.
Itaru.
On Wed, Apr 29, 2020 at 3:00 PM Hans Ekkehard Plesser < hans.ekkehard.plesser@nmbu.no> wrote:
Itaru,
Until we do a very careful assessment about interactions between NEST and multiprocessing, I would say NEVER.
We have a few test cases and maybe also examples which use mpi4py, but only to collect results from MPI-parallel simulations for analysis. When running NEST/PyNEST with MPI, it is also essential that all MPI ranks make *identical* calls to nest for NEST to work correctly.
Best, Hans Ekkehard
On 29 Apr 2020, at 07:46, Itaru Kitayama itaru.kitayama@gmail.com wrote:
Hans,
In what scenarios, Python's multi-processing API is okay to use in NEST? I am still using SLI as an interface to NEST, so I am just curious.
Itaru.
On Wed, Apr 29, 2020 at 2:32 PM Hans Ekkehard Plesser < hans.ekkehard.plesser@nmbu.no> wrote:
Hi Juan,
Creating connections in complex networks can take time. Sometimes, it is possible to improve on connection times by tweaks to the way in which the network is constructed. Given that you have a quite large network, I assume you have a considerable number of layers and thus also a quite large number of calls to ConnectLayers(). In that case, the forthcoming NEST 3 will most likely reduce construction times noticeably, because layers passed to Connect in a much more efficient way. We currently also have not fully thread-parallelised connection construction for "divergent" connections, in contrast to "convergent". We could look into that if switching between "convergent" and "divergent" gives you noticeable improvements in speed.
Please DO NOT USE MULTIPROCESSING with NEST. NEST internally parallelizes network construction and maintains internal data structures in this process. Running several ConnectLayers() calls simultaneously will lead to unpredictable results.
Best, Hans Ekkehard
On 28 Apr 2020, at 20:27, Juan Manuel Vicente juanma.v82@gmail.com wrote:
Hi all,
I'm trying to understand some inner workings of Nest. Rigth now I'm running simulations with close half millons elements, using mpirun in a cluster with 25 nodes. The problem I am having is that the "setup" (layer creation and connections) phase takes close to 8min and the simulation only takes 1min.
So I tried to use python's multiprocessing package to speed it up, with the following code:
nest.ResetKernel() nest.SetKernelStatus({"local_num_threads": 1})
#...
connections = [ (layer1, layer1, conn_ee_dict, 1), (layer1, layer2, conn_ee_dict, 2), (layer2, layer2, conn_ee_dict, 3), (layer2, layer1, conn_ee_dict, 4) ]
# Process the connections. def parallel_topology_connect(parameters): [pre, post, projection, number] = parameters print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
pool = multiprocessing.Pool(processes=4) pool.map(parallel_topology_connect, connections)
The above example takes around 0.9s, but if the last two to lines are changed for a sequential call, it takes 2.1s:
for [pre, post, projection, number] in connections: print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
So far the multiprocessing works great, the problem comes when the "local_num_threads" parameters is changed from 1 to 2 or more, in the cluster it could be 32. The code gets stuck in the topology.Connect without any error, after a while I just stopped it.
Also I realised that the tolopoly.ConnectLayers just spawn one thread to connects layers despite the local_num_threads is setted more than one.
Any idea what is going on?
Thanks in advance Juan Manuel
PD: The full example code is attached (60 lines of code). The local_num_threads and multiprocessing_flag variables change the behaviors of the code. <smalltestcase.py>_______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
Hi Itaru, Hi Juan,
you are both right, this should be made more clear. I created issue #1581 to track the documentation side.
Best, Dennis
On 29.04.20 08:01, Itaru Kitayama wrote:
Then, this should be documented clearly for the Python interface users, otherwise, people tend to fall in this trap.
Itaru.
On Wed, Apr 29, 2020 at 3:00 PM Hans Ekkehard Plesser < hans.ekkehard.plesser@nmbu.no> wrote:
Itaru,
Until we do a very careful assessment about interactions between NEST and multiprocessing, I would say NEVER.
We have a few test cases and maybe also examples which use mpi4py, but only to collect results from MPI-parallel simulations for analysis. When running NEST/PyNEST with MPI, it is also essential that all MPI ranks make *identical* calls to nest for NEST to work correctly.
Best, Hans Ekkehard
On 29 Apr 2020, at 07:46, Itaru Kitayama itaru.kitayama@gmail.com wrote:
Hans,
In what scenarios, Python's multi-processing API is okay to use in NEST? I am still using SLI as an interface to NEST, so I am just curious.
Itaru.
On Wed, Apr 29, 2020 at 2:32 PM Hans Ekkehard Plesser < hans.ekkehard.plesser@nmbu.no> wrote:
Hi Juan,
Creating connections in complex networks can take time. Sometimes, it is possible to improve on connection times by tweaks to the way in which the network is constructed. Given that you have a quite large network, I assume you have a considerable number of layers and thus also a quite large number of calls to ConnectLayers(). In that case, the forthcoming NEST 3 will most likely reduce construction times noticeably, because layers passed to Connect in a much more efficient way. We currently also have not fully thread-parallelised connection construction for "divergent" connections, in contrast to "convergent". We could look into that if switching between "convergent" and "divergent" gives you noticeable improvements in speed.
Please DO NOT USE MULTIPROCESSING with NEST. NEST internally parallelizes network construction and maintains internal data structures in this process. Running several ConnectLayers() calls simultaneously will lead to unpredictable results.
Best, Hans Ekkehard
On 28 Apr 2020, at 20:27, Juan Manuel Vicente juanma.v82@gmail.com wrote:
Hi all,
I'm trying to understand some inner workings of Nest. Rigth now I'm running simulations with close half millons elements, using mpirun in a cluster with 25 nodes. The problem I am having is that the "setup" (layer creation and connections) phase takes close to 8min and the simulation only takes 1min.
So I tried to use python's multiprocessing package to speed it up, with the following code:
nest.ResetKernel() nest.SetKernelStatus({"local_num_threads": 1})
#...
connections = [ (layer1, layer1, conn_ee_dict, 1), (layer1, layer2, conn_ee_dict, 2), (layer2, layer2, conn_ee_dict, 3), (layer2, layer1, conn_ee_dict, 4) ]
# Process the connections. def parallel_topology_connect(parameters): [pre, post, projection, number] = parameters print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
pool = multiprocessing.Pool(processes=4) pool.map(parallel_topology_connect, connections)
The above example takes around 0.9s, but if the last two to lines are changed for a sequential call, it takes 2.1s:
for [pre, post, projection, number] in connections: print(f"Connection number: {number}") topology.ConnectLayers(pre, post, projection)
So far the multiprocessing works great, the problem comes when the "local_num_threads" parameters is changed from 1 to 2 or more, in the cluster it could be 32. The code gets stuck in the topology.Connect without any error, after a while I just stopped it.
Also I realised that the tolopoly.ConnectLayers just spawn one thread to connects layers despite the local_num_threads is setted more than one.
Any idea what is going on?
Thanks in advance Juan Manuel
PD: The full example code is attached (60 lines of code). The local_num_threads and multiprocessing_flag variables change the behaviors of the code. <smalltestcase.py>_______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
--
Prof. Dr. Hans Ekkehard Plesser Head, Data Science Section
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-
Dennis Terhorst -
Hans Ekkehard Plesser -
Itaru Kitayama -
Juan Manuel Vicente