Hi Tom,
As a DIY workaround, you can use the RunManager context to simulate in small steps and break if it gets too slow. I haven't tested the code, just sketching from memory. Instead of calling nest.Simulate(1000), use
with nest.RunManager(): for _ in range(100): t = time.time() nest.Run(10) if time.time() - t > 5: break
The logic is as follows: You split the 1000 ms into 100 times 10 ms. This is fast with Run() within the RunManager(). You then use Python's time to see how long it takes to simulate 10 ms and break if it takes too long, here a 5 s limit. You can then use GetKernelStatus to get the current time in the simulation.
It would be interesting to add this as a kernel feature. Let me know if it works!
Best, Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.nomailto:hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
On 06/05/2021, 16:08, "TOM BUGNON" <bugnon@wisc.edumailto:bugnon@wisc.edu> wrote:
Hi all, Under some circumstances simulations can slow down up to the point where the nest.Simulate() does not advance anymore and stays stuck at a given virtual time, with a "realtime factor" of 0. I suppose this can happen for instance when a network falls into a regime of runaway excitation in which a massive number of spikes are being exchanged.
I'm looking for a way to stop the simulation in such a case, (say when the realtime factor goes below a set threshold, or when the output files are not updated for a certain duration), ideally in such a way that the program can continue running rather than crashing. If anyone has a suggestion about how to go around this issue I'd be happy to hear it. Thanks in advance! Best, Tom
Hi Hans,
Thanks for the tip! I ended up doing something slightly different, since in my case the simulation can go from regular speed to infinitely slow pretty fast (so that the last 10ms Run call before freezing never finishes).
In case it's useful to someone:
* I run in 10ms steps * If the simulation starts slowing such that one of the steps takes longer than 2s, I run the next 10ms in "ministeps" (equal to the kernel resolution) * if 3 consecutive "ministeps" last longer than 1s I raise an error
Below is the corresponding code, Cheers!
``` class HangingSimulationError(Exception): pass
def nest_run(sim_time): import nest, time t = time.time() nest.Run(sim_time) return time.time() - t
def simulate_in_steps(simulation_time): """Simulate in steps and catch hanging simulations.""" import nest step = 10 # (virtual time) ministep = nest.GetKernelStatus('resolution') N_break = 3 split_t = 2 # (real time) (s) Run next chunk in mini steps if previous took longer break_t = 1 # (real time) (s) break if `N_break` consecutive mini-steps takes longer total_sim_time = 0 kernel_time = nest.GetKernelStatus('time') with nest.RunManager(): while total_sim_time < simulation_time: next_step_t = min(step, simulation_time - total_sim_time) step_real_t = nest_run(next_step_t) total_sim_time += next_step_t if step_real_t > split_t: # Run next {step}ms in ministeps # Break if 3 consecutive ministeps last too long count = 0 print(f"Running next {step}ms in ministeps") for _ in range(int(step/ministep)): if simulation_time == total_sim_time: break next_ministep_t = min(ministep, simulation_time - total_sim_time) t = nest_run(next_ministep_t) total_sim_time += next_ministep_t if t > break_t: count += 1 else: count = 0 if count >= N_break: raise HangingSimulationError( f"Simulation froze at {nest.GetKernelStatus('time')}ms" ) assert nest.GetKernelStatus('time') - kernel_time == simulation_time ``` ________________________________ From: Hans Ekkehard Plesser hans.ekkehard.plesser@nmbu.no Sent: Thursday, May 6, 2021 9:49 AM To: NEST User Mailing List users@nest-simulator.org Subject: [NEST Users] Re: Stop hanging simulations
Hi Tom,
As a DIY workaround, you can use the RunManager context to simulate in small steps and break if it gets too slow. I haven't tested the code, just sketching from memory. Instead of calling nest.Simulate(1000), use
with nest.RunManager():
for _ in range(100):
t = time.time()
nest.Run(10)
if time.time() - t > 5:
break
The logic is as follows: You split the 1000 ms into 100 times 10 ms. This is fast with Run() within the RunManager(). You then use Python's time to see how long it takes to simulate 10 ms and break if it takes too long, here a 5 s limit. You can then use GetKernelStatus to get the current time in the simulation.
It would be interesting to add this as a kernel feature. Let me know if it works!
Best,
Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser
Head, Department of Data Science
Faculty of Science and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560
Email hans.ekkehard.plesser@nmbu.nomailto:hans.ekkehard.plesser@nmbu.no
Home http://arken.nmbu.no/~plesser
On 06/05/2021, 16:08, "TOM BUGNON" <bugnon@wisc.edumailto:bugnon@wisc.edu> wrote:
Hi all,
Under some circumstances simulations can slow down up to the point where the nest.Simulate() does not advance anymore and stays stuck at a given virtual time, with a "realtime factor" of 0. I suppose this can happen for instance when a network falls into a regime of runaway excitation in which a massive number of spikes are being exchanged.
I'm looking for a way to stop the simulation in such a case, (say when the realtime factor goes below a set threshold, or when the output files are not updated for a certain duration), ideally in such a way that the program can continue running rather than crashing. If anyone has a suggestion about how to go around this issue I'd be happy to hear it. Thanks in advance! Best, Tom
Hi Tom,
I now created a NEST issue to put in place a proper stopper in case things become too slow: https://github.com/nest/nest-simulator/issues/2033 .
BTW, I assume that you are using the ht_neuron? That might slow done quite a lot because the of the adaptive stepsize solvers for the highly non-linear membrane currents.
Best, Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science
Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.nomailto:hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser
On 06/05/2021, 19:19, "TOM BUGNON" <bugnon@wisc.edumailto:bugnon@wisc.edu> wrote:
Hi Hans,
Thanks for the tip! I ended up doing something slightly different, since in my case the simulation can go from regular speed to infinitely slow pretty fast (so that the last 10ms Run call before freezing never finishes).
In case it's useful to someone: · I run in 10ms steps · If the simulation starts slowing such that one of the steps takes longer than 2s, I run the next 10ms in "ministeps" (equal to the kernel resolution) · if 3 consecutive "ministeps" last longer than 1s I raise an error Below is the corresponding code, Cheers!
```
class HangingSimulationError(Exception): pass
def nest_run(sim_time): import nest, time t = time.time() nest.Run(sim_time) return time.time() - t
def simulate_in_steps(simulation_time): """Simulate in steps and catch hanging simulations.""" import nest step = 10 # (virtual time) ministep = nest.GetKernelStatus('resolution') N_break = 3 split_t = 2 # (real time) (s) Run next chunk in mini steps if previous took longer break_t = 1 # (real time) (s) break if `N_break` consecutive mini-steps takes longer total_sim_time = 0 kernel_time = nest.GetKernelStatus('time') with nest.RunManager(): while total_sim_time < simulation_time: next_step_t = min(step, simulation_time - total_sim_time) step_real_t = nest_run(next_step_t) total_sim_time += next_step_t if step_real_t > split_t: # Run next {step}ms in ministeps # Break if 3 consecutive ministeps last too long count = 0 print(f"Running next {step}ms in ministeps") for _ in range(int(step/ministep)): if simulation_time == total_sim_time: break next_ministep_t = min(ministep, simulation_time - total_sim_time) t = nest_run(next_ministep_t) total_sim_time += next_ministep_t if t > break_t: count += 1 else: count = 0 if count >= N_break: raise HangingSimulationError( f"Simulation froze at {nest.GetKernelStatus('time')}ms" ) assert nest.GetKernelStatus('time') - kernel_time == simulation_time ``` ________________________________ From: Hans Ekkehard Plesser hans.ekkehard.plesser@nmbu.no Sent: Thursday, May 6, 2021 9:49 AM To: NEST User Mailing List users@nest-simulator.org Subject: [NEST Users] Re: Stop hanging simulations
Hi Tom,
As a DIY workaround, you can use the RunManager context to simulate in small steps and break if it gets too slow. I haven't tested the code, just sketching from memory. Instead of calling nest.Simulate(1000), use
with nest.RunManager():
for _ in range(100):
t = time.time()
nest.Run(10)
if time.time() - t > 5:
break
The logic is as follows: You split the 1000 ms into 100 times 10 ms. This is fast with Run() within the RunManager(). You then use Python's time to see how long it takes to simulate 10 ms and break if it takes too long, here a 5 s limit. You can then use GetKernelStatus to get the current time in the simulation.
It would be interesting to add this as a kernel feature. Let me know if it works!
Best,
Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser
Head, Department of Data Science
Faculty of Science and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560
Email hans.ekkehard.plesser@nmbu.nomailto:hans.ekkehard.plesser@nmbu.no
Home http://arken.nmbu.no/~plesser
On 06/05/2021, 16:08, "TOM BUGNON" <bugnon@wisc.edumailto:bugnon@wisc.edu> wrote:
Hi all,
Under some circumstances simulations can slow down up to the point where the nest.Simulate() does not advance anymore and stays stuck at a given virtual time, with a "realtime factor" of 0. I suppose this can happen for instance when a network falls into a regime of runaway excitation in which a massive number of spikes are being exchanged.
I'm looking for a way to stop the simulation in such a case, (say when the realtime factor goes below a set threshold, or when the output files are not updated for a certain duration), ideally in such a way that the program can continue running rather than crashing. If anyone has a suggestion about how to go around this issue I'd be happy to hear it. Thanks in advance! Best, Tom
-
Hans Ekkehard Plesser -
TOM BUGNON