Hi Allen,
It would also be interesting if you could take a look at how memory consumption grows while the simulation is running (using top or similar).
Are you recording spike or membrane potential data to memory during the simulation? With you very long-running simulations, that could flood memory
over time. A quick back-of-the envelope calculation:
648 neurons x 1 membrane potential per ms x 300000 ms x 24B / values (V_m, sender, time) = 4.3 GB
If you do record data to memory during the entire simulation, try running without recording. If problem still occur, data may be collecting elsewhere
and we would need to check further. Have you tried a comparable simulation without MUSIC?
Best,
Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser
Head, Department of Data Science
Faculty of Science and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560
On 24/11/2020, 10:10, "Trensch, Guido" <g.trensch@fz-juelich.de> wrote:
Dear Allen,
a “bad_alloc” exception does not indicate a disk space problem, but rather refers to a memory shortage. Please try to increase the base memory of your VM.
Best regards,
Guido
From: Allen Rabayda [mailto:arabayda@masonlive.gmu.edu]
Sent: Monday, 23 November 2020 17:24
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] error std::bad_alloc: virtualbox
Dear Nest Community,
Has anyone encountered a "bad_alloc" error like the one below and if so, any recommendations? It appears to be a VM memory issue but only using 21% of harddrive space (ref: below).
My simulation successfully completes for 200,000 ms but errors out at 98% complete for 230,000 ms, 75% for 300,000 ms and 56% for 400,000 ms.
I'm running on NEST 2.18.0 VirtualBox lubuntu 18.04 (ref: image of settings below).
Thank you for any suggestions.
Best Regards,
--Allen
**********************************************
**** Error Message *****
>> # SIMULATION
>> nest.Simulate(300000)
Nov 21 17:10:28 NodeManager::prepare_nodes [Info]:
Preparing 684 nodes for simulation.
Nov 21 17:10:28 MUSICManager::enter_runtime [Info]:
Entering MUSIC runtime with tick = 1 ms
Nov 21 17:10:28 SimulationManager::start_updating_ [Info]:
Number of local nodes: 684
Simulation time (ms): 300000
Number of OpenMP threads: 2
Number of MPI processes: 1
75 %: network time: 223698.0 ms, realtime factor: 0.6277Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 246, in stack_checker_func
return f(*args, **kwargs)
File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/lib/hl_api_simulation.py", line 66, in Simulate
sr('ms Simulate')
File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 132, in catching_sli_run
raise exceptionCls(commandname, message)
nest.ll_api.std::bad_alloc: ('std::bad_alloc in Simulate_d: C++ exception: std::bad_alloc', 'std::bad_alloc', <SLILiteral: Simulate_d>, ': C++ exception: std::bad_alloc')
********************************************
**** Folder Space on VirtualBox after Error ****
nest@nestvm:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 5.2G 0 5.2G 0% /dev
tmpfs 1.1G 1.1M 1.1G 1% /run
/dev/sda1 99G 20G 76G 21% /
tmpfs 5.2G 0 5.2G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 5.2G 0 5.2G 0% /sys/fs/cgroup
SharedNest2 917G 447G 470G 49% /media/sf_SharedNest2
tmpfs 1.1G 16K 1.1G 1% /run/user/1000
/dev/sr0 74M 74M 0 100% /media/nest/VBox_GAs_6.0.10
***********************************
**** VirtualBox Settings *******
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------