Why suddenly my nodeos memory(ram with swap) usage been 100%?
and memory full usage then oop
Am I setting chain-thread to 18?
My Nodeos Ver:
- v1.8.0-rc2
Sync status
- completed full sync
nodeos config.ini:
# Maximum size (in MiB) of the chain state database (eosio::chain_plugin)
chain-state-db-size-mb = 45484
# print contract's output to console (eosio::chain_plugin)
contracts-console = false
# Database read mode ("speculative", "head", or "read-only").
# In "speculative" mode database contains changes done up to the head block plus changes made by transactions not yet included to the blockchain.
# In "head" mode database contains changes done up to the current head block.
# In "read-only" mode database contains incoming block changes but no speculative transaction processing.
# (eosio::chain_plugin)
read-mode = irrerversible
# Chain validation mode ("full" or "light").
# In "full" mode all incoming blocks will be fully validated.
# In "light" mode all incoming blocks headers will be fully validated; transactions in those validated blocks will be trusted
# (eosio::chain_plugin)
validation-mode = light
# Disable the check which subjectively fails a transaction if a contract bills more RAM to another account within the context of a notification handler (i.e. when the receiver is not the code of the action). (eosio::chain_plugin)
disable-ram-billing-notify-checks = false
# true: validate that the peer certificates are valid and trusted, false: ignore cert errors (eosio::http_client_plugin)
https-client-validate-peers = 1
# The local IP and port to listen for incoming http connections; set blank to disable. (eosio::http_plugin)
http-server-address = 0.0.0.0:8888
# Specify the Access-Control-Allow-Origin to be returned on each request. (eosio::http_plugin)
access-control-allow-origin = *
# Specify if Access-Control-Allow-Credentials: true should be returned on each request. (eosio::http_plugin)
access-control-allow-credentials = false
# The actual host:port used to listen for incoming p2p connections. (eosio::net_plugin)
p2p-listen-endpoint = 0.0.0.0:9876
# An externally accessible host:port for identifying this node. Defaults to p2p-listen-endpoint. (eosio::net_plugin)
p2p-server-address = 0.0.0.0:9876
# Maximum number of client nodes from any single IP address (eosio::net_plugin)
p2p-max-nodes-per-host = 5
# Can be 'any' or 'producers' or 'specified' or 'none'. If 'specified', peer-key must be specified at least once. If only 'producers', peer-key is not required. 'producers' and 'specified' may be combined. (eosio::net_plugin)
allowed-connection = any
chain-threads = 18 #defaults 2
http-threads = 2 #defaults 2
net-threads = 3 #defaults 1
producer-threads = 2 #defaults 2
txn-test-gen-threads = 0 #defaults 2
# Plugin(s) to enable, may be specified multiple times
plugin = eosio::chain_api_plugin
plugin = eosio::producer_api_plugin
plugin = eosio::state_history_plugin
plugin = eosio::db_size_api_plugin
plugin = eosio::net_plugin
plugin = eosio::http_plugin
plugin = eosio::net_api_plugin
My Server Spec:
CPU:
...
processor : 39
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb00002a
cpu MHz : 2200.061
cache size : 25600 KB
...
Mem:
MemTotal: 65684628 kB
DB Size API:
$curl --request POST --url http://127.0.0.1:8888/v1/db_size/get
...
"free_bytes": "40359701296",
"used_bytes": "7333728464",
"size": "47693429760",
...
dmesg:
[431854.056689] Bits 55-60 of /proc/PID/pagemap entries are about to stop being page-shift some time soon. See the linux/Documentation/vm/pagemap.txt for details.
[477202.342504] systemd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0
[477202.342508] systemd cpuset=/ mems_allowed=0-1
[477202.342510] CPU: 4 PID: 1 Comm: systemd Not tainted 3.10.0-957.21.3.el7.x86_64 #1
[477202.342511] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.7.1 001/22/2018
[477202.342512] Call Trace:
[477202.342521] [<ffffffffb5f63107>] dump_stack+0x19/0x1b
[477202.342524] [<ffffffffb5f5db2a>] dump_header+0x90/0x229
[477202.342529] [<ffffffffb5901292>] ? ktime_get_ts64+0x52/0xf0
[477202.342532] [<ffffffffb59584df>] ? delayacct_end+0x8f/0xb0
[477202.342536] [<ffffffffb59ba834>] oom_kill_process+0x254/0x3d0
[477202.342538] [<ffffffffb59ba2dd>] ? oom_unkillable_task+0xcd/0x120
[477202.342540] [<ffffffffb59ba386>] ? find_lock_task_mm+0x56/0xc0
[477202.342542] [<ffffffffb59bb076>] out_of_memory+0x4b6/0x4f0
[477202.342544] [<ffffffffb5f5e62e>] __alloc_pages_slowpath+0x5d6/0x724
[477202.342547] [<ffffffffb59c1454>] __alloc_pages_nodemask+0x404/0x420
[477202.342552] [<ffffffffb5894dfd>] copy_process+0x1dd/0x1a40
[477202.342554] [<ffffffffb5896811>] do_fork+0x91/0x320
[477202.342556] [<ffffffffb5896b26>] SyS_clone+0x16/0x20
[477202.342560] [<ffffffffb5f761b4>] stub_clone+0x44/0x70
[477202.342562] [<ffffffffb5f75ddb>] ? system_call_fastpath+0x22/0x27
[477202.342563] Mem-Info:
[477202.342569] active_anon:14400252 inactive_anon:1385366 isolated_anon:0
active_file:0 inactive_file:0 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:75895 slab_unreclaimable:32395
mapped:0 shmem:12 pagetables:37529 bounce:0
free:43038 free_pcp:1267 free_cma:0
[477202.342573] Node 0 DMA free:15896kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[477202.342577] lowmem_reserve[]: 0 1710 31888 31888
[477202.342580] Node 0 DMA32 free:121152kB min:864kB low:1080kB high:1296kB active_anon:1103028kB inactive_anon:368332kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1751760kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:48008kB slab_unreclaimable:8748kB kernel_stack:384kB pagetables:2504kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[477202.342584] lowmem_reserve[]: 0 0 30177 30177
[477202.342586] Node 0 Normal free:15232kB min:15248kB low:19060kB high:22872kB active_anon:27436164kB inactive_anon:1714808kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB managed:30902132kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:12kB slab_reclaimable:159428kB slab_unreclaimable:60084kB kernel_stack:6256kB pagetables:60040kB unstable:0kB bounce:0kB free_pcp:1560kB local_pcp:28kB free_cma:0kB writeback_tmp:0kB pages_scanned:151 all_unreclaimable? yes
[477202.342590] lowmem_reserve[]: 0 0 0 0
[477202.342592] Node 1 Normal free:19872kB min:16292kB low:20364kB high:24436kB active_anon:29061816kB inactive_anon:3458324kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33554432kB managed:33014832kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:36kB slab_reclaimable:96144kB slab_unreclaimable:60748kB kernel_stack:5008kB pagetables:87572kB unstable:0kB bounce:0kB free_pcp:3496kB local_pcp:76kB free_cma:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? yes
[477202.342595] lowmem_reserve[]: 0 0 0 0
[477202.342597] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
[477202.342606] Node 0 DMA32: 3166*4kB (UEM) 1658*8kB (UEM) 558*16kB (UEM) 269*32kB (UEM) 194*64kB (UEM) 80*128kB (UEM) 29*256kB (UE) 13*512kB (UEM) 4*1024kB (UM) 4*2048kB (UE) 7*4096kB (M) = 121160kB
[477202.342615] Node 0 Normal: 2924*4kB (UEM) 441*8kB (UEM) 5*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15304kB
[477202.342621] Node 1 Normal: 4624*4kB (UEM) 143*8kB (UEM) 20*16kB (UM) 3*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20056kB
[477202.342629] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[477202.342630] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[477202.342631] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[477202.342632] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[477202.342633] 374757 total pagecache pages
[477202.342635] 374653 pages in swap cache
[477202.342636] Swap cache stats: add 5617285, delete 5242610, find 2003207/2521309
[477202.342637] Free swap = 0kB
[477202.342638] Total swap = 4194300kB
[477202.342639] 16753239 pages RAM
[477202.342640] 0 pages HighMem/MovableOnly
[477202.342641] 332084 pages reserved
[477202.342641] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[477202.342649] [ 5088] 0 5088 15929 39 38 47 0 systemd-journal
[477202.342651] [ 5113] 0 5113 31206 38 28 39 0 lvmetad
[477202.342653] [ 5119] 0 5119 11827 57 25 198 -1000 systemd-udevd
[477202.342655] [10769] 0 10769 28686 0 14 46 0 rsync
[477202.342657] [10776] 0 10776 5487 79 16 70 0 irqbalance
[477202.342659] [10777] 28 10777 198838 119 52 83 0 nscd
[477202.342661] [10779] 0 10779 6116 40 17 36 0 systemd-logind
[477202.342662] [10781] 81 10781 14107 77 32 73 -900 dbus-daemon
[477202.342664] [10823] 0 10823 6051 6 17 43 0 atd
[477202.342665] [11228] 0 11228 26114 1 53 234 -1000 sshd
[477202.342667] [11230] 0 11230 55063 180 64 1017 0 snmpd
[477202.342669] [11231] 0 11231 60534 80 25 170 0 rsyslogd
[477202.342671] [11233] 0 11233 6844 11 18 56 0 xinetd
[477202.342672] [11285] 0 11285 5837 8 15 98 0 ipmievd
[477202.342674] [11846] 0 11846 130807 172 42 1362 0 dsm_sa_datamgrd
[477202.342676] [11866] 0 11866 31093 30 20 121 0 crond
[477202.342677] [11902] 993 11902 5153 34 14 40 0 chronyd
[477202.342679] [12111] 0 12111 27048 1 9 24 0 agetty
[477202.342681] [12357] 0 12357 71167 28 35 127 0 dsm_sa_eventmgr
[477202.342682] [12388] 0 12388 158235 42 44 142 0 dsm_om_shrsvcd
[477202.342684] [12471] 10000 12471 3494 10 10 29 0 noms_nsight
[477202.342685] [12476] 10000 12476 158188 4782 41 326 0 noms_nsight
[477202.342687] [12481] 10000 12481 3494 9 10 30 0 noms_nsight
[477202.342689] [12486] 10000 12486 158188 4737 36 132 0 noms_nsight
[477202.342690] [17331] 0 17331 11723 38 26 169 0 box
[477202.342692] [17337] 0 17337 580152 278 93 1197 0 box
[477202.342693] [23279] 0 23279 11723 43 28 163 0 box
[477202.342695] [23281] 0 23281 580152 414 93 1116 0 box
[477202.342696] [23421] 0 23421 7264 7 18 209 0 box
[477202.342697] [23422] 0 23422 79501 29 28 241 0 box
[477202.342700] [ 3180] 999 3180 152656 90 59 784 0 polkitd
[477202.342701] [133637] 0 133637 64947 327 84 4774 0 salt-minion
[477202.342703] [133641] 0 133641 157658 4145 120 5711 0 salt-minion
[477202.342705] [133646] 0 133646 94607 481 101 6186 0 salt-minion
[477202.342707] [142291] 500 142291 28628388 15394990 36178 1027421 0 nodeos
[477202.342710] Out of memory: Kill process 142291 (nodeos) score 942 or sacrifice child
[477202.342743] Killed process 142291 (nodeos) total-vm:114513552kB, anon-rss:61579944kB, file-rss:0kB, shmem-rss:16kB
