2

Why suddenly my nodeos memory(ram with swap) usage been 100%?

and memory full usage then oop

enter image description here

Am I setting chain-thread to 18?

My Nodeos Ver:

  • v1.8.0-rc2

Sync status

  • completed full sync

nodeos config.ini:

# Maximum size (in MiB) of the chain state database (eosio::chain_plugin)
chain-state-db-size-mb = 45484

# print contract's output to console (eosio::chain_plugin)
contracts-console = false

# Database read mode ("speculative", "head", or "read-only").
# In "speculative" mode database contains changes done up to the head block plus changes made by transactions not yet included to the blockchain.
# In "head" mode database contains changes done up to the current head block.
# In "read-only" mode database contains incoming block changes but no speculative transaction processing.
#  (eosio::chain_plugin)
read-mode = irrerversible

# Chain validation mode ("full" or "light").
# In "full" mode all incoming blocks will be fully validated.
# In "light" mode all incoming blocks headers will be fully validated; transactions in those validated blocks will be trusted
#  (eosio::chain_plugin)
validation-mode = light

# Disable the check which subjectively fails a transaction if a contract bills more RAM to another account within the context of a notification handler (i.e. when the receiver is not the code of the action). (eosio::chain_plugin)
disable-ram-billing-notify-checks = false

# true: validate that the peer certificates are valid and trusted, false: ignore cert errors (eosio::http_client_plugin)
https-client-validate-peers = 1

# The local IP and port to listen for incoming http connections; set blank to disable. (eosio::http_plugin)
http-server-address = 0.0.0.0:8888

# Specify the Access-Control-Allow-Origin to be returned on each request. (eosio::http_plugin)
access-control-allow-origin = *

# Specify if Access-Control-Allow-Credentials: true should be returned on each request. (eosio::http_plugin)
access-control-allow-credentials = false

# The actual host:port used to listen for incoming p2p connections. (eosio::net_plugin)
p2p-listen-endpoint = 0.0.0.0:9876

# An externally accessible host:port for identifying this node. Defaults to p2p-listen-endpoint. (eosio::net_plugin)
p2p-server-address = 0.0.0.0:9876

# Maximum number of client nodes from any single IP address (eosio::net_plugin)
p2p-max-nodes-per-host = 5

# Can be 'any' or 'producers' or 'specified' or 'none'. If 'specified', peer-key must be specified at least once. If only 'producers', peer-key is not required. 'producers' and 'specified' may be combined. (eosio::net_plugin)
allowed-connection = any

chain-threads = 18        #defaults 2
http-threads = 2          #defaults 2
net-threads = 3          #defaults 1
producer-threads = 2      #defaults 2
txn-test-gen-threads = 0  #defaults 2

# Plugin(s) to enable, may be specified multiple times
plugin = eosio::chain_api_plugin
plugin = eosio::producer_api_plugin
plugin = eosio::state_history_plugin
plugin = eosio::db_size_api_plugin
plugin = eosio::net_plugin
plugin = eosio::http_plugin
plugin = eosio::net_api_plugin

My Server Spec:

CPU:

...

processor       : 39
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping        : 1
microcode       : 0xb00002a
cpu MHz         : 2200.061
cache size      : 25600 KB

...

Mem:

MemTotal:       65684628 kB

DB Size API:

$curl --request POST --url http://127.0.0.1:8888/v1/db_size/get

...
  "free_bytes": "40359701296",
  "used_bytes": "7333728464",
  "size": "47693429760",
...

dmesg:

[431854.056689] Bits 55-60 of /proc/PID/pagemap entries are about to stop being page-shift some time soon. See the linux/Documentation/vm/pagemap.txt for details.
[477202.342504] systemd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0
[477202.342508] systemd cpuset=/ mems_allowed=0-1
[477202.342510] CPU: 4 PID: 1 Comm: systemd Not tainted 3.10.0-957.21.3.el7.x86_64 #1
[477202.342511] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.7.1 001/22/2018
[477202.342512] Call Trace:
[477202.342521]  [<ffffffffb5f63107>] dump_stack+0x19/0x1b
[477202.342524]  [<ffffffffb5f5db2a>] dump_header+0x90/0x229
[477202.342529]  [<ffffffffb5901292>] ? ktime_get_ts64+0x52/0xf0
[477202.342532]  [<ffffffffb59584df>] ? delayacct_end+0x8f/0xb0
[477202.342536]  [<ffffffffb59ba834>] oom_kill_process+0x254/0x3d0
[477202.342538]  [<ffffffffb59ba2dd>] ? oom_unkillable_task+0xcd/0x120
[477202.342540]  [<ffffffffb59ba386>] ? find_lock_task_mm+0x56/0xc0
[477202.342542]  [<ffffffffb59bb076>] out_of_memory+0x4b6/0x4f0
[477202.342544]  [<ffffffffb5f5e62e>] __alloc_pages_slowpath+0x5d6/0x724
[477202.342547]  [<ffffffffb59c1454>] __alloc_pages_nodemask+0x404/0x420
[477202.342552]  [<ffffffffb5894dfd>] copy_process+0x1dd/0x1a40
[477202.342554]  [<ffffffffb5896811>] do_fork+0x91/0x320
[477202.342556]  [<ffffffffb5896b26>] SyS_clone+0x16/0x20
[477202.342560]  [<ffffffffb5f761b4>] stub_clone+0x44/0x70
[477202.342562]  [<ffffffffb5f75ddb>] ? system_call_fastpath+0x22/0x27
[477202.342563] Mem-Info:
[477202.342569] active_anon:14400252 inactive_anon:1385366 isolated_anon:0
 active_file:0 inactive_file:0 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:75895 slab_unreclaimable:32395
 mapped:0 shmem:12 pagetables:37529 bounce:0
 free:43038 free_pcp:1267 free_cma:0
[477202.342573] Node 0 DMA free:15896kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[477202.342577] lowmem_reserve[]: 0 1710 31888 31888
[477202.342580] Node 0 DMA32 free:121152kB min:864kB low:1080kB high:1296kB active_anon:1103028kB inactive_anon:368332kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1751760kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:48008kB slab_unreclaimable:8748kB kernel_stack:384kB pagetables:2504kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[477202.342584] lowmem_reserve[]: 0 0 30177 30177
[477202.342586] Node 0 Normal free:15232kB min:15248kB low:19060kB high:22872kB active_anon:27436164kB inactive_anon:1714808kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB managed:30902132kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:12kB slab_reclaimable:159428kB slab_unreclaimable:60084kB kernel_stack:6256kB pagetables:60040kB unstable:0kB bounce:0kB free_pcp:1560kB local_pcp:28kB free_cma:0kB writeback_tmp:0kB pages_scanned:151 all_unreclaimable? yes
[477202.342590] lowmem_reserve[]: 0 0 0 0
[477202.342592] Node 1 Normal free:19872kB min:16292kB low:20364kB high:24436kB active_anon:29061816kB inactive_anon:3458324kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33554432kB managed:33014832kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:36kB slab_reclaimable:96144kB slab_unreclaimable:60748kB kernel_stack:5008kB pagetables:87572kB unstable:0kB bounce:0kB free_pcp:3496kB local_pcp:76kB free_cma:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? yes
[477202.342595] lowmem_reserve[]: 0 0 0 0
[477202.342597] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
[477202.342606] Node 0 DMA32: 3166*4kB (UEM) 1658*8kB (UEM) 558*16kB (UEM) 269*32kB (UEM) 194*64kB (UEM) 80*128kB (UEM) 29*256kB (UE) 13*512kB (UEM) 4*1024kB (UM) 4*2048kB (UE) 7*4096kB (M) = 121160kB
[477202.342615] Node 0 Normal: 2924*4kB (UEM) 441*8kB (UEM) 5*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15304kB
[477202.342621] Node 1 Normal: 4624*4kB (UEM) 143*8kB (UEM) 20*16kB (UM) 3*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20056kB
[477202.342629] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[477202.342630] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[477202.342631] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[477202.342632] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[477202.342633] 374757 total pagecache pages
[477202.342635] 374653 pages in swap cache
[477202.342636] Swap cache stats: add 5617285, delete 5242610, find 2003207/2521309
[477202.342637] Free swap  = 0kB
[477202.342638] Total swap = 4194300kB
[477202.342639] 16753239 pages RAM
[477202.342640] 0 pages HighMem/MovableOnly
[477202.342641] 332084 pages reserved
[477202.342641] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[477202.342649] [ 5088]     0  5088    15929       39      38       47             0 systemd-journal
[477202.342651] [ 5113]     0  5113    31206       38      28       39             0 lvmetad
[477202.342653] [ 5119]     0  5119    11827       57      25      198         -1000 systemd-udevd
[477202.342655] [10769]     0 10769    28686        0      14       46             0 rsync
[477202.342657] [10776]     0 10776     5487       79      16       70             0 irqbalance
[477202.342659] [10777]    28 10777   198838      119      52       83             0 nscd
[477202.342661] [10779]     0 10779     6116       40      17       36             0 systemd-logind
[477202.342662] [10781]    81 10781    14107       77      32       73          -900 dbus-daemon
[477202.342664] [10823]     0 10823     6051        6      17       43             0 atd
[477202.342665] [11228]     0 11228    26114        1      53      234         -1000 sshd
[477202.342667] [11230]     0 11230    55063      180      64     1017             0 snmpd
[477202.342669] [11231]     0 11231    60534       80      25      170             0 rsyslogd
[477202.342671] [11233]     0 11233     6844       11      18       56             0 xinetd
[477202.342672] [11285]     0 11285     5837        8      15       98             0 ipmievd
[477202.342674] [11846]     0 11846   130807      172      42     1362             0 dsm_sa_datamgrd
[477202.342676] [11866]     0 11866    31093       30      20      121             0 crond
[477202.342677] [11902]   993 11902     5153       34      14       40             0 chronyd
[477202.342679] [12111]     0 12111    27048        1       9       24             0 agetty
[477202.342681] [12357]     0 12357    71167       28      35      127             0 dsm_sa_eventmgr
[477202.342682] [12388]     0 12388   158235       42      44      142             0 dsm_om_shrsvcd
[477202.342684] [12471] 10000 12471     3494       10      10       29             0 noms_nsight
[477202.342685] [12476] 10000 12476   158188     4782      41      326             0 noms_nsight
[477202.342687] [12481] 10000 12481     3494        9      10       30             0 noms_nsight
[477202.342689] [12486] 10000 12486   158188     4737      36      132             0 noms_nsight
[477202.342690] [17331]     0 17331    11723       38      26      169             0 box
[477202.342692] [17337]     0 17337   580152      278      93     1197             0 box
[477202.342693] [23279]     0 23279    11723       43      28      163             0 box
[477202.342695] [23281]     0 23281   580152      414      93     1116             0 box
[477202.342696] [23421]     0 23421     7264        7      18      209             0 box
[477202.342697] [23422]     0 23422    79501       29      28      241             0 box
[477202.342700] [ 3180]   999  3180   152656       90      59      784             0 polkitd
[477202.342701] [133637]     0 133637    64947      327      84     4774             0 salt-minion
[477202.342703] [133641]     0 133641   157658     4145     120     5711             0 salt-minion
[477202.342705] [133646]     0 133646    94607      481     101     6186             0 salt-minion
[477202.342707] [142291]   500 142291 28628388 15394990   36178  1027421             0 nodeos
[477202.342710] Out of memory: Kill process 142291 (nodeos) score 942 or sacrifice child
[477202.342743] Killed process 142291 (nodeos) total-vm:114513552kB, anon-rss:61579944kB, file-rss:0kB, shmem-rss:16kB
1

nodeos shared_memory.bin looks like thin provisioning.

so that, There is no problem even if my physical memory is 64GB that less than 128GB of chain_state_db size.

If you set 128GB to chain_state_db then nodeos don't using your physical memory for chain_state_db.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.