2

Why suddenly my nodeos memory(ram with swap) usage been 100%?

and memory full usage then oop

enter image description here

Am I setting chain-thread to 18?

My Nodeos Ver:

  • v1.8.0-rc2

Sync status

  • completed full sync

nodeos config.ini:

# Maximum size (in MiB) of the chain state database (eosio::chain_plugin)
chain-state-db-size-mb = 45484

# print contract's output to console (eosio::chain_plugin)
contracts-console = false

# Database read mode ("speculative", "head", or "read-only").
# In "speculative" mode database contains changes done up to the head block plus changes made by transactions not yet included to the blockchain.
# In "head" mode database contains changes done up to the current head block.
# In "read-only" mode database contains incoming block changes but no speculative transaction processing.
#  (eosio::chain_plugin)
read-mode = irrerversible

# Chain validation mode ("full" or "light").
# In "full" mode all incoming blocks will be fully validated.
# In "light" mode all incoming blocks headers will be fully validated; transactions in those validated blocks will be trusted
#  (eosio::chain_plugin)
validation-mode = light

# Disable the check which subjectively fails a transaction if a contract bills more RAM to another account within the context of a notification handler (i.e. when the receiver is not the code of the action). (eosio::chain_plugin)
disable-ram-billing-notify-checks = false

# true: validate that the peer certificates are valid and trusted, false: ignore cert errors (eosio::http_client_plugin)
https-client-validate-peers = 1

# The local IP and port to listen for incoming http connections; set blank to disable. (eosio::http_plugin)
http-server-address = 0.0.0.0:8888

# Specify the Access-Control-Allow-Origin to be returned on each request. (eosio::http_plugin)
access-control-allow-origin = *

# Specify if Access-Control-Allow-Credentials: true should be returned on each request. (eosio::http_plugin)
access-control-allow-credentials = false

# The actual host:port used to listen for incoming p2p connections. (eosio::net_plugin)
p2p-listen-endpoint = 0.0.0.0:9876

# An externally accessible host:port for identifying this node. Defaults to p2p-listen-endpoint. (eosio::net_plugin)
p2p-server-address = 0.0.0.0:9876

# Maximum number of client nodes from any single IP address (eosio::net_plugin)
p2p-max-nodes-per-host = 5

# Can be 'any' or 'producers' or 'specified' or 'none'. If 'specified', peer-key must be specified at least once. If only 'producers', peer-key is not required. 'producers' and 'specified' may be combined. (eosio::net_plugin)
allowed-connection = any

chain-threads = 18        #defaults 2
http-threads = 2          #defaults 2
net-threads = 3          #defaults 1
producer-threads = 2      #defaults 2
txn-test-gen-threads = 0  #defaults 2

# Plugin(s) to enable, may be specified multiple times
plugin = eosio::chain_api_plugin
plugin = eosio::producer_api_plugin
plugin = eosio::state_history_plugin
plugin = eosio::db_size_api_plugin
plugin = eosio::net_plugin
plugin = eosio::http_plugin
plugin = eosio::net_api_plugin

My Server Spec:

CPU:

...

processor       : 39
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping        : 1
microcode       : 0xb00002a
cpu MHz         : 2200.061
cache size      : 25600 KB

...

Mem:

MemTotal:       65684628 kB

DB Size API:

$curl --request POST --url http://127.0.0.1:8888/v1/db_size/get

...
  "free_bytes": "40359701296",
  "used_bytes": "7333728464",
  "size": "47693429760",
...

dmesg:

[431854.056689] Bits 55-60 of /proc/PID/pagemap entries are about to stop being page-shift some time soon. See the linux/Documentation/vm/pagemap.txt for details.
[477202.342504] systemd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0
[477202.342508] systemd cpuset=/ mems_allowed=0-1
[477202.342510] CPU: 4 PID: 1 Comm: systemd Not tainted 3.10.0-957.21.3.el7.x86_64 #1
[477202.342511] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.7.1 001/22/2018
[477202.342512] Call Trace:
[477202.342521]  [<ffffffffb5f63107>] dump_stack+0x19/0x1b
[477202.342524]  [<ffffffffb5f5db2a>] dump_header+0x90/0x229
[477202.342529]  [<ffffffffb5901292>] ? ktime_get_ts64+0x52/0xf0
[477202.342532]  [<ffffffffb59584df>] ? delayacct_end+0x8f/0xb0
[477202.342536]  [<ffffffffb59ba834>] oom_kill_process+0x254/0x3d0
[477202.342538]  [<ffffffffb59ba2dd>] ? oom_unkillable_task+0xcd/0x120
[477202.342540]  [<ffffffffb59ba386>] ? find_lock_task_mm+0x56/0xc0
[477202.342542]  [<ffffffffb59bb076>] out_of_memory+0x4b6/0x4f0
[477202.342544]  [<ffffffffb5f5e62e>] __alloc_pages_slowpath+0x5d6/0x724
[477202.342547]  [<ffffffffb59c1454>] __alloc_pages_nodemask+0x404/0x420
[477202.342552]  [<ffffffffb5894dfd>] copy_process+0x1dd/0x1a40
[477202.342554]  [<ffffffffb5896811>] do_fork+0x91/0x320
[477202.342556]  [<ffffffffb5896b26>] SyS_clone+0x16/0x20
[477202.342560]  [<ffffffffb5f761b4>] stub_clone+0x44/0x70
[477202.342562]  [<ffffffffb5f75ddb>] ? system_call_fastpath+0x22/0x27
[477202.342563] Mem-Info:
[477202.342569] active_anon:14400252 inactive_anon:1385366 isolated_anon:0
 active_file:0 inactive_file:0 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:75895 slab_unreclaimable:32395
 mapped:0 shmem:12 pagetables:37529 bounce:0
 free:43038 free_pcp:1267 free_cma:0
[477202.342573] Node 0 DMA free:15896kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[477202.342577] lowmem_reserve[]: 0 1710 31888 31888
[477202.342580] Node 0 DMA32 free:121152kB min:864kB low:1080kB high:1296kB active_anon:1103028kB inactive_anon:368332kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1751760kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:48008kB slab_unreclaimable:8748kB kernel_stack:384kB pagetables:2504kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[477202.342584] lowmem_reserve[]: 0 0 30177 30177
[477202.342586] Node 0 Normal free:15232kB min:15248kB low:19060kB high:22872kB active_anon:27436164kB inactive_anon:1714808kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB managed:30902132kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:12kB slab_reclaimable:159428kB slab_unreclaimable:60084kB kernel_stack:6256kB pagetables:60040kB unstable:0kB bounce:0kB free_pcp:1560kB local_pcp:28kB free_cma:0kB writeback_tmp:0kB pages_scanned:151 all_unreclaimable? yes
[477202.342590] lowmem_reserve[]: 0 0 0 0
[477202.342592] Node 1 Normal free:19872kB min:16292kB low:20364kB high:24436kB active_anon:29061816kB inactive_anon:3458324kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33554432kB managed:33014832kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:36kB slab_reclaimable:96144kB slab_unreclaimable:60748kB kernel_stack:5008kB pagetables:87572kB unstable:0kB bounce:0kB free_pcp:3496kB local_pcp:76kB free_cma:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? yes
[477202.342595] lowmem_reserve[]: 0 0 0 0
[477202.342597] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
[477202.342606] Node 0 DMA32: 3166*4kB (UEM) 1658*8kB (UEM) 558*16kB (UEM) 269*32kB (UEM) 194*64kB (UEM) 80*128kB (UEM) 29*256kB (UE) 13*512kB (UEM) 4*1024kB (UM) 4*2048kB (UE) 7*4096kB (M) = 121160kB
[477202.342615] Node 0 Normal: 2924*4kB (UEM) 441*8kB (UEM) 5*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15304kB
[477202.342621] Node 1 Normal: 4624*4kB (UEM) 143*8kB (UEM) 20*16kB (UM) 3*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20056kB
[477202.342629] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[477202.342630] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[477202.342631] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[477202.342632] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[477202.342633] 374757 total pagecache pages
[477202.342635] 374653 pages in swap cache
[477202.342636] Swap cache stats: add 5617285, delete 5242610, find 2003207/2521309
[477202.342637] Free swap  = 0kB
[477202.342638] Total swap = 4194300kB
[477202.342639] 16753239 pages RAM
[477202.342640] 0 pages HighMem/MovableOnly
[477202.342641] 332084 pages reserved
[477202.342641] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[477202.342649] [ 5088]     0  5088    15929       39      38       47             0 systemd-journal
[477202.342651] [ 5113]     0  5113    31206       38      28       39             0 lvmetad
[477202.342653] [ 5119]     0  5119    11827       57      25      198         -1000 systemd-udevd
[477202.342655] [10769]     0 10769    28686        0      14       46             0 rsync
[477202.342657] [10776]     0 10776     5487       79      16       70             0 irqbalance
[477202.342659] [10777]    28 10777   198838      119      52       83             0 nscd
[477202.342661] [10779]     0 10779     6116       40      17       36             0 systemd-logind
[477202.342662] [10781]    81 10781    14107       77      32       73          -900 dbus-daemon
[477202.342664] [10823]     0 10823     6051        6      17       43             0 atd
[477202.342665] [11228]     0 11228    26114        1      53      234         -1000 sshd
[477202.342667] [11230]     0 11230    55063      180      64     1017             0 snmpd
[477202.342669] [11231]     0 11231    60534       80      25      170             0 rsyslogd
[477202.342671] [11233]     0 11233     6844       11      18       56             0 xinetd
[477202.342672] [11285]     0 11285     5837        8      15       98             0 ipmievd
[477202.342674] [11846]     0 11846   130807      172      42     1362             0 dsm_sa_datamgrd
[477202.342676] [11866]     0 11866    31093       30      20      121             0 crond
[477202.342677] [11902]   993 11902     5153       34      14       40             0 chronyd
[477202.342679] [12111]     0 12111    27048        1       9       24             0 agetty
[477202.342681] [12357]     0 12357    71167       28      35      127             0 dsm_sa_eventmgr
[477202.342682] [12388]     0 12388   158235       42      44      142             0 dsm_om_shrsvcd
[477202.342684] [12471] 10000 12471     3494       10      10       29             0 noms_nsight
[477202.342685] [12476] 10000 12476   158188     4782      41      326             0 noms_nsight
[477202.342687] [12481] 10000 12481     3494        9      10       30             0 noms_nsight
[477202.342689] [12486] 10000 12486   158188     4737      36      132             0 noms_nsight
[477202.342690] [17331]     0 17331    11723       38      26      169             0 box
[477202.342692] [17337]     0 17337   580152      278      93     1197             0 box
[477202.342693] [23279]     0 23279    11723       43      28      163             0 box
[477202.342695] [23281]     0 23281   580152      414      93     1116             0 box
[477202.342696] [23421]     0 23421     7264        7      18      209             0 box
[477202.342697] [23422]     0 23422    79501       29      28      241             0 box
[477202.342700] [ 3180]   999  3180   152656       90      59      784             0 polkitd
[477202.342701] [133637]     0 133637    64947      327      84     4774             0 salt-minion
[477202.342703] [133641]     0 133641   157658     4145     120     5711             0 salt-minion
[477202.342705] [133646]     0 133646    94607      481     101     6186             0 salt-minion
[477202.342707] [142291]   500 142291 28628388 15394990   36178  1027421             0 nodeos
[477202.342710] Out of memory: Kill process 142291 (nodeos) score 942 or sacrifice child
[477202.342743] Killed process 142291 (nodeos) total-vm:114513552kB, anon-rss:61579944kB, file-rss:0kB, shmem-rss:16kB

1 Answer 1

1

nodeos shared_memory.bin looks like thin provisioning.

so that, There is no problem even if my physical memory is 64GB that less than 128GB of chain_state_db size.

If you set 128GB to chain_state_db then nodeos don't using your physical memory for chain_state_db.

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .