https://jira.akraino.org This file is an XML representation of an issue en-us 9.4.5 940005 11-04-2023 https://jira.akraino.org/browse/IEC-26 Integrated Edge Cloud <p>This problem happens on aarch64 when trying to prepare the env for cord-tester testing framework. Because the pynacl python package is not compiled for aarch64, the setup_venv.sh will try to build it and in the process also runs the tests (i.e. calls make check). One of the 72 tests (pwhash_scrypt) fails after timing out and what seems to be an out of memory error.</p> <p>The same problem has been reported last year here:<br/> <a href="https://github.com/jedisct1/libsodium/issues/721" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jedisct1/libsodium/issues/721</a></p> <p>One of the solutions suggested is to increase the memlock ulimit, which for some reason is very low by default on aarch64.</p> <p>I have tried setting up this ulimit in the container we use for testing and it works when running on the jenkins slave (baremetal). However the test still crashes when running the framework from the same cord-tester image but on the master node of an aarch64 virtual pod.</p> <p> </p> IEC-26

[IEC][SEBA][PONSim] cord-tester setup_venv fails: pynacl pwhash_scrypt out of memory

Bug Medium Done Done Unassigned Ciprian Barbu Fri, 25 Oct 2019 16:35:10 +0000 Wed, 30 Oct 2019 13:39:47 +0000 Wed, 30 Oct 2019 13:39:47 +0000 0 2 <p>My bug was closed on the libsodium github repo, but I replied to this thread on pynacl:</p> <p><a href="https://github.com/pyca/pynacl/issues/486" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/pyca/pynacl/issues/486</a></p> <p>I have opened a new issue for the github repository:</p> <p><a href="https://github.com/jedisct1/libsodium/issues/890" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jedisct1/libsodium/issues/890</a></p> <p>So it looks like there are two problems here. Most of the times (if not always) the build of pynacl will fail during the tests because the ulimit for max memory lock is very low. This is also described in the bug report mentioned in the description.</p> <p>The second problem was seen when running the build on the IEC master node. Here the test would suddenly start taking up more and more memory, exhausting all of the available RAM and then going on to consume the swap memory.</p> <p>I eventually started debugging this test case and found there is a negative test in pwhash_scrypt <span class="error">&#91;1&#93;</span> which by the looks of it specifies an invalid encoded string which probably is supposed to cause a failure down the line when allocating memory <span class="error">&#91;2&#93;</span> for the decryption process. I noticed that the computed required memory goes in the range of TB, which is really not supposed to work.</p> <p>Searching for references on mmap not failing when large amounts of memory are requests, I ended up on this thread <span class="error">&#91;3&#93;</span>, called:<br/> In a 64 bit process, will my mmap / malloc request ever be denied?</p> <p>One of the answers in this thread talks about memory overcommit limit, which can be set via sysctl vm.memory_overcommit. This parameter controls the amount of memory overcommit checks that the kernel performs during syscalls like mmap and related. On our IEC K8S master, this was set to 1, which means no overcommit handling is performed.</p> <p>Additionally, the way libsodium implements scrypt alloc_region function, it forces it to also populate the pages, which is useful under normal circumstances, but in this case, combined with the value of vm.memory_overcommit, caused the OS to exhaust all the memory trying to perform the mmap request.</p> <p>So the solution is to set the vm.memory_overcommit value to something that allows extra memory checks, values of 0 or 2 both work fine.</p> <p>But in my opinion the libsodium implementation of maybe the test case is not very well thought, since this is one valid example of configuring the OS, to allow better control for memory in virtualized environments. One could also consider this a security threat.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://github.com/jedisct1/libsodium/blob/1.0.16/test/default/pwhash_scrypt.c#L269" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jedisct1/libsodium/blob/1.0.16/test/default/pwhash_scrypt.c#L269</a></p> <p><span class="error">&#91;2&#93;</span> <a href="https://github.com/jedisct1/libsodium/blob/1.0.16/src/libsodium/crypto_pwhash/scryptsalsa208sha256/scrypt_platform.c#L45" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jedisct1/libsodium/blob/1.0.16/src/libsodium/crypto_pwhash/scryptsalsa208sha256/scrypt_platform.c#L45</a></p> <p><span class="error">&#91;3&#93;</span> <a href="https://stackoverflow.com/questions/4923116/in-a-64-bit-process-will-my-mmap-malloc-request-ever-be-denied" class="external-link" target="_blank" rel="nofollow noopener">https://stackoverflow.com/questions/4923116/in-a-64-bit-process-will-my-mmap-malloc-request-ever-be-denied</a></p> Development Rank 0|i002hw: