Malloc is extremely expensive

mmap()

Memory mapping of the block-data

The basic premise of blockchain as a concept is that the block data is per definition immutable. Once written, it can never be changed. There can sometimes be reorgs, but in that case blocks are discarded and still not changed.

Based on this knowledge we realize that there is only one canonical form of a block and that is the one exactly how it is written to disk.

Modern operating systems allow us to use a file on disk directly in an application without first copying it into memory and they call this process “memory-mapping of files”.

The process is that the application finds a file on disk, it locks the file by opening it and the application requests data like the total size of the file.
The application can then ‘map’ a section (or the whole) file into memory which returns a memory location that the operating system has assigned.

Any reads or writes in the memory area allocated for the mem-mapped file will result in the operating system loading or saving those changes to the file. Without any additional copying of data.

Advantages

This system was introduced in 2017 to Flowee the Hub, the advantages are in comparison to the system that was there before. Which is the same system that most competing nodes still use today.

  • Reading of a block-file now is always exactly one ‘malloc’. Regardless of how many blocks actually reside in it. Actual memory used is minimal (buffers only) and managed by the operating system.
    In comparison, if you read a 100 MB block in the old system there were 400 MB worth of mallocs, to the count of 4 per transaction. This is slow and expensive.

  • A single block can be mapped and then passed between any number of readers in any number of threads without risk of copying more data, and thus with no slow down. In other words, blocks are quite cheap to share between threads.
    In comparison, the old system did not allow sharing of block data between threads without mutex locking (std::vector is not reentrant), unless the full data would be copied.

  • Blocks are essentially a list of transactions. Mapping a block automatically maps all transactions.
    The FastBlock class has a method to return a list of FastTransaction objects, the exceptional part of that API is that the list of transactions all use the same memory-mapped data that the parent block used and no actual data is copied. The mapped block-data ref-count is just increased.
    In comparison, the old system would deep-copy the entire transaction data, which has scalability issues as blocks grow.

For developers

As the main data-format we have a ConstBuffer class. This is essentially a simple wrapper around a shared_ptr to a memory area. This shared pointer will end up pointing to the memory-mapped buffer.

namespace Streaming {
class ConstBuffer {
public:
    ConstBuffer(); // invalid buffer
    explicit ConstBuffer(std::shared_ptr<char> buffer, char const *start, char const *stop);
    bool isValid() const;

    char const* begin() const;
    char const* end() const;
    char operator[](size_t pos) const;
    int size() const;
    ConstBuffer mid(int offset, int length) const;
};
}

The block and transaction objects are called FastBlock and FastTransaction and they are essentially simple wrappers around the ConstBuffer class. A block is thus backed by a shared_ptr to a memory-mapped file.

In the Blocks::DB class there is a method to actually load the data:

namespace Blocks {
class DB {
    FastBlock loadBlock(CDiskBlockPos pos) {
        return FastBlock(d->loadBlock(pos, ForwardBlock, 0));
    }
};

Streaming::ConstBuffer DBPrivate::loadBlock(CDiskBlockPos pos, BlockType type, const uint256 *)
{
    auto buf = mapFile(pos.nFile, type);
    return Streaming::ConstBuffer(buf, buf.get() + pos.nPos, buf.get() + pos.nPos + blockSize);
}

The mapFile() method is essentially using the boost::iostreams::mapped_file class to open and map the entire file. It also makes sure that on de-reference of the last shared-pointer the file will be closed.

The actual mapping and unmapping of the file locks a mutex, any other access of these files is lock-free.