c++ - Set sparsity pattern of Eigen::SparseMatrix without memory overhead -

i need set sparsity pattern of eigen::sparsematrix i know (i have unique sorted column indices , row offsets). it's possible via setfromtriplets unfortunately setfromtriplets requires lot of additional memory (at least in case)

i wrote small example

const long nrows = 5000000; const long ncols = 100000; const long ncols2skip = 1000; //it's quite big! const long ntriplets2reserve = nrows * (ncols / ncols2skip) * 1.1; eigen::sparsematrix<double, eigen::rowmajor, long> mat(nrows, ncols);  std::vector<eigen::triplet<double, long>> triplets;  triplets.reserve(ntriplets2reserve); for(long row = 0; row < nrows; ++row){     for(long col = 0; col < ncols; col += ncols2skip){         triplets.push_back(eigen::triplet<double, long>(row, col, 1));     } } std::cout << "filling mat" << std::endl << std::flush; mat.setfromtriplets(triplets.begin(), triplets.end());  std::cout << "finished! nnz " << mat.nonzeros() << std::endl; //stupid way check memory consumption std::cin.get(); 

in case example consumes 26gb @ peak (between lines "filling mat" , "finished") , 18gb after all. (i made checks via htop). ~8gb overhead quite big me (in "real world" task have bigger overhead).

so have 2 questions:

1. how fill sparsity pattern of eigen::sparsematrix little overhead possible
2. why setfromtriplets requires many memory?

please let me know if example wrong.

my eigen version 3.3.2

ps sorry english

edit: it's looks inserting (with preallocation) each triplet manually works faster , requires less memory @ peak. still want know possible set sparsity pattern manually

ad 1: can bit more efficient plain insert using internal functions startvec , insertback, if can guarantee insert elements in lexicographical order.

ad 2: if use setfromtriplets need approximately twice final size of matrix (plus size of triplet container), since elements first inserted in transposed version of matrix, transposed final matrix in order make sure inner vectors sorted. if know structure of matrix ahead, quite waste of memory, intended work on arbitrary input data.

in example have 5000000 * 100000 / 1000 = 5e8 elements. triplet requires 8+8+8 = 24 bytes (making 12gb vector) , each element of sparse matrix requires 8+8=16 bytes (one double value, 1 long inner index), i.e., 8gb per matrix, in total require 28gb 26 gib.

bonus: if matrix has special structure, can stored more efficiently, , willing dig deeper eigen internals, may consider implementing new type inheriting eigen::sparsebase<> (but don't recomment this, unless memory/performance critical you, , willing go through lot of "sparsely" documented internal eigen code ...). however, in case easier think intend matrix, , try implement special operations that.