[redland-dev] Exponential memory growth with 'hashes'.

[Roel Janssen]

Dear list,

I'm trying extract triples from a custom file format.  I'm trying to use
Redland so that I can output to various RDF formats.

First I tried to initialize the storage with:
librdf_new_storage (config.rdf_world, "memory", NULL, NULL);

But since the number of statements can grow quite large, this seems to
be rather slow.

So I tried with a storage initialized with:
librdf_new_storage (config.rdf_world, "hashes", NULL, "hash-type='memory'");

But unfortunately, it seems that the memory grows exponentially when adding
more than ~1.800.000 nodes.

This doesn't seem to happen when I initialize the storage with:
librdf_new_storage (config.rdf_world, "hashes", "redland", "new='yes',hash-type='bdb',dir='/tmp'");

But this heavily writes to disk, which I'm trying to avoid.

Is the exponential memory growth expected behavior?  And what could I do
to avoid it, maximizing performance when I only serialize data to an RDF format?

Thanks for your time!

Kind regards,
Roel Janssen