[redland-dev] Messing up bdb

[Dave Beckett]

On Wed, 15 Jun 2005, Danny Ayers wrote:
> Dave, I think I might have found what was causing my db foulups
> earlier, looks like an overflow (or break in the middle of a stream).
> Symptom was -
> 
> Error:Illegal node encoding 'u' seen
> 
> I checked the db42 install, reinstalled Redland, ran the crawl again.
> Same error, except this time it was encoding 'a'. It appeared in the
> same bunch of triples as before, from:
> http://www.greenrobe.com/node/feed
> XSLT'd with:
> http://purl.org/net/syndication/subscribe/feed-rss1.0.xsl
> 
> The last bit of that feed is a <description> about 110,000 chars long,
> which would have been added as a literal, or at least attempted. Time
> I started trimming ;-)

Yes, redland uses a 2-byte field to store the length of literals
when it encodes and stores them. Suggestions what to do when it's too
long  are welcome - probably at the least it should warn.  I can add
support for literals with 4-byte lengths if people want them (with a
new encoding type).

Note though, that the BDB indexing means that you copy literals
for every time 'o' (triple object) is mentioned in an index, so
that means 300K bytes+.

Maybe the SQLite store would work a little better as it normalises a 
little, but it's still new so take care.

Dave