[redland-dev] Re: Redland: Problem With Python-Bindings

[Dave Beckett]

On Wed, 11 Aug 2004 21:10:07 +0100
"Sean B. Palmer" <sean+redn at infomesh.net> wrote:

> Hi Dave et al.,
> 
> I've been unable to get Redland's python language bindings to fully
> work, despite a significant amount of effort in trying to do so.
> 
> The installation of Redland went without a hitch: I installed Raptor
> 1.3.2 first, and then Redland 0.9.18 using the system Raptor. When I
> came to install the 0.9.18.1 bindings, however, I encountered numerous
> problems.
> 
> At first, there were lots of syntax errors along the lines of:
> 
>     /usr/local/share/redland/Redland.i :
>       Line 335. Syntax error in input.
> 
> Since I'm using Debian Stable, I decided that this was due to an
> out-of-date version of SWIG, and grabbed the latest package from the
> SWIG site. ...

This is correct.  SWIG changed syntax.  However there was a packaging
bug in redland 0.9.18 that caused problems with redland-bindings
0.9.18.1 due to timestamps.  Basically all the SWIG-generated files
were all newer than the Redland.i but when 'make install' installed
Redland.i, it destroyed this relationship.  I've fixed that in
Redland CVS.  So this shouldn't be a problem in future.


> ... This stopped the error in make, but the Redland module
> still wouldn't import in Python.
> 
> I did "$ ldd -d Redland.so", and it came up with some unresolved
> dependencies, which looked promising. Googling on this led me to find
> a message that you sent earlier this year [1] that advised setting
> LD_LIBRARY_PATH, so I did:
> 
>     $ export
>     LD_LIBRARY_PATH=/usr/local/lib:/usr/local/BerkeleyDB.4.2/lib

OK, you have your own compiles of BDB and others.

> and Redland.so got fully linked this time. I've no idea why it
> required me to set LD_LIBRARY_PATH since I'd previously configured
> Redland using"--with-bdb=/usr/local/BerkeleyDB.4.2", and ./configure
> in the bindings package had detected Redland properly. I'd of course
> done "make install" when I built Redland too.

Even if you compile with a dynamic link against libraries
like -L/usr/local/lib -lfoo -L/usr/local/BerkeleyDB.4.2/lib -ldb
it does NOT guarantee that they will be found at run time.

(This does depend on the system.  On other systems such as Solaris,
OSX, this may not be the case.  This is one of many reasons why
libtool exists.  However, burnt-in dynamic library search paths
are a problem in their own.)

They typically must be in the systems's standard shared library path
which on linux is found in /etc/ld.so.conf.  For debian that does
not usually include /usr/local/lib or the BDB directory (since BDB is
normally available in /usr/lib).
 
> Anyway, unfortunately--and this is the problem that I still can't get
> around--I was getting the following kind of error:
> 
>     $ python2.3 example.py
>     Traceback (most recent call last):
>     File "example.py", line 23, in ?
>     import RDF
>     File "/home/sbp/redland-bindings-0.9.18.1/python/RDF.py",
>        line 1851, in ?
>     _world=World()
>     File "/home/sbp/redland-bindings-0.9.18.1/python/RDF.py",
>        line 187, in __init__
>     Redland.librdf_python_world_init(self._world)
>     AttributeError: 'module' object has no attribute
>        'librdf_python_world_init'
> 
> Looking at the ldd output again showed the problem:
> 
>     $ ldd -d Redland.so
>             librdf.so.0 => /usr/local/lib/librdf.so.0 (0x4000f000)

working here, since LD_LIBRARY_PATH was set, as I said above.

>             [...]
>     undefined symbol: _Py_NoneStruct        (./Redland.so)
>     undefined symbol: PyString_Type (./Redland.so)
>     undefined symbol: PyExc_TypeError       (./Redland.so)
>     undefined symbol: PyTuple_Type  (./Redland.so)
>     undefined symbol: PyInt_Type    (./Redland.so)
>     undefined symbol: PyList_Type   (./Redland.so)
>     undefined symbol: PyInstance_Type       (./Redland.so)
>     undefined symbol: PyType_Type   (./Redland.so)
>     undefined symbol: PyExc_NameError       (./Redland.so)
> 
> I did quite a bit of Googling on the "undefined symbol" errors, and
> decided that this was due to Python not having been built using
> --enable-shared. ...

Seems plausible.  So you are running your own compile of python
as well as BDB.

> .. I downloaded Python 2.4 and rebuilt Redland and the
> bindings, and this resolved some of the symbols to the point where
> example.py managed to run half way through, and then was unable to
> resolve q.run_as_bindings.

Never tried with python 2.4 (is it released?).  Typo?

> Looking at ldd again, only the following symbols were undefined this
> time: _Py_NoneStruct, PyString_Type, PyExc_TypeError, PyInt_Type,
> PyType_Type, PyExc_NameError. I decided that since I'm using Debian
> Stable, perhaps something wasn't up-to-date enough for Redland and it
> wasn't detecting it (a la the SWIG problem mentioned earlier).
> 
> Using testing/unstable pacakges on stable isn't really an option, so I
> decided to reinstall Debian from scratch on a new filesystem and use
> Debian unstable (Sid). It took a few hours, but then I was able to use
> "aptitude install rdflib0" to install redland, ...

librdf0 I assume

> ... and get the Python
> bindings too....

python2.3-librdf

> ... Note that the advice in [2] is out-of-date now: the
> packages for Redland are in Debian Unstable, so you *don't* need to
> add the lines to /etc/apt/sources.list as it says.

Actually, they are never out of date and especially at this time.

The latest debian python(perl,ruby) debian packages are only on the
redland site since the debian archive is currently processing
redland-bindings 0.9.18.1.  The python(perl,ruby) packages in the
debian archive are older, and do not have the query language support.

The librdf0 in the debian archive *is* the latest (0.9.18) but *will
not* work with the python2.3-librdf in the debian archive (0.9.16)
due to internal changes and additions like the query api.

> Need I say how the python bindings installation went?
> 
>     $ python2.3
>     Python 2.3.4 (#2, Aug  5 2004, 09:33:45)
>     [GCC 3.3.4 (Debian 1:3.3.4-7)] on linux2
>     Type "help", "copyright", "credits" or "license" for more
>     information.
>     >>> import RDF
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>       File "/usr/lib/python2.3/site-packages/RDF.py", line 118, in ?
>         import Redland
>     ImportError: /usr/lib/python2.3/site-packages/Redland.so:
>     undefined symbol: librdf_error
>     >>>

That is probably caused by running an python2.3-librdf (0.9.16)
against a newer librdf0 (0.9.18) which no longer users that internal
symbol.  What does ldd /usr/lib/python2.3/site-packages/Redland.so
say?  I bet it links to a mismatched librdf.so

So, I recommend you grab the packages in-sync from [2] until
the debian archive accepts the latest packages (probably a couple
of weeks).  I'm running them now of course and they do work on debian
unstable.

> I also tried building the redland bindings package straight again, but
> this time it fails to even find the system version of Redland that I
> installed using aptitude (the replacement for apt-get).

You likely need the librdf-dev package installed, so that the
redland-config program is available.  That's what redland-bindings
looks for.

> So that's my installation saga! The closest I managed to come was by
> using Python2.4-alpha on Debian Stable, but even then it wasn't really
> working. I'd very much like to be able to do some Python development
> using Redland, so I'm really hoping that I've given you enough
> information here to be able to resolve this. I know it's frustrating
> when someone lodges a five page bug report--especially when there's
> probably a two line solution--but as you can see, this wasn't a quick
> five-minutes-and-give-up sort of situation.
> 
> Cheers,
> 
> [1] [redland-dev] Problems with Tcl interface...
> http://lists.usefulinc.com/pipermail/redland-dev/2004-March/000451.html
> [2] http://www.redland.opensource.ac.uk/dist/binaries/debian/README

Dave