[redland-dev] Digests, Endianness and Bit Order

[Dave Beckett]

On Wed, 30 Mar 2005 13:21:28 +0200, Simon Cross <simon at sanbi.ac.za> wrote:

> Greetings,
> 
> I'm currently using Redland on both PPC (Gentoo, Mac G4, for
> development) and Intel (FreeBSD, production).  It seems that the MD5
> digests created by the internal md5 digest code differs on the two
> platforms.
> 
> Looking at rdf_digest_md5.c I can see a byteReverse function which gets
> called when WORDS_BIGENDIAN is defined so I modified the Gentoo ebuild
> to add -DWORDS_BIGENDIAN and recompiled just to be sure.  This didn't
> seem to change anything so I assume ./configure was already correctly
> detecting the endianness.
> 
> This just leaves the bit ordering.  PPC seems to be MSB while the Intel
> machine is LSB.  I assume the constant bit fields used in
> rdf_digest_md5.c would thus be incorrect on PPC?
> 
> I'm not really a bit-level C expert so I'm hoping someone can (a)
> confirm that I am not making some stupid mistake and (b) suggest a good
> way of fixing the problem.
> 
> I've attached a small C file which demonstrates the problem (you'll need
> to run it on two machines with different bit orders).

MD5 checking is part of the standard redland 'make check' test suite
and it *should* be working.

However, I turned your test into a self-running script (attached) and
ran it on various machines here and in sourceforge's compile farm:

(AMD64 Linux 64bit little endian)
ht: System: Linux 2.6.9-1.667smp x86_64
ht: Compiling
hash-test.c: In function `main':
hash-test.c:14: warning: cast to pointer from integer of different size
hash-test.c:18: warning: cast to pointer from integer of different size
ht: Running
Expect: 9772427386146006452
Digest: 9772427386146006452

(x86 Linux 32bit little endian)
ht: System: Linux 2.6.10-1.770_FC3 i686
ht: Compiling
ht: Running
Expect: 9772427386146006452
Digest: 9772427386146006452

(x86 FreeBSD 32bit little endian)
ht: System: FreeBSD 5.4-STABLE i386
ht: Compiling
ht: Running
Expect: 9772427386146006452
Digest: 9772427386146006452

(Alpha Linux 64bit little endian)
ht: System: Linux 2.2.20 unknown
ht: Compiling
hash-test.c: In function `main':
hash-test.c:14: warning: cast to pointer from integer of different size
hash-test.c:18: warning: type mismatch in implicit declaration for built-in function `memcpy'
hash-test.c:18: warning: cast to pointer from integer of different size
ht: Running
Expect: 9772427386146006452
Digest: 9772427386146006452

(PowerPC OSX 32bit big endian)
ht: System: Darwin 7.8.0 powerpc
ht: Compiling
ht: Running
Expect: 9772427386146006452
Digest: 12980931450867195527

Oops!

It also looks like I have no 64bit big endian systems available to
test that combination.

Anyone want to buy me a G4? :)

Dave
-------------- next part --------------
#!/bin/sh
program=`basename $0`

echo "$program: System: `uname -s` `uname -r` `uname -p`"

p=
for dir in `redland-config --prefix` `raptor-config --prefix` `rasqal-config --prefix`; do
  p="$p:$dir/lib"
done
p=`echo $p | sed -e 's/^.//'`
LD_LIBRARY_PATH=$p
DYLD_LIBRARY_PATH=$p
export LD_LIBRARY_PATH DYLD_LIBRARY_PATH

cat <<EOT > hash-test.c
/* Test Termfinder Query */

#include <redland.h>
#include <sys/types.h>

int main(char* aArgs) {
    librdf_world *oWorld = NULL;
    librdf_digest *oDig = NULL;
    u_int64_t iHash;
    
    oWorld = librdf_new_world();
    librdf_world_open(oWorld);
    
    oDig = (librdf_digest*) librdf_new_digest(oWorld,"MD5");
    librdf_digest_init(oDig);
    librdf_digest_update(oDig,"foobar",sizeof("foobar"));
    librdf_digest_final(oDig);
    memcpy(&iHash, (void*) librdf_digest_get_digest(oDig), sizeof(iHash));
    printf("Digest: %llu\n",iHash);
    librdf_free_digest(oDig);
        
    librdf_free_world(oWorld);
}
EOT

echo "$program: Compiling"
gcc -o hash-test hash-test.c `redland-config --cflags` `redland-config --libs`
echo "$program: Running"
echo "Expect: 9772427386146006452"
./hash-test
rm -rf hash-test hash-test.c
exit 0