Surprising Link

| No TrackBacks
English: Input and output file types of the li...

English: Input and output file types of the linking process. (Photo credit: Wikipedia)

Just when I thought the g++ compiler and linker couldn't surprise me, it sneaks around and breaks my mind. This time it has to do with linking against shared libraries and what it does with global objects found therein.

Here are the source files:

Here's a zip file containing all the source files for the project: tsproj.zip

Pretty straight forward, really. There is a set of functions in trace.cpp that push, print and pop strings from a globally defined 'history' std::deque and it is put into a static (.a) library file. ts1 and ts2 merely add another layer on the interface and are shared libraries (DLLs) that statically link in the trace library. tso.cpp uses ts1, ts2 and, if trace.h is included, trace to push a few strings on and print them out via dynamically linking against ts1 and ts2 and statically linking the trace.cpp library. tsd.cpp using dlopen and dlsym to load ts1 and ts2 at run time, rather than link time. It also can optionally link directly against the trace library.

Here's what I would expect both tsd and tso to print out:

 $ LD_LIBRARY_PATH=. ./tsd
print history1:
 This is shared 1 history
 This is local from the shared 1 library

print history2
 This is shared 2 history
 This is local from the shared 2 library

print local trace history:
 this is line 1
 this is line 2
 this is line 3
 this is line 4
 this is line 5
 this is line 6
 this is line 7
 this is line 8
 this is line 9
 this is line 10
 this is line 11
I would think that each standalone entity - the app and the 2 different shared libraries - would get their own copy of the history deque defined at the global level in trace.cpp and thus you would get the above output

But instead, it all depends on the order the .so files are linked in, whether the app uses trace.cpp and on the mysterious -Bsymbolic flag, with many of the options crashing on exit! The only time it works like the above is if you use trace.cpp, load the .so files at run time, and use -Bsymbolic.

If you don't use -Bsymbolic:

$ LD_LIBRARY_PATH=. ./tsd
print history1:
 this is line 6
 this is line 7
 this is line 8
 this is line 9
 This is shared 1 history
 This is shared 2 history
 this is line 10
 This is local from the shared 1 library
 This is local from the shared 2 library
 this is line 11

print history2
 this is line 6
 this is line 7
 this is line 8
 this is line 9
 This is shared 1 history
 This is shared 2 history
 this is line 10
 This is local from the shared 1 library
 This is local from the shared 2 library
 this is line 11

print local trace history:
 this is line 6
 this is line 7
 this is line 8
 this is line 9
 This is shared 1 history
 This is shared 2 history
 this is line 10
 This is local from the shared 1 library
 This is local from the shared 2 library
 this is line 11

*** Error in `./tsd': free(): invalid pointer: 0x00007f4f80cd8660 ***
======= Backtrace: =========
...
Aborted (core dumped)
As you can see, this is really weird. First off, loading the second library seems to reset the history deque. And they all share the global history deque. And then it crashes on exit, because the deque's constructor is called multiple times. Irregardless of how you feel about sharing the global variable between the calling program and the shared libraries, it certainly shouldn't crash it on the way out as it tries to call the destructor two or three times.

The -Bsymbolic flag is, according to the ld man page:

       -Bsymbolic
           When creating a shared library, bind references to global
           symbols to the definition within the shared library, if
           any. Normally, it is possible for a program linked against
           a shared library to override the definition within the
           shared library. This option is only meaningful on ELF
           platforms which support shared libraries.
I have no idea why it is called 'symbolic' but basically if you use it when you build your .so file, it says to always use its own local copy and to never use the copy of the program linked against it, which seems to be the default (I won't call it a normal) case.

And here is the output if you run the tso program, which links against the .so, instead of dynamically loading them, like tsd does:

 $ LD_LIBRARY_PATH=. ./tso
print history1:
 This is line 1
 This is shared 1 history
 This is line 2
 This is local from the shared 1 library
 This is line 3
 This is shared 2 history
 This is line 4
 This is local from the shared 2 library
 This is line 5

print history2:
 This is line 1
 This is shared 1 history
 This is line 2
 This is local from the shared 1 library
 This is line 3
 This is shared 2 history
 This is line 4
 This is local from the shared 2 library
 This is line 5

print local trace history:
 This is line 1
 This is shared 1 history
 This is line 2
 This is local from the shared 1 library
 This is line 3
 This is shared 2 history
 This is line 4
 This is local from the shared 2 library
 This is line 5

*** Error in `./tso': free(): invalid pointer: 0x00007f2d043e3660 ***
======= Backtrace: =========
/usr/lib/libc.so.6(+0x72ecf)[0x7f2d040b0ecf]
....
Aborted (core dumped)
So you can see where, without the -Wl,-Bsymbolic passed to the link line of the two shared libraries, they and the calling program all share a single global instance of the history std::deque and it still crashes on exit. Ouch.

It is still confusing in the tso (which, remember, links against the .so libraries) when we use -Bsymbolic:

$ LD_LIBRARY_PATH=. ./tso
print history1:
 This is line 1
 This is shared 1 history
 This is line 2
 This is local from the shared 1 library
 This is line 3
 This is line 4
 This is line 5

print history2:
 This is shared 2 history
 This is local from the shared 2 library

print local trace history:
 This is line 1
 This is shared 1 history
 This is line 2
 This is local from the shared 1 library
 This is line 3
 This is line 4
 This is line 5
At least it doesn't crash! But in this case, the calling program and the first shared library that is linked against, share an instance of the history deque, while the second one gets its own copy. Which seems to be another bug, I'm pretty sure.

If I don't include the trace interface in the calling programs, things behave more or less as expected when using the -Bsymbolic:

$ LD_LIBRARY_PATH=. ./tso
No trace.cpp included

print history1:
 This is shared 1 history
 This is local from the shared 1 library

print history2:
 This is shared 2 history
 This is local from the shared 2 library
But if I build the .so files without -Bsymbolic, things get weird again in the static linked version:
$ LD_LIBRARY_PATH=. ./tso
No trace.cpp included

print history1:
 This is shared 1 history
 This is local from the shared 1 library
 This is shared 2 history
 This is local from the shared 2 library

print history2:
 This is shared 1 history
 This is local from the shared 1 library
 This is shared 2 history
 This is local from the shared 2 library

*** Error in `./tso': free(): invalid pointer: 0x00007f9caecc5660 ***
======= Backtrace: =========
/usr/lib/libc.so.6(+0x72ecf)[0x7f9cae992ecf]
/usr/lib/libc.so.6(+0x7869e)[0x7f9cae99869e]
[...]
Aborted (core dumped)
As you can see, even though the calling program no longer pulls in its own copy of trace, the .so files still share a copy of the global and it still crashes. The dynamically loading version works just fine, even without the -Bsymbolic, as long as it doesn't use trace itself.

So after all this, there is only one recommendation I have - always build your .so files with -Wl,-Bsymbolic. To me, it only makes sense that programs and shared libraries have their own copies of globals. Of course, if you don't want this to be true, you are currently screwed, as they all crash. And even despite that recommendation, it doesn't work correctly in some cases anyway. Ugh.

For the record:

$ g++ --version
g++ (GCC) 4.8.1 20130725 (prerelease)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The Use of -Bsymbolic (Linker and Libraries Guide)

Enhanced by Zemanta

No TrackBacks

TrackBack URL: http://linux.amazingdev.com/cgi-bin/mt/mt-tb.cgi/562

About this Entry

This page contains a single entry by Jonathan published on October 10, 2013 10:07 AM.

Drop The X was the previous entry in this blog.

Getting The Right Sound is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.