GDB remote debugging: defer loading of .so shared object libraries until break

I'm trying to debug a remote application with gdbserver. GDB has a feature to load shared libraries used by the application (and their debug symbols) from the target system, which is usually super handy.

Unfortunately, I'm running into a problem where the delay caused by automatic loading of shared objects is causing me trouble. The sequence looks like this:

1. Application is running in its idle loop.
2. I launch gdbserver --attach on the remote system.
3. I run gdb and target remote on the build machine.
4. gdb attaches and retrieves all currently loaded shared objects and their debug symbols from the remote system. (Great!)
5. I set my breakpoint and run continue.
6. I trigger the event that should lead the application to reach the breakpoint.
7. The application dynamically loads a few additional shared objects (for PAM in this case, though it's not relevant to the code I'm trying to debug).
8. GDB immediately fetches the new shared objects and debug symbols, seemingly pausing execution while it does so.
9. The delay triggers a timeout in the code, and as a result by breakpoint is never reached.

I want to tell GDB to allow the application to run all the way to the breakpoint without stopping, and not try to load any additional shared objects along the way.

Reading the documentation, I thought I should be able to run set auto-solib-add off in GDB before attaching to the remote debug target, and then load them later (if needed) with sharedlibrary. Unfortunately, that doesn't seem to be sufficient: while that option does appear to cause GDB to refrain from fetching the split debug symbols, it will still fetch all of the loaded *.so files themselves, which results in the same problem.

Is there any way to prevent GDB from fetching shared object files at all until requested? (With the possible exception of libpthread.so.0, libc.so.6, and ld-linux-x86-64.so.2, which I understand may be necessary for GDB to properly interpret backtraces.)

Currently I'm working around the problem by modifying the code to raise SIGSTOP at the start of the section I want to debug, running it without the debugger until it stops, and then attaching the remote debugger. This works, but is a bit annoying.

Note: I know it's theoretically possible to clone the filesystem of the target locally and pass it to GDB with set sysroot for faster symbol loading, but that would be very inconvenient in this case.