lsmod will tell you who is using the module. Remove the user module first.
First, find out, which other modules use the module sdhci:
lsmod | grep sdhci
You will get a list like this:
module size used_by
and then try unloading these modules (used_by) before or together with the module you want to unload.
