xserver: libdrm is not needed for tinyx

libdrm is only needed for the modular server, and as libdrm
needs largefile support and tinyx doesn't, this dependency could
cause a build failure.

Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>