Shutting down X Server

[Loriel]

It's not nice to tell someone to change a random 5 to a 3

Rather, you should change the standard runlevel at boot to the one that does not automatically bring up X, so that you can do your changes.
This, of course, is done by changing a 5 in /etc/inittab to a 3.

Also, it does not truely work everywhere.
The startup of X could be done via a the (hidden) .profile file in one's home directoy containing the line "startx".
Gentoo Linux has named runlevels "boot", "nonetwork", "default" and "reboot". So you would rather need to remove the command to start X from your current runlevel.

Unrelatedly, your window manager should provide an option to shut down X, or you can just press CTRL-ALT-Backspace. But your system could be configured to log you out/shut down after that