In "Shut the f**k up" shut up is the phrasal verb and "the f**k" is inserted for emphasis, so in that case "f**k up" is neither verb nor noun. f**k up can be a noun or separable phrasal verb in its own right though as in "What an almighty f**k up" or "Ade Azeez through on the keeper, he's going to f**k it up". Those uses are common in English but I've only ever encountered it as an imperative in the west of Scotland and that's with a different meaning.
Thanks Diego. Not fully up-to-scratch on grammar tbh, so this is very interesting. The thing I'm complaining about is when, for example, someone on RangersMedia says something unpopular (such as 'Hibs actually played well') and someone else will respond with 'f**k up'. So are they issuing it as a command, or telling the person that they are a f**k up? It's not something I've heard in real life, but it annoys the tits off me when written