My previous job only lasted 3 weeks, it was a full time post but didn't work out as they felt they required something different from the role. I would like to leave it off my CV, is this wise or will a prospective new employer find out somehow?

A 3 week job can easily be left off your resume. They have no way of finding out about this job unless you tell them about it.

Thanks for that. I'm in the UK and my worry is that when they process my PAYE they'll find out from the inland revenue about my previous employment, am I worrying too much?

Yes

Is it suppose to be listed, esp if you are using a detaled list of places worked, yes,

Would most people leave it off, yes