I've recently had a new high performance muffler system installed on my Honda Civic, and I've noted that a lot of other tuners don't have a catalytic converter. What are the differences and pros & cons of not having one?

Other than being a federal offense to remove a cat converter and operate on the road and cause more CO and hydrocarbons, nothing. I believe is costs $2G + if you are caught.

Whilst I currently live in Kansas where emissions testing is not done I do believe there is no consequence here... but for the sake of not having any money ill leave it with the cat...
But I have heard that it may add horsepower... is this true ?

It is a Federal Offense to alter, remove or replace an emission system component that does not meet the minimum Federal Emission Standard. If you have a car accident, because you altered the safety equipment of the vehicle(yes, emissions are now considered safety equipment, you could possibly face having your insurance company refusing to pay a claim. The small gains that are realized from removing the cat system is minimal.
Keep the system as original as possible.

I said FEDERAL offense, not state. It is of consequence if caught.

Well again... as the limitations of money and the fact that I like the way it sound... ill leave the cat on