On my 2004 Pontiac sunfire my brake pedal vibrates when braking at high speeds. Is it time to replace or should I be worried at all?

You need brakes and rotors. Its only going to get worse..

Install new rotors and premium ceramic brake pads (e.g. Wagner ThermoQuiet). Ensure the caliper pins are cleaned and lubricated with silicone grease. Clean all rust from the face of the hub. This is a good time to also change the brake fluid.