Not to worry. When you make a withdrawal you use form 8606 to determine how much of your withdrawal is taxable, and that calculation requires you to think of all your IRA accounts as if they were all co-mingled into one big pot. In other words, when you make a withdrawal the tax calculation does not allow you to specify whether the money you withdraw first is pre-tax contribution, after-tax contribution, or earnings. For example, suppose over the years you made $4000 in after-tax contributions, $2000 in pre-tax contributions, and have earnings of $14000, so that the net value of all your IRA is $20K. Hence out of the $20K you have already paid tax on $4000, or 20% of it. Now, if you make a withdrawal of, say $5000, the IRS assumes that 20% of it is attributable to your after-tax contribution, and the rest is either pre-tax contributions or earnings, and so you end up paying taxes on 80% of the withdrawal. Here's a web site that explains this perhaps better than I:
How IRA Distributions Are Taxed
The reason why it is generally advised that you keep your IRAs funded with pre-tax money separate from those funded with after-tax money is that it can simplify accounting if you were to roll these accounts over to other retirement vehicles, such as your 401(k) plan at work or a Roth IRA.