A long-held family vacation property was deeded to the siblings years ago.
When selling shares to another sibling can I transfer the proceeds into an IRA
Account rather than pay capital gains?

If payment is made in installments when are the capital gains paid?

Are you asking whether you can use the proceeds from your sibling will pay to fund your IRA? Sure you can - up to the annual limit of $4K (or $5K if you are 50 years or older). Can you somehow avoid capital gains on the sale? No. Can you transfer your portion of the property into your IRA, then sell it and thus avoid paying capital gains? Unfortunately not - you aren't allowed to transfer property you already own into your IRA. Here's an article that gives a good overview on the subject of real estate in an IRA:

Ground your retirement fund with real estate - MSN Money

If the sale is on an installment basis, then the gain is recognized pro rata based on the amount of cash received. I believe it is Form 6252 that applies for installment sales.

You do not "roll" it over, you can put that money into an account and take that credit. You will still have to claim the money from the sale but will be able to show an equal amount going into a retirement account