Is your family Catholic, missionary, baptist, etc....
It's not really important, I am just curious.
The way you and your family tackled the issue is pretty accurate with what needs to happen. However, lighting candles has no business in removing spirits really...and very easily can turn into much of the rituals found in the occult. You guys were spot on with what needs to happen other than that. I'm only going on what you told me here, but I think things fell short in a couple areas. First of all, to say that the spirit had no right there was not true. If it did indeed come through the playing of the ouija board, it DOES have a right there and it does have a right to your brother. Any involvement in the occult and it's practices needs to be very specifically dealt with. All sin in the occult needs to be repented of and renounced. This is more so important with your brother, but it still needs to be brought up. Only after repentance can it be said that the spirit no longer has a right to be there.
Another thing that needs to be kept in mind is that some spirits are stronger than others. This isn't myth, it is from Jesus directly. Some spirits can only be removed through much prayer and fasting. Spirits of the occult and witchcraft can have a strong hold on a family. It is very possible this demon is simply being stubborn. However, stubborn or not, in the end it still has to go.
One more thing I might mention is that prayer and authority THROUGH Jesus Christ is a must. This is how Jesus taught to exorcise spirits. You did mention that used the name of Jesus in the prayers, which is correct...but it has to be to a point where the person knows that it is true. It can't be used in a timid way or in a way where you are HOPING it will work. You need to KNOW that Jesus is in control and through His name there is COMPLETE authority. As I said, some spirits will hold on stronger than others. Don't give up on this yet.
Here are my suggestions. I recommend you get your pastor involved..and if he doesn't seem to know too much about the subject I recommend you find a ministry nearby that does. This is also a very important time for you and the rest of your family to draw close to God together. Make the home very uncomfortable for the spirit. Fill the house with praise and worship. Have bible studies and prayer meetings there for a couple weeks. This will give the home a different atmosphere. Also, I don't anything about your brother's situation..but prayer for him just as you pray for your home. It still needs to be COMMANDED to leave. It can simply be said "By the authority of Jesus Christ and by His blood we COMMAND you to leave this home and this family!"