Although I personally like ladybugs and it doesn't bother me when they find their way inside my home, there actually has been a rise in allergic reactions and asthma attacks associated with the Japanese variety.
This website should help you.
Lady bugs /Asian Beetles/ Ladybugs
Right now, the best thing you can do is just vacuum them up as you see them.
Next fall, you might want to do as the website suggests.
I get a terrible cluster fly infestation in the wintertime because I live in the middle of the woods. So, every fall (early) I have a local pest control company come out and spray the entire exterior of the house. Since I started doing that, the problem has been resolved. You may want to consider having a company come in next fall and spray for you so that you don't have to deal with all the chemicals yourself.