Timing is everything. Are you timing your wants for AFTER you think/know he has been masturbating? Are you accusing him of being too tired from masturbating when he rolls over and goes to sleep when he may just be tired and need sleep? Why didn't you go downstairs and let him know you were in the mood, too. Then you start the game playing and getting back at him. THAT will cause problems in your relationship. How many of his sessions are because you said no because you were upset with him for what he did (or you thought he did) earlier?
You said that you have told him you were interested in watching with him, but have you asked him to join you in watching? Is the only time that you initiate intimacy when you think he won't participate?
Do you masturbate or do you expect him to take care of all your needs? Quite, frankly, I am a firm believer in being able to pleasure yourself. It is how an individual learns what his/her own needs are both physically through touch and mentally through fantasy.
You are making yourself feel inadequate and insecure. From what you have written, he has sex with you more than he views porn and, maybe, masturbates (viewing does not mean acting every time.) However, you want to accuse him of not being into you because 2-4 times a week (not many times a day like some women deal with, not even daily) he masturbates.
Talk with him.