\frac{|1-a^2|}{5a}>3, && a\epsilon R

How do I calculate the value of a from this expression?

It is an inequality, not an equation, what you have as solution is something like "a greater than... / a in... ".

First look at signs: what happens if a<0?
Absolute values are always non negative: if 1-a^2<0 then |1-a^2|=-(1-a^2), if 1-a^2\geq 0 then |1-a^2|=1-a^2. Solve these 2 cases separately (they are II degree equation), and do not forget the constrains on a put when you split the absolute value in these 2 cases.