Ever since I can remember I have loved to feel silky items in between my fingers. My mom has told me stories about me going up to women when I was very young 3-4 and grabbing their slip under their skirt and rubbing it between my fingers. Now I will grab anything that looks silky sheets, pillow cases, guys basketball shorts etc. Just curious if this is classified as anything or if anyone else even does this?

Does this habit adversely affect your life?

Right. If it doesn't adversely affect your life, it's a like, a preference, a habit, an interest, a hobby.
Otherwise it's an addiction, a compulsion, an obsession....
Sure, out of 7 billion people in the world, there are plenty who like to touch silky things.
Silk has been a favorite of royalty since the Chinese discovered the silkworm.
Now most people like it, and we have real silk and fake silk.
Maybe you could take some textile classes and become an expert in all silky materials, and have a very happy career.