Can your "driving privilege" be taken away in a state you DO NOT hold a license in?

For all intents and purposes I think so. This link to a law firm explains some of the factors involved in state reciprosity agreements or compacts as this firm calls it.
If your license is suspended or revoked in the state in which you are licensed to drive, you have no right to drive in any other state, because you no longer have a valid driver's license.. I do not think that you can hold an automobile driver's license in more than one state at a time. So I guess it is fair to say that any state that has a reciprocity agreement with the state in which you are licensed, can cause your state to suspend or revoke your license, under certain circumstances some of which are discussed in the link.
Driver's License Compact- Reciprocity with other states | Shaffer & Engle Law Offices, LLC | Harrisburg Pennsylvania (http://www.shafferengle.com/Traffic-Violations/Drivers-License-Compact-Reciprocity-with-other-states.shtml)

Yes, this actually happened to me. I lost my license (this was before photo IDs) many years ago. The person who found it was a trucker and used it as ID when he got stopped for speeding in another state. I was not about to travel to that state just to fight the ticket and the state would not accept other proofs that it was not me, so they suspended my license.

I also have a personal anecdote:
I got ticketed for speeding in Vermont in the early '80s. I figured I was probably never going to return there, so I didn't pay the ticket. About 20 years later, after the states entered into those mutual compacts, my home state (Alaska) notified me that unless I paid the ticket my license would be suspended. I paid the ticket. :)

Yes... I live in NC. I wus arrested in SC. The state of SC revoked my driving privileges in SC for 1 year. No other states were affected... Drive safe!