I have tried to solve this problem, but I can't figure it out. I think I must be setting it up wrong.

A picture frame holds a picture that is 6" by 4". The frame leaves a border that is equal on three sides (x) but larger on the fourth (3x - 0.5). If the total area is 80.5 sq in, solve for x.

I thought using length x width = area would give me
L = 3x - 0.5 + 6 + x = (4x + 5.5)
W = x + 4 + x = (2x + 4)

So

(4x + 5.5)(2x + 4) = 80.5

but plugging the simplified form into the quadratic equation does not give me an answer. What am I doing wrong?

You have perhaps wrongly interpreted the question. In your work, it shows that two the thicker border is at one of the lengths. It could be at one of the widths. I tried this and got x as 1.45.