How should the cost accountant of a small company that manufacture plastic boxes assign factory overhead to plastic boxes made in January? Should only factory overhead bills received and paid in January be assigned to the production of boxes in January?

No.

Accrual base accounting. Record all expenses accrued, regardless if you have received and/or paid the bill for it.