Windows 8.1 adds new language after every restart

Up until recently, after every reset, I’d get a new language added to my language bar. And it was a strange one, Upper Sorbian.
It would always appear in my bar, but never in the language settings pane

I had a workaround that would remove the language until the next restart. It required manually adding the language and then removing it again.

Then, one day I finally clicked the Change date, time, or number formats link on the left.
Apparently, my “Non unicode” settings under the Administrative tab were for Upper Sorbian. Why? No one knows.

I changed my locale there and I made sure I clicked the Copy Settings button and copied my User Settings to my other screens (Welcome screen and New User Accounts).

Problem solved!