I may have figured out the issue.
I think my ISP must have changed something with the config files of modems or something similar with their systems.
I turned QoS off entirely and did a bunch of speed tests. (To be safe I also verified all coaxial connections to verify nothing was loose).
For years now, my 50/6 would typically test about 52-53/6.5.
For some reason lately, I am lucky for the speed test to reach 51.5 now.
The ClearOS Speedtest came up with 47820 so I did 90% (43038) and 5742 for upload.
It actually appears to be working but I figure I better let time pass before I consider it actually solved.