Well I finally stumbled on what definitely appears to be what causes the problem, but finding the solution is what I need to find next.
It appears that it's something with the download program I am using on the PC, which is downloading a DASH Video over HTTPs. It downloads in fragments and then combines the video.
Anyway, when that downloader is running, the network acts weird.
I tried doing a 1GB test download at full speed, the network acts fine.
It seems like for some reason, the downloading in fragments is what appears to be causing the network issue.
It does it with or without QoS on and it doesn't seem to matter even if you set QoS to an even lower Internet speed than 90% of your Max.