It appears that NVIDIA has limited the number of NVEncoding streams on consumer GPUs. Guess it is so people have to buy the more expensive professional cards.
Fortunately, the limit is only applied to the driver, and there is a patch available that let’s us bypass the limiter.
https://github.com/keylase/nvidia-patch
Install Patch
This assumes you already have the driver installed. If you do not, or run into issues with the commands below, refer to the above link.
Download the tool
https://github.com/keylase/nvidia-patch/archive/refs/heads/master.zip
wget https://github.com/keylase/nvidia-patch/archive/refs/heads/master.zip
Unzip the file
unzip nvidia-patch-master.zip
Run the patch script
cd nvidia-patch-master sudo bash ./patch.sh
And we are finished!
Further reading
NVIDIA has a matrix of which cards support how many streams etc.
https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
And while we are on the topic of artificial limits, check out the vGPU license bypass