NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver.

NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

相关gpu驱动问题 可以通过卸载删除相关文件重装

Even with those commands, the issue wasn’t solved.
Eventually, the fastest way to fix 2 machines with a package manager is to purge all Nvidia & Cuda,did it by:

sudo apt-get remove --purge '^nvidia-.*'
sudo apt-get remove --purge '^libnvidia-.*'
sudo apt-get remove --purge '^cuda-.*'

Then after it’s clean ran that:
sudo apt-get install linux-headers-$(uname -r)

From here - it’s the same for all VMs:
Download latest run file from Nvidia site, and run it, accept if needed to upgrade current, or install from scratch.
The driver is back to work.

The issue was started after did some updates, and the Linux kernel was changed.