1 GB recommended for 4K — required for Smart Zoom/Smart Scale/Smart adaptive deinterlacing, and GPU-accelerated video processing.
For hardware rendering (NVEnc):
• GeForce 600 series onwards (6xx, 7xx, 9xx, 10xx)
• Quadro Kxxx, Mxxx and Pxxx
Requires an OpenCL-enabled GPU and Catalyst driver 11.7 or later with a Radeon HD 57xx or higher GPU. If using a FirePro GPU, FirePro unified driver 8.85 or later is required.
Intel Skylake or newer processor required for QSV accelerated 8-bit HEVC/AVC decoding and encoding, Intel KabyLake or newer processor required for QSV accelerated 10-bit HEVC encoding and decoding.
Hopefully someone who has done some "real world" testing can add provide anecdotal information (which is what I believe you were asking for).