The patch at 81833a1c is necessary on an arm64 system running librtlsdr-dev, otherwise you get the error
Reading samples in async mode...
Allocating 15 zero-copy buffers
The workaround I did was to install the .deb file from Sid on top of my Ubuntu 20.04 installation, which I hope to undo as soon as I can. It also worked to build that library from source.
Other than that, my experience with Ubuntu 20.04 on the Pi 4 running arm64 has been just fine. I'm looking forward to updating the rest of my Pi menagerie, some of which is stuck on old Debian Jessie.