OpenVZ VPS is similar to advanced CloudLinux based shared hosting where resources are containerized and shared. KVM VPS is much like a dedicated server where resources are dedicated and private. KVM provides true isolation and is generally more stable than OpenVZ.
KVM VPS and OpenVZ's primary difference is that KVM VPS is a virtual machine powered by kernel virtualization (also called hardware virtualization). OpenVZ VPS is not a virtual machine; it is just a container.
KVM is true virtualization; each VPS is independent of the host node (the parent server where the VPS is hosted). OpenVZ is container based virtualization (also called OS level virtualization) that relies on the operating system and the host node's kernel.
You cannot install a custom operating system or modify the kernel on OpenVZ VPS. On the other hand, you can fully customize the kernel and install any operating system on KVM VPS.
The memory of an OpenVZ VPS is not entirely physical RAM, some of it is swap, and some is RAM. The memory of a KVM VPS is entirely physical RAM. Resources are fully dedicated and private.
All OpenVZ VPS share the same underlying operating system, kernel, and the pagecache of the host node. Everything runs under a single kernel. KVM has no restrictions. Each KVM VPS has its own kernel and operating system. It does not share them with other VPS or with the host node.