My question: How do I actually physically notice the difference between these kernels?

Generally, you don’t. You can look for some benchmark to try and find a difference between them, but if you don’t notice a difference in your day to day tasks, then it’s all the same. In my experience you should pick a kernel based on your desired experience. For my needs this is how the kernels differ:

• Generic kernel: a sane default for most regular users
• LTS: only makes sense if you’re worried about regressions in the generic kernel causing issues, and only viable if you can afford to stay behind on hardware driver updates, ie you use old hardware and/or optimal performance is not required
• Zen: sometimes better for gaming, but often indistinguishable from the generic kernel
• Realtime: rarely what you want, it sounds “faster” but it’s basically optimized for very specific use cases and if you’re not among them you’ll see the same or worse performance

Difference goes from negligible to imperceptible. They were only really necessary for back in the day specific use cases like digital audio latency.

From what I recall the completely fair scheduler (CFS) used by default on most Linux systems has a lower average latency than the RT kernel. The RT kernel just gives you more consistency, hence the CFS having lower latency “on average”

So honestly for opening Firefox it’ll probably depends more on your SSD data rate, but in theory it’ll open faster on a “regular” distribution most of the time.

Real time is good for things like audio processing where having better guarantees that a process will get its share of the CPU is a benefit.

Have you given the CachyOS kernel a try? It’s got some of the Clear Linux patches and some other custom patches, and it might have slightly better performance than the others you’ve listed here

Although expect to only really see any noticeable improvements in games or benchmarks and the like

Most of them won’t be that different when you’re not running anything that’s pushing your system to its limits. Zen might be a bit faster in games or benchmarks, RT really won’t do much unless you’re running software that needs Real-time processing (you shouldn’t use it for general use).

Hardened and zen are the only ones you might benefit from, but not really massively.

Zen and “mainline” (default/vanilla) are generally fine for “desktop use” and gaming. Zen is basically the mainline kernel with some tweaks. They are mostly concerned with latency, reducing the maximum time a process can spend blocking the processor - among other things.

This can lead to less input lag or a “smoother” desktop experience, but overall performance is as good as mainline at most. Slightly worse in some scenarios.

Hardned is a tradeoff afaik. You will stay behind mainline a bit, but get extra hardening. This can also impact performance, but rarely does in a meaningful way. If you don’t have any specific reason to use it, e.g. you carry it around on a laptop with sensitive data, I would look at other ways to harden my system first (firewall, encryption, access control, anti-virus, sandboxing, VPN…).

Pretty much the same goes for LTS, but with the focus more on stability than security.

RT is only for special applications.

I don’t think there will be a noticeable difference. Real Time kernel is intended for things like microcontrollers I believe.