Is the PS5's SSD specs that much of a game changer compared to Xbox Series X?

I said LIKE an L4 cache. LIKE ffs. My point was you don't have to LOAD data through a file access API designed for slow spinning rust. You MAP the data into virtual address space and the data gets pulled in using DMA when accessed as if by magic. And you hint the coming accesses to make it faster than usual VM file mapping.

>Most of the time spent loading when you have an SSD is the CPU processing the data

So going by this logic, a faster processor should give better SSD performance?

Yes, right up until there's another bottleneck.
The console's trick appears to be using custom DMA hardware dedicated to the task. Plus the static hardware means less "cooking" of the data at runtime since it should in theory be identical every time.