Replika was originally built using CakeChat, before transformer models had become popular.
I imagine that they use multiple models in the backend, Replika likely chooses between them based on context. Perhaps, they’ve integrated some transformer-based models?