Should developers listen to their most loyal fans? Or the entire audience? (a roguelike problem)

I was part of a certain community on discord for a roguelike game. This game was still in early access so there were lots of suggestions and discussions on balance. And one of the trends I noticed was that the most "loyal" gamers who spent large amounts of time on the server had a certain way of looking at things.

Challenge was always welcome. Most (though not all) suggestions suggesting lowering of difficulty were met with absolute rejection by that community. Recommended changes to some mechanics were met with the "but that would change the point of the game" line. I felt differently, of course, but I couldn't help but wonder; should the dev team listen to this hardcore audience? Even if they ended up making the game too frustrating for normal players? Should they stay true to that original view that's shaped?

I found a similar trend in another community, where I heard players referring to certain difficulties as "not that hard". I know some players who face the toughest challenges and are like "it was ok".

I've always considered myself a somewhat skilled player, and I've come to realize there are players there who astoundingly more skilled than I am. How do developers face this issue? I'm curious to your thoughts and input.

NOTE: I didn't want to mention it as it might alter the perception of the post, but I'm still interested; the first game was Risk of Rain 2, the second was Dead Cells.