Here are three principles for when to engage symbolic mechanics (like dice or saving throws) in fear and charm situations in adventure-style TRPGs (like D&D):
I always wanna mind the mood. Description, tone of voice, situation, resources, danger can all make a situation tense or scary. It can get wrecked by whipping out the dice.
I try to normalize some mechanics by speaking of them so often and so bluntly that they become part of the interface, part of the conversation, almost making them invisible through their overuse. They find something, whether it’s some pocket lint or the holiest of grails? I say the item size. They try to defend themselves? I say the save DC or HP cost.
In one of WotC’s haunted house modules, there’s a voice that tries to scare the characters, which prompts saves, and… those who fail get disadvantage on ability checks for one hour. That is weaksauce. It’s not worth wrecking the mood with dice rattling for penny stakes. Just skip it. Have the voice but don’t roll for it. Just let it be a spooky moment for its own sake, not for an extrinsic motivator.
This is something 5e seems to already be pretty good at out of the box, but I wanna note it explicitly because I appreciate it and I want other games to follow suit:
Whenever there’s a fear save or charm save, it’s because someone is using a spell or magic item or monster ability to manipulate or enhance abilities. Never just because “you see something scary” or “some handsome guy is smoothtalking you”.
The dice are guns on the table. Our own conversation is more than enough to handle interaction with the mundane world, even when meeting horrors or joys. We only pick up the dice to deal with the liminal.
Let’s say Alice and Bob make their saves and Carol and Ted fail it. So Carol and Ted fall a spell that causes them to run away, or to attack each other, or put their own heads in a trap or something. Instead of telling Carol and Ted what they have to do, the trick is this: turn to Alice and Bob and tell them what Carol and Ted are doing. Instead of asking for who failed their save, ask for who made it.
I lucked into this technique and I’ve been following it strictly since. Sometimes it falls flat and sometimes it’s bone-chilling. Often it’s just the least-awkward, least-bad way to handle it.