Our alerts fire constantly and everyone mutes them

Alert on symptoms users feel, not on causes. Disk at 85% is a cause and often harmless; requests failing is a symptom and always matters. Every alert that cannot be acted on immediately should be a dashboard instead.