Enhancing the Reliability of Federated Inference: From Bias Mitigation to Model Calibration