Probabilistic programming languages (PPLs) provide a principled framework for expressing Bayesian models and automating inference, yet writing correct programs remains challenging. Large language models (LLMs) can accelerate this process by drafting candidate programs, but their outputs frequently fail due to semantic and syntactic errors. We introduce RefineStat, a framework for reliable probabilistic program synthesis in PyMC using compact, open-weight models. By enforcing statistical semantics during generation and applying targeted refinements guided by Bayesian diagnostics, RefineStat produces programs that are both syntactically sound and statistically reliable. Across different datasets and multiple small open source models, RefineStat consistently improves run rate, convergence behavior, and predictive stability compared to unconstrained or syntax-only baselines, while remaining cost-efficient.