Close the Flue Vent.
Done with the fire and it is out? Shut the vent. We lose just as much A/C out the flue as we do heat when we forget to keep this small little vent shut. Consider adding some glass panels or a fireplace blocker to limit the heat loss but still enjoy the ambiance.