You don't say how low the model is when you are hovering, but it seems you are in ground effect. The blades are generating lift and the descending air is hitting the ground which creates a cushion of air. This now becomes more dense that the air around it and therefore the model rises until it gets to an altitude where ground effect disappears.

 

I was hovering at around 2 to 4 meters when it started slowly rising to 8 or so meters, then starts sinking.

This particular soccer field is sunken into the ground, and the rest of the land area outside of the soccer field area is 8 or 10 meters above the soccer field, I am wondering if the soccer field traps cold dense air.

Could it be the ground effect cushion of air is much higher than normal under these conditions?

 

It's possible. Cold air is pretty dense and will settle on the ground.