XBox One S, 4k HDR

I bought a new TV, I know it's not true 4k on a One S, but looks good...HDR colours are great, but in the night it's extremely dark, sometimes like a full black screen...if I wouldn't know this game so well it would be almost unplayable...tried different settings, TV and ingame...when I find a good setting for the night, it's too bright at daytime...now I found a setting that works, ok, not ideal but ok...could be better...
Tried it with Forza Horizon 4 and I don't have problems there...
So my question, does anyone have experience with a XBox One S and HDR? Did you have the same problems? How did you solve it? Is it maybe better to turn HDR off on that console?