If You Go-Around On A Visual Approach Under IFR, Do You Need To Contact ATC Immediately?

Visual approaches are usually simple. But what if you're flying a visual approach under IFR to a non-towered airport and have to go-around?