No rivalry romance with Fenris?

I tried romancing Fenris in the rivalry path. He visited my house at the
end of Bitter Pill, and I chose all the flirt options, but he still
left anyway, without even a kiss or sex. Anyone know what went wrong? I
made sure to ask him what he wanted during Bitter Pill, chose all the
flirt options, and even helped the elf girl. I'm above 50 rivalry too.