Have you finished the Hinterlands yet?
I decided I would romance him in my second playthrough. My aim was to be in the relationship before Halamshiral because I had heard there was a cutscene with the LI (which I had missed on my first playthrough with Dorian). Also, I knew I would be freeing the mages so I needed his approval as high as possible. What I did was get him as soon as I had the power to go to the Storm Coast. I took him with me to the Hinterlands because I had noticed, in my first playthrough, that some little quests increased his affection. And did they! There were three full approvals for riding the horse provided by Dennett, one for getting Speaker Anais to have her organization collect info for the inquisition, getting Ritts as a spy for Leiliana (this is what I remember), taking the flowers to the grave for the widowed elf in Redcliffe, and a bunch of slight approvals when doing other quests. I did the Fallow Mire and Storm Coast before I went to Skyhold as well, but I don't recall approvals there. My judgment of Chief Movran the Under (exile) cancelled out IB's disapproval over freeing the mages. His approval was maxed at that point.
Oh wait, I did get the Blades of Hessarian at the Storm Coast. That sounds like something IB would have appreciated.