It means that Soris did loved her until she was taken by Tevinter slavers and he abandoned her over a human instead of tracking them down and rescue her from those barbarians. He is a coward and a weakling.
Soris wasn't in love with her, he'd only met her briefly before they were meant to be wed, when Vaughn abducted her.
Dialogue between the City Elf and Soris during the rescue mission make it clear that while they want to rescue the other women, it was Shianni who was the most important person to them to get back, since she was their cousin and they'd grown up together.
And as I said before, Soris has no idea whether Valora was dead or enslaved, because he was stuck in a dungeon for months.
A lot of people died in the purge or were taken to Tevinter. It's even commented on during the "Unrest in the Alienage" quest that because they didn't realise that the disappearances weren't because of deaths from the "plague", but because people had been sold into slavery, it was impossible to know how many were taken, where they had been taken to, or have any hope of getting them back.
What is Soris supposed to have done exactly, gone to Tevinter and search the entire Imperium for her with a fine-tooth comb? Good luck trying that, he'd not get within 100 miles of the Tevinter border without ending up sold into slavery himself for all his troubles?