The way I look at it, this deal is a perfectly acceptable move to improve our team in the short run.
1) Yes, we aren't actually paying Kemp $5 million a year, however accepting that Olivera's money was a sunk cost does in fact leave us with about $5 million a year in extra cost to add Kemp to the roster while giving up no assets in return. If we had cut Olivera this year, then signed Kemp for a 3 for $15 million deal, I'd find it perfectly defensible. 2 years ago the expected $ value per 1 WAR was $6 million. Even without adjusting for inflation of that number in two years, I think it's perfectly reasonable to expect Kemp to average around 1 WAR per year.
2) Kemp might be a little more valuable than he seems on paper. If you exclude his awful May performance, he's averaged about a 135 wRC+ this year. I'll take that any day from the RF spot. If we can team him up with Mallex and Inciarte in the short term to mitigate his defensive limitations, he might be an actual useful piece for us. No, he won't be worth his contract, but again we have to look at Olivera's deal as a sunk cost.
Edit: nsacpi pretty much beat me to my entire point in far fewer words.