[1.5.x] Fixed #16436 -- defer + annotate + select_related crash

Correctly calculate the ``aggregate_start`` offset from loaded fields,
if any are deferred, instead of ``self.query.select`` which includes all
fields on the model.

Backpatch of 69f7db15 from master.