[1.7.x] Fixed #22650 -- Fixed regression on prefetch_related.

Regression from f51c1f59 when using select_related then prefetch_related
on the reverse side of an O2O:

Author.objects.select_related('bio').prefetch_related('bio__books')

Thanks Aymeric Augustin for the report and tests. Refs #17001.

Backport of bdf3473e from master