Optimise walking by matching directories only once

Directories were matched twice before: once when checking the parent
directory's entries and once when loading rules. Now both are done in
one step.