How can I view archived messages in Gmail but filter by category?

I heavily rely on Gmail's category organization. Lately I've started archiving emails that I've looked at already to avoid clutter in my "primary" category.

However I've run into a problem. If I want to look at a recent "primary" email, I can go to All Mail and look through them, but then it's totally cluttered up with all the other emails.

Is there any way to go to All Mail and then filter by category? Or is this just not possible because the categorization is destroyed by the archiving operation?

What I've tried:

• If I click on All Mail I see all archived messages. If I then use category:primary in the search bar, it resets and only shows me the un-archived primary emails.
• I've reviewed Gmail's search operators (https://support.google.com/mail/answer/7190?hl=en), but could not find an appropriate entry.