In short, a pay raise should not be your #1 requirement with a job move. A lateral move, even one slightly downward, might make sense if it's to a ladder you actually want to be on.
His advice clarified something I had always instinctively understood. For instance, my first high school job was at a golf store. It was intentional. I thought I was going to be a marketing major in college, and thought having golf experience on my resume would be beneficial and look more professional than my friend who worked at Chuck'E Cheese.
(Note: At the time, I was under the impression that guys in marketing golfed all day. That's largely why I chose marketing. 7 years into my marketing career, I have NEVER golfed for business.)
Does your current job look great on the career ladder you want to be climbing? If not, it's time to jump to a new one.