How often you anneal is up to you and your goals. If you want to keep things as consistent as possible, anneal every time. Annealing before every load will not harm the brass.
If you just want more loads from you brass, every 3-4 firings is about right.
Every time you fire the cartridge, the brass expands. Every time you resize the brass, it contracts. This will make the brass brittle over use and will eventually split or crack around the neck or shoulder area. You can use a thin piece of metal to see this.... use a piece of shim stock, and old feeler gauge, or maybe a piece of a soda can. Bend it enough where it has a crease. Now work this crease back and forth.... at first it bends back and forth but after so many bends it will crack or split at the crease. The crease on rifle brass is the neck and shoulder area, annealing will keep the brass stretchy in this area to get more loadings.
Not all rifle brass is created equal.... some brass is just more brittle than others. With one brand, you might only get 3 firings, no matter whether you anneal or not, while other brands may give you several loadings without annealing.