Select a leader, rest are followers.
Visit by a follower:
If switch A is on, toggle switch B.
If switch A is off and you have not previously toggled switch A, and you have previously seen switch A on, toggle it on, otherwise toggle switch B.
If switch A is off and you have previously toggled switch A, toggle switch B.

Visit by the leader:
If switch A is off, toggle it on.
If switch A is on, toggle it off. If you did not turn switch A on during your previous visit, increment the count of prisoners.

Switch A states are used by leader for counting, switch B is there so that follower can toggle it if they have already toggled Switch A.

Unknown initial state is solved because only leader would initially toggle the switch A.

Don't ask how I came up with the solution, you probably would find out soon. :P