If you're growing potatoes in an area that small children or pets can easily get to, it is wise to remove the potato flowers to prevent them from developing into toxic green potato berries.