Here's why:
* Elephants were extremely rare in Europe during the medieval period. They were expensive to acquire, transport, and care for. While some European rulers did have elephants (usually as gifts from foreign rulers), they weren't common.
* Polar bears are native to the Arctic region. They would have been impossible to keep alive in the climate of medieval England.
* The Tower of London was primarily a fortress and prison. It was not designed to house exotic animals.
While elephants and polar bears weren't kept in the Tower of London, the Tower did house other animals like lions and leopards. These animals were kept for display, as symbols of power, and sometimes for entertainment.
The legend of the Tower housing a menagerie might stem from the fact that royal menageries did exist in England, but they were not located in the Tower of London.