Here, the behavior "shaking your legs", so called fidgeting, is really hated. If you do this in front of Japanese, we will be absolutely annoyed.
It is called "Binbo Yusuri" in Japanese, "Binbo" means "poor", "Yusuri" does "shaking". The reason it called like this is that the behavior makes you look like you are poor and your mind is always occupied with financial matters.
Actually, this phrase seems to have been created in Edo era. Seen in Haiku or Senryu, which are Japanese traditional way of poem, people in that time also detested this behavior with same reason above. Incidentally, another phrase "Binbo Gami", which means god of poverty, was also made in this era. At that time, especially merchants really dreaded poverty, and what's more, there was a superstition that shaking legs conjured the god of poverty, so this behavior started to be called like this.