While it may look like a bug, this is the actual behavior and it is like that because there are 2 types of macros in XWiki: inline and standalone.
For examplle, the mentions macro is an inline macro that can be used anywhere in the content.
On the other side, the standalone macros include formatting macros, like the box macro which is not expected to be used inline, in a paragraph or as part of a list item.
In order to make this difference more explicit, this topic was recently created to help avoiding any confusion.
Additionally, I was not able to reproduce the issue on XWiki 17.1.0 with your first test example. I edited a page in the wiki content and placed the macro without adding lines but I didn’t get any formatting mess, only the display was improper, but expected as the macro was inline and not standalone (separated from the rest of content).