If a boss is sufficiently difficult enough to challenging when your group is uncontested, it will be borderline impossible when your group is contested, even if you're contested by a single stealthed rogue.
If a boss is sufficiently easy enough to be done while your group is contested, it will be boring when your group is uncontested.
For an example of the first type of encounter design, consider that it took 346 wipes before method was able to get their first kill on jaina. Would that fight be possible at all if, during any of the critical mechanics, an enemy player can try to stun you? The encounter design would stop working.
The second design turns the boss into something to fight over, similar to Baron Nashor in League of Legends. The boss is dangerous, and has HP, and might perform big telegraphed attacks or weaken whoever is currently trying to fight it, but it's less of a hugely complicated mechanical skill check and more of a thing-that-drives-player-conflict. When there are no other players, then it's not serving its purpose and the fight itself feels empty.
I love the idea of open-world bosses, but I also don't want to lose my lovingly-crafted encounter design that's been perfected over the years by FFXIV and WoW. Here's a potential solution:
Keep some of the open world bosses like the above. Preserving "something to fight over" as a type of content like Baron Nashor is important.
Then, make it so that some of your open-world bosses have the following pattern: players fight over the ability to challenge the boss uncontested while everyone else has to watch. If that group wipes, they respawn, and the process repeats.
You can flavor this in any number of ways. WoW, for example, has the Gurubashi Arena. You can make it so that whichever party successfully opens the Gurubashi Arena Chest spawns the boss. Everyone else is booted up to the stands and they have to watch as the boss attempt goes down. If the raid wipes, then everyone can go duke it out again and try to open the chest for another shot. If the group kills the boss, playtime is over.
If you're fighting a shadowy demon boss, maybe before the boss spawns, their Envoy spawns, as a miniboss to squabble over like Baron Nashor. Whichever group manages to defeat the Envoy gets a limited-time, limited area buff that makes them all shadowy (curse of shadows). Other players can't interact with them in any way, and only those players can interact with the boss that's about to spawn. If they wipe to the boss, the boss goes back to the shadows, and the Envoy respawns.
There are any number of ways to flavor it, but the goal is the same! We're just trying to find a way to circumvent "if a boss is sufficiently difficult enough to challenging when your group is uncontested, it will be borderline impossible when your group is contested".
Traditional answers to the above problem are along the lines of "guard the entrance to the area" or "use diplomacy". I think these answers either underestimate how difficult encounters like Jaina were (346 wipes, people) and don't think about how you can't let a single person through (stealthed or otherwise). Or don't think about the ratio of guards to raiders that you would need for the guards to adequately protect the raiders for 346 wipes. Or how mind-numbingly boring that would be as a guard.
On the diplomacy front, this is something that a single stealthed rogue can ruin. If one upset nerd has the power to wipe your group over and over, you bet your bottom dollar they'll try over and over.
Credit to @Noaani on the Ashes of Creation forums for exposing me to the idea!