Do magic systems in fiction need rules? I don’t know. But I do know that Brandon Sanderson, an author that has written a lot of books, has three (four) rules for how to write magic.
What is magic?
Magic is whatever awesome or unusual stuff you have invented for your story. By this definition, the made-up science in science fiction is magic. The powers of superheroes are magic too. Anything supernatural or strange – it is all magical from a storytelling perspective.
Writing about Jedi knights fighting a long time ago and far far away, or wizards and dragons, or the X-Men fighting bad guys is, from an author’s perspective, the same thing.
Sanderson’s rules, it turns out, are a really useful tool for writers to think about what they are writing and the emotion they will get from their readers. They are really just a way to think about good foreshadowing. They work for discovery writers (pantsers) as well as planners.
Brandon Sanderson explains these rules far better than I can so I am going to sketch them out very briefly but after that, there is a video of Brandon Sanderson giving a university lecture on these rules. Yeah, this guy knows his stuff and teaches it.
Sanderson’s first rule of magic
Your ability to solve problems with magic in a satisfying way is directly proportional to how well the reader understands said magic.
Where the magic is not well understood by the reader you can engender a sense of wonder but if you try to resolve plot like that the reader will feel betrayed
Sanderson’s second rule of magic
Flaws are more interesting than the powers themselves.
The things the characters cannot do (limits) or struggle to do with the magic (or super science) make for the best stories.
The same goes for the cost of the power. In Dune the space travel needs the spice (which is what makes the powers work).
Sanderson’s third rule of magic
Go deeper into a magic not wider
Exploring a smaller system in an interesting way will make the world feel more real.
Sanderson’s zeroth rule of magic
Always err on the side of what is awesome.
At the end of the day, tell amazing stories. That’s pretty much the best advice anyone can give you.