The Lord of the Rings contains many instances of magic, but in no place does it explicitly define the internal system of magic; nor does it give any clear example of the magic. Thus, it is left up to the reader to attempt to define the magic.
The Lord of the Rings contains two distinct types of magic, each of which is initiated by the spirit of the user: a type which affects the natural world (Type A) and a type which affects other minds (Type B).
Type A Magic is best exemplified by four cases: the creation of the Rings, the Rings' modus operandi, the use of spells to fortify buildings (most notably Tol-in-Gaurhoth and Barad-dur), and the use of spells to weaken buildings (c.f. the witch-king's spell against the gates of Gondor); also, Gandalf's use of a 'locking spell' tells us about the style of invocations.
Type B Magic is best exemplified by four cases: the various disguises and cloaks employed by characters (specifically the elven cloaks worn by the members of the Fellowship, which allowed them to go undetected in close proximity to conceivably hostile forces), the effect of the Rings on culture and the minds of users, the Mirror of Galadriel, and the operation of foresight and the Palantir, as well as the seeing-chair on Amon Hen.
Exploration of Ainur/Maiar
Physical Form
The Ainur/Valar and Maiar (Ainulindale, QS) are spirits, created by Eru (Iluvatar) before the creation of the world. It is apparent in the Ainulindale the Ainur/Valar and Maiar (henceforth referred to as Ainur or Valar; QS explicitly places them as the same class of entities, differentiated only by power) are spirits, created by Eru from his mind. This leads to the apparent classification of the spirit as the medium of thought. Ainur do not have physical form -- they were created before the physical universe (whatever that means) -- but upon descending into the physical universe, they were able to clothe themselves in physical form. However, that form is a product of their thoughts; thus, we see the Valar are capable of manipulating the physical properties of the universe with only their thoughts.
Powers
The Valar are capable of 'raising mountains' (seen in the first few chapters of QS); Melkor's feat in raising the Misty Mountains in the path of the migrating Elves springs to mind. Thus, mountain-shaping provides a benchmark for the powers of the Valar. Varda is accredited with 'hanging stars'; it is unknown if stars in the Lord of the Rings are the same as stars in our world. Since Varda is not as powerful as Manwe, it seems they are not (10^23 J is not enough energy to influence solar formation; it isn't even enough to destroy a planet).
Taking into account the Valar's ability to raise mountain chains, we arrive at a rough upper estimate placing the powers of a Vala in the neighborhood of 10^23 J. It should be noted this applies most specifically to Melkor (and, by extension, to Manwe); the Valar are listed in order of declining power.
Thus, we see the Valar operate on incredibly large scales of energy. However, they work over 'countless ages', so their rate of energy expenditure is presumably quite low; if we assume the Valar shaped Middle Earth over a single millennia, this puts their power outputin the neighborhood of 10^13 W.
The one important limit on the powers of the Valar is this: they cannot produce sentient life. They can produce puppets (c.f. Aule and the Dwarves, QS), and they can manipulate evolutionary pathways (Melkor corrupting Elves into Orcs), but they cannot produce sentient life. This is because sentient life requires control of the 'Hidden Fire' (secret flame?), which is only possessed by Eru. Melkor had searched for it (Ainulindale), but did not find it; thus, as he fell, he could not create new life, but rather had to corrupt and make a mockery of the life already created. While seemingly trite, this does reveal evolution is at work in Tolkien's universe, and thus we can classify the differen species with relation to each other.
A note on the 'Hidden Fire' is in order: it is required for sentient life; it may even be argued this is the soul/spirit. Because there is no other proposed mechanism for the wielding of magic, and because we have canon evidence of spirits manipulating the physical universe, I postulate the mechanism for wielding magic is the presence of a spirit. However -- a note of caution here -- there is no strict evidence for a causal connection between spirit and magic; merely a strong correlation. If a more parsimonious mechanism is proposed, I will duly set it forth and analyze it.
Classification of Creatures
Elves, Men, Orcs
Elves, Men, and Orcs can all interbreed (Beren and Luthien, Tuor and Idril, Aragorn and Arwen; the Uruk-hai). Since this is established, canon, fact, and since the presence of evolution is also established canon, we can classify Elves within the same species as man (Homo Sapiens). Orcs are a slightly different matter; physically, they appear far removed from Elves and Men. Thus, we postulate Orcs are a subspecies of Homo Sapiens, much like the Neanderthals were a subspecies of Homo Sapiens in the real human evolutionary pathway.
Indeed, when Melkor bred Orcs out of Elves, one may conclude he was emphasizing a throwback to the common ancestor in terms of strength, quick breeding, and ferocity, and attempting to keep equal in terms of intelligence. He also apparently cultivated a unique culture which lacked inhibitions and instead relied on law, as handed down from a position of strength, to keep order: witness Orcs 'multiplying like flies' (LotR), and the debacle in the tower of Cirith Ungol (RotK), to see specific examples of this particular Orc culture.
However, Melkor's job was only half adequate; quite possibly due to superstition, Orcs quail at sunlight (or possibly due to their origin as a subspecies and their continual subterranian habitation). This defect was not corrected until Saruman crossed Men with Orcs during the War of the Ring. Melkor and his lieutenants may have introduced a fear of magic into Orkish culture to keep Orcs under control; such a taboo may be part of the reason there are no Orkish wizards, and the reason Orcs are terrified by the thought of magic (as when Sam comes up the steps in Cirith Ungol, or when the Elves first landed in Beleriand).
There are taboos against Elves marrying Men, though not as ingrained as the taboos against relations with Orcs; the Elf-Man taboo has been broken three times explicitly, and possibly more, but not mentioned (RotK, Prince Imrahil has Elvish blood). The children are considered the best of both worlds, and have powers and gifts equal to or beyond those who belong distinctly to each subspecies.
Hobbits
Hobbits are most likely a subset of humans. They share a common language with the Rohirrim (TT), originated in the same region (Appendices) -- along the Anduin between Mirkwood and the Misty Mountains -- and are, in all appearance, merely stunted humans. Perhaps they were a castigated segment of society, made pariah as the Rohirrim made the cultural transition to nomadic horse-herders. In any case, there are no visible genetic throwbacks to the original human stock (perhaps because such a child would die and kill the mother in childbirth), and no apparent desire to interbreed -- no evidence of any hybrid Hobbit - human children -- so Hobbits may have become a separate species in their own right, distinct from Humans/Elves/Orcs.
Dwarves and Ents
Dwarves are not evolutionarily related to any of the other species in Middle Earth. They were explicitly defined as created by Aule before the advent of Humans/Elves; but they were not sentient. Eru took pity on Aule, though, and gave Dwarves sentience. Aule had created them in the image of Elves/Humans, in anticipation; but though they appear similar, they are not related.
Ents are more ambiguous. Yavanna begged for Ents as a counterpoint to Aule's dwarves; though she did not create them herself, she asked for some of them to be sentient. This wish was granted, and thus we can assume some trees took evolutionary pathways toward sentience, and eventually became thus; they were, however, educated by the Elves. These were the Ents. Other trees became half-sentient; they are seen in TT as Huorns.
Magic: Instances and Rationalizations
Type A
Type A Magic is defined as any sort of magic which creates physical effects.
Gandalf v. Balrog
In the battle in the Mines of Moria, Gandalf flies down the steps after attempting a locking spell, presumably to render the door unable to be opened with anything less than an attempt to break it down; the Balrog utters a counterspell, and Gandalf states a Word of Power, which breaks off the conflict and causes the entire room to collapse.
From this exchange, we can infer the locking spell somehow creates a connection between the door and the wall, possibly even effectively fusing it. The counterspell, invoked in the middle of Gandalf's spell, would have begun to undo the work. The Word of Power then abolished the Balrog's spell, and also breaks off Gandalf's spell. This may have caused the connection created to reinforce the door to collapse, bringing down necessary support for the roof of the chamber.
Later, Gandalf is facing the Balrog on the Bridge of Khazad-Dum. Gandalf cracks the bridge with his staff. This may be an application of magic, or it may instead be a comment on the nature of the bridge's stone; however, given the fact the Balrog stood on the bridge, I lean toward a magical application of Gandalf's power.
The One Ring and Barad-Dur
The One Ring is claimed to be the foundation of Barad-Dur, and indeed, it supports the Dark Lord's tower. When the Ring is destroyed, Sam has a vision of Barad-Dur crumbling. Interestingly, the tower is vulnerable to other powers: at the end of the Second Age, the Last Alliance destroyed the tower, but not its foundations.
This indicates the Ring is the only thing holding the tower's foundations together: Barad-Dur itself is built well enough to withstand the different massive wind forces acting on it, but the foundations require the strength of a spell (or several spells). The Ring was evidently their focus; since Sauron would have laid the spells himself, and the spells would have been focused in him, when he put 'the greater portion of his strength' into the Ring, the focus for the spells would also have transferred.
Thus, when the Ring was destroyed, Barad-Dur's foundations were no longer strong enough to support the tower, and it was destroyed.
Incidentally, this allows us to derive an upper limit for the strength of those spells: if Barad-Dur was a cylindrical tower of iron 1000 m high, 150 m wide, then the spells must have supported weight in the neighborhood of 10^8 N.
Rings -- physical effects
The Rings of Power work to slow decay in the realm of those who wield them. The 'slowing of decay' may mean one of two things: either they slow cultural decay, or they slow physical decay. If it is the latter, then the rings may nullify the effects of entropy within a certain region.
The One Ring, in particular, allows the user to go invisible (especially if he is a lesser person). This may be a physical phenomenon, but the evidence points to a mental phenomenon.
In making the rings, the Elves (and Sauron) put a 'portion of their strength' into the objects. This points to the ability of skilled magic wielders to actually extend their spirits into inanimate objects. Once in the inanimate object, the spirit is no longer attached to the maker, but becomes, in essence, a wild card: anyone who chooses to do so can wield the object. Physical contact (actually putting the ring on, in this case) triggers a joining of the object's spirit and the user's spirit.
Another phenomenon related to this is the ability of makers to magnify their powers by putting them in the objects. The Elven rings are able to halt decay in their realms, something Celebrimbor most certainly could not have done himself. Thus, he was able to put his spirit into the rings preset to a particular power, and the spirit was magnified to act over a whole region, as opposed to merely acting over Celebrimbor's body.
Sauron put his spirit into the Ring, and thus quite possibly concentrated and magnified it. In this case, we see another aspect of the spirit: the ability to regenerate itself. Sauron was overthrown at the end of the Second Age, and the Ring was taken from him, effectively crippling him. However, while the Ring was still in existence, he was diminished, not destroyed. His spirit could grow again, regaining a measure of his former power, but not completely return to his old measure of strength; however, when the Ring was destroyed, Sauron was thrown down, and could never regain his power again. This points to two possibilities: either Sauron was slowly siphoning power from the Ring to pull himself back together, or, more likely, Sauron had tied his spirit to the Ring by putting so much of his power into it that if it were destroyed, the power he had regained would go with it.
The One Ring (along with the nine given to mortal men) prolongs the life of mortals far beyond its natural span. This, one of the Ring's minor powers, may be attributed to the effects of joining an immortal spirit (that of Sauron) to a mortal spirit (the mortal user).
The Rings are indicative of the wider phenomenon of magic: they anchor physical spells, but their effects, for the most part, are mental.
Palantiri
The Palantiri are seeing-stones, which allow the user, based on his strength, to 'look abroad' and see what people are doing. The apparent range is hundreds of miles; Aragorn can look into Mordor from Rohan and see what Sauron's troops are doing. However, the Palantiri are attracted to each other, and can even, to some extent, control each others' vision; this is how Sauron drove Denethor to despair.
The phenomenon of the Palantiri is a combination of mental and physical; physically, a Palantir gathers information and relays it back to the stone, where another mechanism converts the information to light and sends it out of the stone. A Palantir also has a holding pattern, like a screen saver, when not in use: Pippin is mesmerized by lights floating deep within the stone when he picks it up out of the water at the wreck of Isengard.
The attraction of the stones to one another indicates the existence of two spells. The information-gathering spell used by separate stones are apparently the same, and the two spells are probably attracted to each other: when they meet, a connection is established between the two Palantir, and two-way communication is possible. The two-way communication is mental, with the two participants thinking at each other (TT).
Tol-in-Gaurhoth is Sauron's watchtower in the valley between Hithlum and Dorthonion. It is noteworthy only because it indicates foundation spells are keyed to a single person, and that key can be transferred. When Huan subdues Sauron, Sauron surrenders up mastery of the tower to Luthien, and she causes the spell uniting the stones to be undone, making the tower collapse.
This indicates a spell binds the stones in a tower together; Sauron holds that spell, and he gives it to her (the form of the giving is not specified; possibly a key word or remnant of the incantation to tell Luthien what the spell is and giving her the power to undo it). The spell must have been strong enough to withstand the everyday forces on a tower, including its own weight, and wind.
During the siege of Minas Tirith, the battering ram Grond is deployed by Sauron's forces. It only needs to batter three times; on the third time, the gates burst. The Witch King utters a spell which apparently either weakens the gates or strengthens the force of Grond's blow.
Since the Witch King is, like any good commander, sitting at the back of his army, this instance demonstrates the range of a spell. Active spell-casting is apparently line-of-sight. This also serves as a counterpoint to Gandalf's locking spell.
Type B Magic is defined as any sort of magic which affects the perception of another being, sentient or not.
In the Lay of Leithian, Luthien creates a garment, sewn from her hair, which puts to sleep everyone it touches. It works without fail, even putting Morgoth-Melkor to sleep. The spell, in this case, is worked into the cloak, and activates upon contact.
When Finrod leads his company with Beren to assault Angband, he disguises them to appear as Orcs. They would have gone unnoticed, fooling even Sauron, had they not failed to report to Sauron as all Morgoth's servants were ordered to do. When they were waylaid and brought before Sauron, Sauron attempted a spell to strip them of their disguise; Finrod countered it, but Sauron's spell was more powerful. Their exchange illustrates the power of words, which are the vessel of magic. Finrod certainly did not actually alter their appearances; more likely, he created a spell which activated upon sight and changed the perception in the minds of others from 'oh, look, Elves!' to 'oh, look, Orcs!'
The Rings of Power have far more mental effects than physical effects. They can be made invisible to the eyes of others (FotR); the Elven rings stop cultural decay; the list goes on.
The main power of the Elven rings is to stop decay. If the decay is cultural, the effect of the rings is to halt the spread of apathy, crime, and indolence, and to keep the culture within a particular area at its height. This may be analogous to the effects of the rings on mortals: stretching out the life of the mortal beyond its capacity.
The nine rings given to mortal men were used by those men to increase their power over others; they could be used to make the men invisible. This invisibility came at a price, though; the longer they wielded the rings, the more they lost of their humanity, becoming invisible wraiths. This was most likely a perception in the minds of others; judging by the One Ring's conferrance of invisibility (examined below), the ringwraiths are invisible in the minds of all others, when they go naked. However, the Nazgul are also subject themselves to spells: everything they see is at Sauron's command, and much of it is fake, 'phantoms and delusions of Sauron' (FotR). They have no free will beyond that which Sauron allows them, another consequence of their usage of the rings: their will is slaved to Saurons. Presumably, he does not micromanage their movements, but rather he has corrupted their desire to be the same as his.
The One Ring confers invisibility upon its mortal users; this invisibility is most likely a mental effect projected upon any watchers because the shadow is still visible. The user still casts a shadow, so light is not bent around him, but nobody else can see him, which indicates the Ring projects a line-of-sight spell which wipes the user from the mind of whoever sees him.
The subject of invisibility bears one final note. Invisibility also allows the user of the Ring to see wraiths; in other words, it also affects his perception, and things which would not have been visible are visible; things which would have been clear are much dimmer and less perceivable (FotR, A Knife in the Dark). Thus, it can be safely stated using the Ring inverts vision between the perceptible and the imperceptible.
The most important mental aspect of the Ring (and of the lesser rings) is the fact that Sauron has tainted them. The Ring was Sauron's will, and which is, at its purest form, a desire for power and dominance over others. When the Ring is used, its spirit is joined with the user's for the duration of usage. Depending on intent, the Ring will eventually corrupt the user into union with it. However, the Ring will, in the process, become an addiction, thus accelerating the fall into union. It may be Sauron built this in as a safeguard, trusting his ability to detect a ringbearer before the ringbearer masters the Ring.
Sam is attempting to enter guard tower of Cirith Ungol, and he cannot. There are two statues called the Watchers which forbid his entrance. This is interesting because it is evidence of a physical compulsion spell, which is broken by the Phial of Galadriel.
In QS, Melian, a maia, weaves a maze of magic around Doriath. Nobody came through except at an invitation from Thingol or Melian (with the exception of Beren). This maze apparently took the form of a barrier of physical and mental compulsion which turned a person around as he moved forward and deposited him back at the borders with no memory of his attempted intrusion.
Melian's border spell extended for kilometres, and ended up conflicting with Sauron's necromancy and wizardry on the northern borders in the valley of Nan Dungortheb. This resulted in many unintentional spells -- of note because it raises the interesting specter of unintentional side-effects of combining spells -- and no life dwells there except for the spiders descended from Ungoliant. The only person who passed through there was Beren, and he did so only along the edge. He was quite traumatized, and never spoke of it again.
The nature of those unintended mixed spells was probably a mix of the confusion and memory-wiping of Melian's border, and the terror and black magic of Sauron's wizardry. One can only imagine what sort of horror a mixing of those two could create.
In yet another example of the powers of names, the Nazgul appear to react violently to the name of Elbereth Gilthoniel (Varda). It may be only their reaction to the idea of a prayer to her, but it gives rise to interesting speculation as to the origin of this reaction. Perhaps Sauron built it into the Nazgul's consciousness, but that does not seem likely; more likely is the possibility of a reaction to light and Varda's connection to light in general may arouse a shadow of that reaction. Laid beside the fact the Nazgul are also afraid of fire, this is an interesting commentary on the Nazgul's relationship with one of the most fundamental aspects of nature.
Type A Magic is magic which affects the natural world. It is apparently the lesser-used of the two magics, and we see it most often used by maiar (Sauron, Gandalf, Durin's Bane), and used only several by a Man or Elf (Luthien taking control from Sauron, Feanor constructin Palantiri, Witch-King). Thus, it seems likely a spell affecting the actual world takes an enormous amount of spiritual strength (Luthien and Feanor were among the most powerful Elves, and the Witch-King ), and the spells are generally limited to reinforcing or weakening physical materials, or gathering information.
Type B Magic is magic which affects other minds. It is used more often than Type A magic, and is used consistently by Elves (who have stronger spirits than Men). The most common usages appear to be disguises: invisibility and the changing of shapes. Spells of confusion (like Melian's Girdle) are less common. Physical compulsion is also common enough to be used at a simple guard tower in Mordor, and has apparently been automatized.
It also bears noting that there are two ways to leave spells: keyed to the person, and keyed to an object. The spell at Tol-in-Gaurhoth was keyed to Sauron; he held the power to disengage it. The spell Luthien cast on her cloak, however, was keyed to the cloak, and so the spell worked on whomever the cloak touched.
Spell v. Minas Tirith's gates
Type B
Beren and Luthien, instances
Rings -- mental effects
Watchers at Cirith Ungol
Minas Morgul, the Morgul-Knife and effects on Frodo
(Pending a re-reading of the pertinent sections of LotR)
Melian's defenses of Doriath
Nazgul, effects of names thereon
Conclusions
Type A rationalizations and descriptions
Type B rationalizations and descriptions
Addendum