A few words first: this is not really my idea, but I thought someone should start writing it down. There was a thread on it, there's 4Es ritual magic system, and there are Incantations. Still, I think it's a generally good idea, and should be done.
Secondly, it does not fix magic on it's own. Far from it. A whole lot of spells take longer to cast, need a feat and are more widely available, yes, which probably means a lot of skill monkeys will start learning how to do them. But the spells themselves will, generally, still be just as powerful, just take longer and normally not be available in combat. To really fix anything, the spells themselves would have to be rewritten for balance, which I am too lazy to do.
Rituals
Rituals are great magical ceremonies of usually greater, or longer-lasting effects, than mere spells. They require complicated gestures and diagrams, long chants, focuses and rare ingredients, but promise great power to the user.
Rituals come in nine levels, just as spells.
Learning a ritual
Rituals are normally learned from arcane libraries or great temples, found written down in ancient scriptures, but they can also be thought by wise masters or arcane entities, or found as inscriptions in ancient ruins.
To learn a ritual, one needs a copy of the ritual, either written down, or thought by someone already knowing it. Next, before committing it to memory, it has to be practiced. This takes one day per level of the ritual, and during each of those days, at least 8 hours have to be spent studying without interruption. Furthermore, this costs an amount of gold equal to (3*level -1)*50 gold pieces, plus another five gold pieces per amount of experience point cost the ritual would have, if it has one, plus an amount of gold equal to it's material components, if it has any (this is the same cost as a scroll of a spell of the same level).
Finally, to learn a ritual, one must have the Ritual Magic feat, and a number of ranks in a knowledge skill (Arcana, Nature, Religion or The Planes) equal to two times the rituals level plus two.
{table=head]Level| Required ranks
1| 4
2| 6
3| 8
4| 10
5| 12
6| 14
7| 16
8| 18
9| 20[/table]
Casting a ritual
Rituals can be cast by anyone who has studied them and meets the prerequisites for casting them, be they a spellcaster or not.
Casting a ritual always requires a ten feet by ten feet square of free space, and it always involves verbal, somatic and material focus component. If the ritual is based on a spell, the focus of the spell has to be present and any XP cost and material component has to be paid.
The casting time of any ritual is at least ten minutes per level of the ritual or the casting time of the spell it is based on, whichever is longer.
If the caster is interrupted during the ritual, such as by restraining him, dealing damage to him or removing components, the ritual fails, but it's material and XP components are not consumed. Vigorous and violent motions, as well as weather effects, can be resisted by the use of the concentration skill.
Optional rule: Ritual Failure (mere suggestions)
The game master may wish for rituals to carry the potential of catastrophic failure, to make them more dangerous to the caster. In that case, he can add the requirement of a spellcraft check to the ritual. I recommend a DC of about 10+3*the ritual's level. While I have not completely run the math on this, it seems to about come out right, with a DC of 13 for level one (which a wizard with 18 intelligence and 4 ranks succeeds on on a 5+, and a non-intelligence based caster with 14 intelligence and 4 ranks on a 7+), and a DC of 37 for level 9 (which a caster with 24 intelligence and 20 ranks makes on a 10+, which can certainly be improved easily with feats and items, and a non-intelligence based caster with, say, 16 intelligence, 20 ranks and the skill focus feats will still make on an 11). I recommend not allowing items which give a bonus to spellcraft checks to be used, or to limit their boni to a small amount, like +2 or +4, at most.
On a failure, the ritual fails and may, at the DM's discretion, also carry negative consequences, such as:
Backlash damage: the caster suffers hit point or mental ability damage in proportion to the rituals level (this should not generally be enough to kill him).
Dimensional rift: conjuration spells may open a rift to another plane, which may suck the caster through, allow interplanar travel, or even allow hostile creatures from other planes to invade.
Scrambling or Off-course: similar to a teleport spell gone awry, a transportation spell could damage the caster and the teleported subjects, or leave them stranded in a distant location, or even on another plane.
Creature breaking free: A called or summoned creature may, instead of being controlled or imprisoned by the caster, simply smile and step over the caster's protective circles. A creature of animal-like intelligence or less may instead fly into a mindless rage, attacking everything in sight, including the caster.
Gone wrong: Instead of the creature to be summoned, a horrible monstrosity from the far realms steps through the created interplanar pathway.
What have I done? A creature to be brought back from the dead instead rises as an undead monstrosity, or a magical or undead servant shakes off the control of it's caster and breaks free.
(Temporary) Insanity: The caster's mind suffers terrible damage, making him suffer from confusion or a temporary mental illness of some kind. This may be purely a fluff effect, or cause sanity or other damage, if such rules are used.
In general, the DM is advised not to make rituals too dangerous, as they would become too much of an all-or nothing affair otherwise.
Ritual Magic (General)
Prerequisites: Knowledge (Arcana, Nature, The Planes or Religion) 4 ranks, Spellcraft 4 ranks
Benefit: You can cast rituals, based on the Knowledge skill you have used as a prerequisite for the Ritual Magic feat.
Special: Archivists, Cloistered Clerics and Wizards can take this feat as a bonus feat on level one instead of Scribe Scroll.
Circle Magic: (General)
Prerequisites: Knowledge (Arcana, Nature, the Planes or Religion) 4 ranks, Spellcraft 4 ranks, Ritual Magic
Benefit: when casting a ritual, you can cooperate with others to make the task simpler.
To cast a ritual in circle magic, a number of other participants up to the spell's level can cooperate with you. They all must have the Ritual Magic feat, but they do not have to know the ritual you are attempting to cast, nor do they have to have the Circle Magic feat.
A ritual cast as circle magic takes only half as much time to perform as it normally would. Furthermore, you may evenly share the XP cost of the spell among all participants, though no participant may spend an amount of XP that would cost him a level.
If the Ritual Failure rule is used, you gain a +4 circumstance bonus on the spellcraft check.
General Guideline: Which spells should, in my opinion, be rituals?
-Spells that have an effect which lasts one day or more, especially permanent ones.
-Spells that permanently change or create anything, like permanent magical servants or structures.
-Spells that summon or call a creature for more than a short time. Generally, the entire (calling) subschool of conjuration.
-Travel spells, especially those allowing instant travel over large distances, like Teleportation, but not those for combat mobility, like Dimension Door.
-Spells that remove a long-lasting negative condition, such as reviving the dead, breaking a curse or curing ability damage.
-Divination spells that give information about distant or future events.
Spells (from the SRD) that should (in my opinion) be rituals (Coming later)
Spoiler
Alarm
Align weapon (ritualized below)
Analyze Dweomer
Animal Messenger
Animal shapes
Animate Dead
Antilife shell
Antipathy
Arcane Eye
Arcane Lock
Augury