The Force ghost was the soul and essence of a deceased Force-sensitive who denied the will of the Force upon death, yet was able to interact with the living, albeit not physically. Some Sith Lords learned similar techniques, which in some cases allowed them to physically interact with their environment.
According to the Jedi tradition, death was a part of life, and it meant becoming one with the Force. In death, sentients lost their ability to communicate with the living, but mastery of certain obscure paths of learning could avert it.
The origins of that knowledge, as well as its history, are not known. Ancient Jedi of the Galactic Republic such as Arca Jeth, as well as Sith Lords, knew a secret which could enable the user's essence to "survive death", but this knowledge was somehow lost over the millennia, perhaps with the destruction of the Jedi library world of Ossus in the Exar Kun Wars or the First Jedi Purge. However, it was secretly preserved over the millennia by the native Ysanna, descendants of Jedi trapped on the world and slowly driven to barbarism.
Qui-Gon Jinn was the first of the recent Jedi who rediscovered this secret with the assistance of a shaman of the Whills. Qui-Gon's spirit guided Yoda and Obi-Wan Kenobi and revealed to them the secret of retaining one's individuality after becoming one with the Force. Yoda during his exile taught it to another Jedi Master, Qu Rahn. The Dark Woman also learned of it at some point — perhaps Qui-Gon also appeared to her during her years in hiding on Cophrigin 5.
Death for those Jedi were different: their spirits could retain their individuality and could return as voices, dreams, or apparitions at will, so that they could help those in need of their guidance. On death, their bodies vanished; this was not the case with Qui-Gon, however, probably being a technique evolved by Obi-Wan and Yoda, under Qui-Gon's guidance.
This state was temporary, as Force ghosts were an intermediate state between life and afterlife; after a certain amount of time, they would then have to move on to the Netherworld of the Force, another realm of existence. Darth Vader learned of this ability through Darth Sidious, and upon his death, the spirits of Kenobi and Yoda completed his training by granting him the last step in becoming a spirit. Anakin's spirit appeared as that of his younger self, prior to his fall to the dark side.
Certain Sith Lords had also discovered similar techniques. Occasionally their bodies would fade as Jedi would, though like Jedi this was not a prerequisite for becoming a Force Ghost. It was not uncommon for the spirits of Sith Lords to remain, bound to the living world through an object, location, or simply due to the force of their malice and hatred. Unlike the Jedi, however, the Sith could find no peace in this form, instead existing for restless centuries, eventually driven insane by solitude, memories, or dark side inflicted madness. In many ways, these techniques differed from those taught by the Jedi; the most distinct difference being the user's limited ability to interact with his surroundings. Arguably, the most proficient user of such a technique was Exar Kun.
It is assumed that spirits of Force-sensitives, although not fully trained Jedi when they lived, could also learn this technique; at least they could be manifested by the Force in Force nexuses, such as the Valley of the Jedi. It appears that masters of the Baron Do Sages knew a similar technique
In Order of Death:
At unknown times:
During the original trilogy, fans and Expanded Universe writers believed that this happened to all Jedi who died, even fallen Jedi who repented at the end of their lives, such as Darth Vader and the Jedi Ulic Qel-Droma. In the latter case, the vanishing was a sign that his spirit was redeemed. In works such as Tales of the Jedi, most Jedi who die are seen dissolving, and sometimes later appearing as ghosts.
Many questions arose during The Phantom Menace, when Qui-Gon Jinn's body did not vanish after his duel with Darth Maul, and instead was cremated on a Jedi funeral pyre like Vader. In Revenge of the Sith it is made known that "dying AND maintaining one's identity" is not in fact what happens to every Jedi, but an ability acquired by Qui-Gon's spirit and passed on to Yoda and Obi Wan.
On the other hand, Qui-Gon Jinn was the first character to show Star Wars audiences that not all Jedi became Force ghosts upon dying (by virtue of his body not disappearing at death), but he was established in Attack of the Clones as a voice to Anakin Skywalker, and also in Revenge of the Sith as the first G-canon character to learn how to become a Force ghost, despite his body not disappearing at death. The implication is that one can learn how to be a Force ghost even after dying a standard death. Mara Jade Skywalker did something similar after dying. She only faded away when Jacen (secretly Darth Caedus) entered the room in a last ditch effort to inform the Jedi about him, which would seem to imply that a Force user who has mastered the ability to turn into a Force ghost has control of their body.
It is explicitly stated in Champions of the Force and the databank at the official site that the organic part of Anakin's body disappeared, and that Luke just burned his suit and mechanical parts for ritualistic purposes.
Controversy sparked when in the DVD of Return of the Jedi, Sebastian Shaw's Force ghost was replaced by that of Hayden Christensen's image, thus removing Shaw's Force ghost from canon and placing Christensen's in its stead.
Another problem that remains to be retconned is the existence of spirits of Sith and Dark Jedi. In both the novel and the comic book version of Revenge of the Sith, the spirit of Qui-Gon Jinn tells Yoda that only light-siders may live on as Force ghosts, and that this is something that dark-siders cannot learn. However, in Dark Lord: The Rise of Darth Vader, Darth Sidious (who mastered and perfected the technique with the use of clone bodies himself) reveals that the ancient Sith Lords once had the ability to "survive death", but this knowledge was somehow lost over time. It is unclear which Sith Lords qualify as "ancient" in this context, although Kaan and Qordis were the last known Sith to have escaped being lost in the Netherworld of the Force before Palpatine. It could be possible that Qui-Gon was misinformed or rather the concept of it only applying to Light side users was simply one of his many (often radical) theories. There is also the distinct possibility that light side force ghosts and dark side force ghosts are fundamentally different, and that's what Qui-Gon meant. Indeed Jedi Force ghosts seem to be "freed" into the netherworld by the passing of their physical forms and conversely, Sith Force ghosts seem to be "trapped" or attached by death, be it to a relic (the Muur Talisman, Ulic Qel-Droma's Sith amulet, etc) or to specific locations (such as the Massassi Temples in Yavin IV, ancient Sith tombs, etc). This however has yet to be confirmed.
This question applies to non-Jedi, such as Morgan Katarn who have appeared in that form. It is assumed that spirits of Force sensitives, although not fully trained Jedi when they lived, can be manifested by the Force in a Force nexus (places where it is extremely strong), such as the Valley of the Jedi. Also, the Sith Lord Ajunta Pall lived on through the Force in such a manner as a Force ghost, though it is relatively unknown how or why he remained. However, it can be speculated that his extreme rage and anger made his presence linger. This has been applied to Exar Kun, who used the dark side to feed off of the life force of the Massassi just before he died. It may be possible that spirits of other Sith Lords latched onto a place strong in the dark side, feeding off of it like a parasite. The process may be explained in a future novel or other source.
The non-canonical characters Tag Greenly and Bink Otauna become Force-ghosts after dying in the explosion of the Second Death Star, and appeared on Endor next to Obi-Wan, Yoda, and both a younger and older vision of Anakin Skywalker.
Mace Windu appears as a Force ghost in the alternate history comic Star Wars Infinities: The Empire Strikes Back in a vision to Darth Vader, though this may only be a mental vision in Vader's mind as opposed to an actual Force ghost.
In the online multiplayer game Star Wars Galaxies, it is possible for players who have played as a Jedi before the New Game Enhancements to become Force ghosts. Since the players in the game exist only in one server at a time, they are non-canon.
The Force ghost versions of Anakin Skywalker, Yoda, and Ben Kenobi are playable characters on Lego Star Wars II: The Original Trilogy and Lego Star Wars: The Complete Saga.
In the Robot Chicken: Star Wars special, Jar Jar Binks is turned into a Force ghost after Darth Vader ejects him from an airlock.