Frazier was born in Asheville, North Carolina, grew up in Andrews and Franklin, North Carolina, and graduated from the University of North Carolina in 1973. He earned an M.A. from Appalachian State University in the mid-1970s, and received his Ph.D. in English from the University of South Carolina in 1986. A 1985 published work by Frazier was a trail guide to the Andes and environs for the Sierra Club.
Frazier taught English, first at University of Colorado Boulder, then English at North Carolina State University. His wife convinced him to quit in order to work full-time on his novel. His friend and fellow North Carolina novelist, Kaye Gibbons, presented his unfinished novel to her literary agency, which led to the publication of Cold Mountain.
Cold Mountain was his first novel, published in 1997 by Atlantic Monthly Press. It traces the journey of Inman, a wounded deserter from the Confederate army near the end of the American Civil War. It follows his harrowing journey from deserting the army to finding his way back to the woman he left behind, Ada, who waits for him, dealing with all kinds of hardships herself. The power of Ada and Inman's love, and their dedication to reuniting, is the driving force of the novel, along with Frazier's incorporation of historical context. The work is rich in the culture and sensibilities of the North Carolina mountains, and is based on local history and stories handed down by Frazier's father about Frazier's great-great-uncle, William Pinkney "Pink" Inman (1839- 6 February 1865). Inman, who was from Haywood County, the area around Cold Mountain in western North Carolina, served in the Confederate Army, from which he deserted after being wounded twice, and is buried in the Bethel Community Cemetery, Bethel, Haywood County, North Carolina. "Pink" Inman served as a private in Company F of the 25th North Carolina Infantry, and his regiment did participate in the fighting in the Siege of Petersburg, including the Battle of the Crater.
Frazier's second novel, Thirteen Moons, published in 2006, traces the story of one man across a century of change in America. Also set in western North Carolina, the novel traces one white man's involvement with the Cherokee Indians just before, during and after their removal to Oklahoma. It is a story of struggle and triumph against the emerging U.S. government's plan to remove native Cherokee people to Oklahoma. Based on the success of Cold Mountain, Frazier was offered an $8 million advance for Thirteen Moons.