Clifford's grandfather, Henry Clifford Rudisill, was born in 1870 in Cherryville, North Carolina, son of Jacob Rudisill (1836-1925) and Susan Elmina Rhyne (1840-1919). He married Mary Lenora Lively (1876-1962) from Chamblee, Georgia. Their first two children, Clarence and Sybil, were born in North Carolina. In the early 1900's they moved to Texas because Clifford worked for the railroad and a friend moved to Texas and encouraged him to follow. He was the only one of his siblings to leave Gaston County. They lived in several towns in East Texas.

Clifford Rudisill established the Rudisill Family Foundation in 1998 and plans to donate the Rudisill House in Nacogdoches, Texas, to house the archives and artifacts of the greater Rudisill family. The Rudisill House belonged to Clifford's grandparents, Henry Clifford Rudisill and Mary Lenora Lively. Clifford is named after his grandfather who died when Clifford was only 10 months old. Clifford's grandmother was an incredibly gifted and gentle person whose memory continues to nourish Clifford's soul. He has only happy memories of the times he spent in the house during his youth.

Clifford's father, Paul Lester Rudisill, was born in Cushing, Texas, not far from Nacogdoches. They finally settled in Nacogdoches in 1920. The railroad was the Texas and New Orleans, owned by Southern Pacific. The train station where Clifford's grandfather worked, has recently been restored as a museum as there are no longer any passenger trains traveling through Nacogdoches. Clifford was born on January 21, 1938, and his only sibling, Dorothy Kay Rudisill, was born on December 1940, near Houston. Kay died on April 16, 2001 in Lenox, Massachusetts.

As a child Clifford traveled on the train from Houston to Nacogdoches by himself and has vivid memories of those trips. The train porter was trusted to tell the young Clifford when to get off. The train trip from Houston to Nacogdoches took about 4 hours but now one can drive it in about 1 1/2 hours.

Clifford's ancestors are descended from Thomman Rudi-sule, who was born in Fremsen, Switzerland, in 1605, and died in Sax (date unknown). His son was Hans Rudisuli, born April 20, 1643, in Fremsen and died May 26, 1723 in Michelfeld, Germany. His son, Hans Jakob Rudisile, was born October 4, 1666, in Michelfeld and died September 16, 1748, in Michelfeld. His son, Johan Jakob Rudisile, born in Michelfeld April 10, 1706, arrived in Philadelphia on August 17, 1729, on the "Mortonhouse" from Rotterdam, one of five brothers and cousins who migrated to the United States. Like the Rudisills to follow, this Johan Jacob settled in York, Pennsylvania, where he died in 1787. His son, John Phillip Rudisill, was born in York on April 19, 1738, and died in Mecklenburg County, North Carolina, in 1794. His son, Michael Rudisill, was born in 1765 in Lincoln County, North Carolina, and died in 1828 in Lincolnton, North Carolina. His son, Jocob Rudisill, was born December 26, 1798, in Beaver Dam, North Carolina, and died there in November 11, 1857. His son, Jacob Rudisill, was born November 28, 1836, in Beaver Dam, North Carolina, and died August 19, 1925, in Cherryville, North Carolina. This Jacob Rudisill was Clifford's great grandfather.

Jacob Rudisill and his wife are buried in the cemetary at St. Marks Lutheran Church outside Cherryville, where many other ancestors of Cliffords are buried. Clifford's father, Paul Lester Rudisill, was born January 7, 1907 in Cushing, Texas, and died December 2, 1971, in Greenwich, Connecticut, while visiting Clifford. Clifford's mother Dorothy Lucile Guthrey Berry, was born March 9, 1915, in El Paso, Texas, the daughter of Minnie Herring and Travis Austin Guthrey. She died in Houston, Texas, October 24, 1996.