Clifford graduated from Deshler High School in 1962 and Vanderbilt University in 1967. He was a three sport athlete for the Tigers earning letters in football, baseball and track. He was awarded the Dr. H.B. Beck Memorial Award in 1961 as Deshler's Most Outstanding Offensive player in football and was named to the Birmingham News along with the Birmingham Post-Herald All-State Teams. Clifford was chosen to play in the North-South All-Star football game in Tuscaloosa and was elected as a Co-Captain for the North squad. In baseball he was chosen to play in the East-West All-Star Baseball Game. Following high school he received a football scholarship to Vanderbilt and he played and coached in the service at Wurzburg, West German. He has also been active in coaching youth football and baseball while in Germany and in Wilmington, North Carolina. From 1988-1991, Clifford was and assistant football coach at John T. Hoggard High School in Wilmington.