His very early years were difficult. His parents marriage ended in divorce in 1896 when Mattie left George and took six-year old Jimmie (as he was known then) from Tucson to Willcox and the NO Springs Ranch operated by Mattie's brother John Riggs; Mattie was already skilled in handling horses and cattle.

 Shortly after their arrival, Mattie met and married John Duncan in 1897 and John joined in the ranch operation. In 1899, John was killed at the NO ranch by Wiley Morgan; the immediate motive for the murder was disputed ownership of a calf but Wiley was also avenging for Mattie's earlier rejection of a marriage proposal by Wiley's brother George. Jimmie was a witness to the shooting and John died in his arms. At Morgan's pretrial and then trial in December that year, Jimmie is reported to have provided exemplary testimony against Morgan and two others involved.

 His mother married H.J. 'Doc' Goodin in 1900 and in 1903 they bought a large cattle ranch near Canelo. James may well have helped with that operation; he is said to have started his ranching career with 35 head near Elgin. Doc died of sepsis a few years later in 1908.

 On November 11, 1911, he married Margaret Igo. By 1917, he had prospered in his ranching efforts and gave money to endow the operation of the cemetery, Black Oak, near Canelo, which his mother Mattie had successfully established by requesting lease of 10 acres of Coronado National Forest. In 1919, his first son Thomas was born, and his second son Jack in 1922.

 He was a member of the Arizona Livestock Sanitary Commission and chairman of that organization for six years. He was an inspector for the Federal Intermediate Land Bank of Berkeley. He was a member and chairman of the Santa Cruz County Board of Supervisors in the late 1920s. His memberships in professional organizations included the Arizona Cattle Growers Association and American Quarterhorse Association and he was a director of the American Society of Range Management.

 In 1962, he and his wife Margaret sponsored the construction of an outdoor roofed community chapel at Black Oak Cemetery "in memory of [his mother] Mattie Riggs Johnson whose relatives were the first to be [layed] to rest in this cemetery".

 When he died, he owned ranches in five counties in Arizona: Santa Cruz, Cochise, Navajo, Yavapai, and Maricopa. One of his quarterhorse fillies won a race at Prescott Downs just shortly before his death.