Son of Alexander Walker of St Fort and Jean Fleming. He was presented as Minister to the Parish of Leuchars, north of St Andrews, on August 15, 1732. In 1740 he married Katherine Montgomery, a sister of Robert Montgomery, who served as Lord Provost of Edinburgh 1756-1758.
Boswell and Dr Johnson stopped by Leuchars on their journey from St Andrews to Montrose on August 20, 1773, having noticed the church tower from the road. Boswell wrote, in his Journal of a Tour to the Hebrides:
The manse, as the parsonage-house is called in Scotland, was close by. I waited on the minister [James Walker], mentioned our names, and begged he would tell us what he knew about it. He was a very civil old man; but could only inform us, that it was supposed to have stood eight hundred years. He told us, there was a colony of Danes in his parish; that they had landed at a remote period of time, and still remained a distinct people. Dr Johnson shrewdly inquired whether they had brought women with them. We were not satisfied as to this colony.
Walker, referred to by Boswell only as a very civil old man, died just a few months after their visit.