Boswell first met and fell in love with Isabella in Inverness in 1761, when he travelled with his father on the northern judicial circuit. They met again in Aberdeen on August 22, 1773, during his and Johnson's tour of Scotland and the Hebrides, when they received an invitation for tea from Mrs Riddoch, as she then was. He later wrote of their meeting that "I was in a kind of uneasiness from thinking that I should see a great change upon her at the distance of twelve years. But I declare I thought she looked better in every respect, except that some of her fore-teeth were spoiled. She was the same lively, sensible, cheerful woman as ever." 

Boswell referred to Isabella as his cousin, although they were very distantly related.