I would add London Lights: The Minds that Moved the City that Shook the World, 1805 - 1851 by James Hamilton, from 2007. It's about the whole scene including Ada and Babbage and many others -- where the worlds of science, the arts, and literature overlapped and mingled more than they usually do now.
From Chapter 5, around pages 110 - 115:
"Ada Byron began to make a habit of going to Babbage's with the Somervilles (mathematician and popular science author Mary and her husband William). Ada Byron was intense by nature. A highly articulate, overwrought mathematical genius from a high social background, she attended lectures at the Royal Institute, routs in glittering ballrooms, soirees at Babbage's, and, aged seventeen in 1833, was presented to the King and Queen at court. Her natural brilliance in the abstract language of mathematics that few understood, her disturbed upbringing as Byron's daughter, and the elevated position in society it was her lot to display and maintain created a confused and uncertain young woman. ... It was the rare piece of real good fortune for Ada, however, that she had the sensible and structured example of Mary and William Somerville's family life to balance her self-absorption, the insistence of her many suitors, and Charles Babbage's frantic self-certainty. ..."
From Chapter 5, around pages 110 - 115:
"Ada Byron began to make a habit of going to Babbage's with the Somervilles (mathematician and popular science author Mary and her husband William). Ada Byron was intense by nature. A highly articulate, overwrought mathematical genius from a high social background, she attended lectures at the Royal Institute, routs in glittering ballrooms, soirees at Babbage's, and, aged seventeen in 1833, was presented to the King and Queen at court. Her natural brilliance in the abstract language of mathematics that few understood, her disturbed upbringing as Byron's daughter, and the elevated position in society it was her lot to display and maintain created a confused and uncertain young woman. ... It was the rare piece of real good fortune for Ada, however, that she had the sensible and structured example of Mary and William Somerville's family life to balance her self-absorption, the insistence of her many suitors, and Charles Babbage's frantic self-certainty. ..."