Check WorldCat.org to find a nearby library that holds this volume, along with other potentially useful titles. Scottish Emigration to Colonial America, 1607-1785 by David Dobson, for example, might help you understand your ancestor’s passage to America.
Subscription site Ancestry.com has several databases for finding early arrivals from Scotland. Its Passenger and Immigration Lists Index, 1500s-1900s lists some 600 Scottish rebel prisoners transported to the American colonies in 1716. Among these, interestingly, is a James Dunlap, who potentially could be connected to your family.