The first Quaker known to visit the colonies of Maryland and Virginia was Elizabeth Harris, who came in 1655 or 1656 and found an immediate response. She was followed by a stream of others traveling in the ministry of the new faith. Many people of Maryland and Virginia joined the new movement. Although few early records of Virginia Yearly Meeting exist, it appears that George Fox initiated the first movement toward organization in that colony during his visits in 1672 and 1673.

In Fourth Month 1672, John Burnyeat, who was about to return to England after a lengthy ministry, called a General Meeting (to last several days) on West River, south of present-day Annapolis, for all Friends in the Province of Maryland. It happened that George Fox and several other English Friends had been visiting in Barbados and Jamaica, and arrived in Maryland in time for that historic meeting, which marks the beginning of Baltimore Yearly Meeting of Friends.

In his Journal George Fox recorded this event:

Then there was a meeting appointed by John Burnyeat about three score miles off, which held four days, which we went to though we were weary. And there came to it ... many considerable people of the world, and a glorious meeting we had. After the public meeting there were men's and women's meetings [for business] and I opened to Friends the service thereof and all were satisfied.

(George Fox, 1672)

Although little opposition was met in Maryland, which tolerated any Christian sect, the situation was different in Virginia, where only the established Church of England was allowed. There was much persecution, particularly on the Eastern Shore, forcing the Quakers to migrate northward into Maryland. Elsewhere in Virginia, the Quaker movement prospered in spite of opposition.

By 1700 there were about 3000 Quakers in Maryland, possibly the largest religious body in the colony at that time. The Yearly Meeting for Maryland held two sessions annually, one at West River and the other at Third Haven (now Easton) on the Eastern Shore. After 1774 sessions were held but once a year, alternating between the eastern and western shores of the Chesapeake Bay. In 1785 the western shore meeting place was transferred from West River to Baltimore.

With the building and improvement of roads on the Eastern Shore, Friends there were drawn toward Philadelphia as a center of commerce. At the same time the Friends from Philadelphia Yearly Meeting who were migrating to Northern Virginia, Western Maryland and adjacent parts of Pennsylvania and establishing meetings there, found Baltimore to be their urban magnet. In 1790, by mutual agreement of the two yearly meetings, all Maryland's Eastern Shore meetings were assigned to Philadelphia Yearly Meeting and all meetings in Northern Virginia, Western Maryland, Nottingham Quarter and meetings farther west in Pennsylvania were assigned to Baltimore Yearly Meeting.