Head west of Almonte on Wolf Grove Rd. until you hit the 12th concession on your right. It's about 16km out of town, past Union Hall Road with the tall antenna.
At the end of the 12th is the narrows between Taylor and Clayton Lakes. There used to be a famous floating bridge here. It's shown on my Lanark County map from 1879 and was in use until 1964 when it was destroyed by Hurricane Connie. Photos of the bridge from the 1950s show it was in pretty rough shape by the end. If you're paddling here, apparently you can still see the timbers on the lake bottom.
In the photo above you can just see a light spot on the far shore where the 12th continues. This stretch of water used to be a lot narrower until about the 1980s, when the dam in Clayton was improved and the lake levels raised considerably.
The bridge served as an important shortcut through the lakes and swamps that appear along a line drawn between Clayton and Middleville. If you lived north of that line, you'd want ready access to the Upper Perth Road, which you can see today just south of where the 12th intersects Wolf Grove Rd. The Upper Perth Road was the main drag from Blakeney to Perth via Bennies Corners (intersection near Mill of Kintail--another ghost town whose population left to settle Almonte), Clayton, Fergusons Falls, and Balderson. The ride's a piece of cake today, but accounts of this trail from the mid-1800s describe it as a muddy track winding through massive old-growth pine and hardwood forests.
Google Maps shows the town of Lloyd just before the floating bridge. It's not shown on my 1879 map and there's nothing to see today.