#1. The solenoid he was referring to sits on top of the starter and is connected to it. I think the one you were looking at is above the injection pump.
#2. No need to remove the starter. Your problem is electrical, not mechanical. There are 2 identical relays in the control panel box. They are positioned one above the other in the approximate center of the control box. One activates the preheat and glow plugs, the other the starter. Too darned cold or I'd go out and get their ID info. I believe the bottom one is the starter relay. It may be bad. On a good one (so long as your dc breaker is on) you can short between the two larger leads and crank the starter regardless of switch position. Same goes for the other activating the heating circuit.
If your solenoid was bad, I don't think it would engage the starter. The starter itself is obviously not the problem, its just doing what it is told to do.....crank.
The identical relays may be blue or black bakelite with about 6 wires coming off them. I have several if you need one, but there and the switch is where you should start, IMHO.
Jerry