Oh man, Yellowstone National Park area is one of my favorite places to visit. If you have been there before then you will already know some of this info.
Some parts of Yellowstone have a long hard winter, and then fall arrives early. That means that you are pretty much guaranteed to find some road construction delays.
I don't think that there is anywhere in the park that has a speed limit above 45mph. You may find some places where your motorhome engine will be gasping for more air and your maximum speed might be 30mph. Or else you will be following a motorhome that can't get above 30mph without any chance of passing. Or sitting still in a line of vehicles that are gawking at the local wildlife. So the summary of this paragraph is just don't expect to get somewhere quickly.
Are you going to spend a few days around the town of West Yellowstone? Let me rephrase that. You should spend at least a couple of days basing from West Yellowstone. Just a bit west of there is Quake Lake with a nice visitor center. If you have any interest in geology then you will need a couple of hours in that visitor center learning why the side fell off the mountain.
Do not be surprised by the night time temperatures dropping into the 30s in August. Pack several different jackets etc. so that you can dress for the variety of weather.
Jackson Hole is a neat place. If you want to spend your time in Yellowstone then don't plan on basing from Jackson Hole because it is at least an hour of driving to get into Yellowstone (see above about road work).
Now finally to get to your question. I spent a couple summers working on a ranch in that corner of Montana, but that was before I got the 4wheeling bug. I know that there are some trails that get back into the mountains, at the very least for mining and logging activity. There should be some close to West Yellowstone. To go a little farther west from West Yellowstone head over (US 287) to Ennis and then go up into the Gravely range. Just several days ago I was looking at that area in Google Earth and realized that you can go over the mountain and come down into the town of Dillon. If you are interested in fishing then this would put you into some good places (I have heard them mentioned as world class).
If you like tourist-y places then continue west from Ennis (MT 287) over the hill to Virginia City.
The west side of the Teton Mountains in Idaho is a pretty area too. It is a big valley with lots of agriculture between the mountains.
Enjoy the trip.