Are you going up to Imnaha? We've done that many times before without a trailer...I remember some steep roads!! We've also driven from Joseph to Baker City - again, without a trailer, but nothing stands out that it would be difficult in the summer. I wasn't looking through the eyes of someone pulling a trailer though, so don't take my word for it!
The Joseph area is beautiful....in my opinion it is one of the prettiest parts of Oregon. Be sure to take the tram that is located just outside of the state park...the views from there are amazing!
We are planning on heading over there just as soon as school gets out! There are some pretty places to camp and hike in the Blue Mountains on the way, as well!
The photos here are of the mountain range heading into town, Wallowa Lake just outside of the state park and downtown Joseph...a fun place to wander and shop!
Best of luck to you