He wants to bring back Swapping/Swinging in our marriage

My husband and I have swapped a few times but I told him it made me uncomfortable and that I honestly didn't like it and he said he was ok with this decision. I felt that it was taking from the marriage, not bringing us closer as it started to consume daily living and weekend time ( also very expensive getting hotels and the high dollars to get into an exclusive club) He recently ordered more viagra and this morning... I had gotten a picture text from one of the couples from the site (naked) and he told him that not to be suprized if some strange woman calls me...... I don't want start an arguement, but feeling very put out.... the problem is.. he knows I am bi... I am attracted to women however, I have never had a relationship or done anything with female on my own or seperate from him.... i feel he is trying to push the box open again...i just don't want to be tossed into the lifestyle.... he keeps saying he met me knowing I was bi and brought him into this lifestyle... although that is not true... we were both in a different relationship to other people and it was him and my x that were trying to set us girls up together for a night of fun.... so how is it that i dragged him into this lifestyle? Shouldn't my no be enough? or did i allow a pandora's box be open by fullfilling a fantasy ? Help!