Fireplace is leaving smoke inside. any suggestions?

My wife and I just bought a 40 year old house with a fireplace. The fireplace was just swept professionally before we moved in. I lit a fire the other day. For the first hour, it was great. Afterward when I added another log, smoke started billowing out and into my living room. The flue was open as was the damper. I also heated the inside of the chimney before I lit the fire. Does anyone have any suggestions as to why the smoke is coming into the house?