You may have some sort of air leak in the party wall between the two houses.
This might be caused by faulty mortar in the party wall causing air and smoke to pass between the two houses.
In some older properties the party wall does not reach the roof in the attic and this may also cause the problem.
I would recommend you getting either a building surveyor or an architect to investigate the problem.
If necessary there are also specialist firms that can do an air tightness test on your house.

It is probable you have an air leak in the party wall between your two houses. This may be faulty/broken down mortar in the party wall allowing air (and smoke) to pass through or, as in some older properties, where the party walls do not extend to the roof in the attic there is adjoining air space which may be the cause of the problem.
I advise you get a building surveyor or an architect to investigate the problem and if necessary there are specialist firms that carry out air tightness testing on your house.
This should locate the problem area.