Walter Oliver: ‘Once you’ve grown cotton, it’s hard to get away from it’