The growth of the Internet of Things (IoT) has led to the increase of new threats, particularly IoT botnets that are a serious cybersecurity concern. This paper proposes a Machine Learning (ML) framework using black-box models for IoT botnet detection. It offers a trade-off between performance - high model accuracy - and interpretability - by providing security experts with explainable results. Our experimentation on real traffic data infected with Mirai and Bashlite malwares, showed that the framework achieves the best accuracy using random forest and extra tree models. In addition, it provides security experts with information on features that are important for a particular instance (i.e., local interpretation) and for the whole dataset (i.e., global interpretation), allowing them to trust the models results and save time and resources.