There is a significant interest in developing ML algorithms whose final predictions can be explained in terms understandable to a human. Providing such an "explanation" of the reasoning process in domain-specific terms can be crucial for the adoption of ML algorithms in risk-sensitive domains such as healthcare. This has motivated a number of approaches that seek to provide explanations for existing ML algorithms in a post-hoc manner. However, many of these approaches have been widely criticized for a variety of reasons and no clear methodology exists in the field for developing ML algorithms whose predictions are readily understandable by humans. To address this challenge, we develop a method for constructing high performance ML algorithms which are "explainable by design". Namely, our method makes its prediction by asking a sequence of domain- and task-specific yes/no queries about the data (akin to the game "20 questions"), each having a clear interpretation to the end-user. We then minimize the expected number of queries needed for accurate prediction on any given input. This allows for human interpretable understanding of the prediction process by construction, as the questions which form the basis for the prediction are specified by the user as interpretable concepts about the data. Experiments on vision and NLP tasks demonstrate the efficacy of our approach and its superiority over post-hoc explanations. Joint work with Aditya Chattopadhyay, Stewart Slocum, Benjamin Haeffele and Donald Geman.