Large language models (LLMs) have shown increasing capability in problem-solving and decision-making, largely based on the step-by-step chain-of-thought reasoning processes. However, it has been increasingly challenging to evaluate the reasoning capability of LLMs. Concretely, existing outcome-based benchmarks begin to saturate and become less sufficient to monitor the progress. To this end, we present a process-based benchmark Mr.Bean that demands a meta reasoning skill, where LMs are asked to locate and analyse potential errors in automatically generated reasoning steps. Mr.Bean is a comprehensive benchmark comprising 6,006 questions collected from human experts, covering various subjects such as physics, chemistry, logic, coding, and more.Through our designed metrics for assessing meta-reasoning on this benchmark, we identify interesting limitations and weakness of current LLMs (either open-source or close-source models). For example, open-source models are seemingly comparable to GPT-4 on outcome-based benchmarks, but they lag far behind on our benchmark, revealing the underlying reasoning capability gap between them.Our anonymous dataset and codes are submitted with the paper and will be publicly available.