Reflection is a quiet business that requires focus, and it often helps to actively seek solitude, to keep distractions away and to avoid the influence of those who promote competing values. (Such things as people take for values, like money, fame or pleasure.)
When you withdraw, observe yourself: do you subtly make sure that people notice it, that you are seen as doing something that looks grave and important? Do you manipulate the perception of others so that your reclusiveness looks like a deliberately chosen way of living, do you try to make that impression?
It is all well that you are searching for truth in self-dialogue, seeking out weaknesses in yourself, in your actions, views and feelings, your character and your overall ideas of how to live your life. And these are things of some momentum; they really are important; they are the only things that matter in the end. But then what other people think (or perceive) should be of no interest here, right? This should be a dialogue exclusively with yourself. Perhaps you'll ask others for advice, or discuss ideas with them — in this sense, it's not required to exclude others. But obviously, these are not activities you have to withdraw for, quite the contrary. When your withdrawal has a hidden agenda with respect to others, then it never is an interest in getting their support, but in being recognized: a desire to be admired for it. Turning your back on people is a showy sort of action; think hard, however: what would be that important about your not being around? Priding yourself on your retreat and your philosophy is itself a sort of ambition and of boasting.
And so is being too secretive and totally withdrawn from people's eyes — it is more subtle, and it takes a little longer for people to notice, but the idea comes from the same source. It's attention-getting in disguise (making your absence felt so that it draws people's attention).
There's not much point in trying to persuade yourself (and others) that you are someone who thinks so important thoughts that your solitude mustn't be interrupted for the good of mankind. If you want to be able to focus, by all means make room for it, and take care that you get the quiet you need. But it's not something that anyone besides yourself needs to know. Take care.