Yes, unfortunately

We do augmented reality (aka "Heads-Up-Display") and we use a calibration algorithm which gives us the raw projection matrix. AFAIK in general 3D engines do this
glMatrixMode( GL_PROJECTION );
glFrustrum(left, right, bottom, top, near, far)
which calculated a standard perspective projection matrix and we'd need to use
glMatrixMode( GL_PROJECTION );
glLoadMatrixf( &customMatrix);
do you think this could work in xith?
Well, I guess, you could simply use a little trick. Just compose the view matrix with the inverse of the projection matrix (from the left). This should neutralize the projection matrix and you could use your own custom projection matrix as the regular view matrix. I guess, this should work. But we could also provide the possibility to apply a custom projection matrix. But I would need to check, what would have to be done.
I guess, the simplest solution would be to setup the view and the canvas/viewport in a way, that the automatic internal calculation does exactly, what you need.
In any case, it is possible.
Does xith support orthogonal projection together with HUD, because if it does it should be no problem and I could just try to implement it

Yes. Definitely. Xith supports orthogonal or perspective projection for each RenderPass separately. It also has a complete HUD implementation ready to be used out of the box. And it of can course be rendered on top of anything without being affected by it.
Marvin