How can I create a streaming server where I can stream live and pre-recorded audio and video to http clients using html5 on the client end?

Is it a simple matter of implementing the rtp/rtsp protocols and such along with (d)encoding or is there a lot more work?