Punched tape was first developed as a way of storing messages for teletypewriters. The idea was to type in the message to the paper tape, and then send the message at "high speed" from the tape. The tape reader could "type" the message faster than a typical human operator, thus saving on phone bills.
When the first computers were being released many turned to the teletypewriter as a low-cost solution for printer output. This is why computers today still use ASCII, which was intended to be the standard character set for operating teletypewriters. As a side effect the punched tape readers became a popular medium for low cost storage, and it was common to find a selection of tapes containing useful program in most computer installations.
The little pieces of paper punched out of the tape are known as chad.