Windows CE is optimized for devices that have minimal storage - a Windows CE kernel may run in under a megabyte of memory. Devices are often configured without disk storage, and may be configured as a 'closed' system that does not allow for end user extension (for instance, burned into ROM). Windows CE conforms to the definition of a real-time operating system, with a deterministic interrupt latency. It supports 256 priority levels and provides for priority inversion. Unlike UNIX-like operating systems, the fundamental unit of execution is the thread, providing for simpler, faster concurrent programming (see Thread (computer programming)).
Back in the day when palmtops were up and coming, Microsoft trimmed down windows into Windows CE (CE = Compact Edition). The first version, nicknamed "Pegasus" featured a windows-like GUI and a number of microsoft's popular applications, all trimmed down for smaller storage, memory and speed of the palmtops of the day.
Since then, Windows CE has evolved into - according to Microsoft's docs - a component-based, embedded, real-time operating system. It's no longer only targetted at hand-held computers. Many platforms have been based on the core Windows CE operating system, including Microsoft's Handheld PC, Pocket PC, Pocket PC 2002 and Smartphone 2002.
It's often - incorrectly - stated that Windows CE 3.0 and Pocket PC are the same thing, or that Pocket PC is the successor to Windows CE 3.0. This isn't true. Windows CE 3.0 is a selection of operating system components, some of which provide subsets of other components' features (e.g. varying levels of windowing support; DCOM vs COM), others which are mutually exclusive (bitmapped or TrueType font support) and others which add additional features to another component. You can buy a kit (the Platform Builder) which contains all these components and the tools with which to develop a custom platform. The applications like Pocket Word etc are not part of this kit.
Pocket PC is a Microsoft-defined custom platform for general PDA use, and consists of a Microsoft-defined set of minimum profiles (Professional Edition, Premium Edition) of software and hardware that is supported. The rules for manufacturing a Pocket PC device are stricter than those for producing your own Windows CE-based platform.