[MAY SLEEP: SEE BELOW]
These routines are used to dynamically request pointer-aligned chunks of memory, like malloc and free do in userspace, but kmalloc() takes an extra flag word. Important values:
May sleep and swap to free memory. Only allowed in user context, but is the most reliable way to allocate memory.
Don't sleep. Less reliable than GFP_KERNEL, but may be called from interrupt context. You should really have a good out-of-memory error-handling strategy.
Allocate ISA DMA lower than 16MB. If you don't know what that is you don't need it. Very unreliable.
If you see a kmem_grow: Called nonatomically from int warning message you called a memory allocation function from interrupt context without GFP_ATOMIC. You should really fix that. Run, don't walk.
If you are allocating at least PAGE_SIZE (include/asm/page.h) bytes, consider using __get_free_pages() (include/linux/mm.h). It takes an order argument (0 for page sized, 1 for double page, 2 for four pages etc.) and the same memory priority flag word as above.
If you are allocating more than a page worth of bytes you can use vmalloc(). It'll allocate virtual memory in the kernel map. This block is not contiguous in physical memory, but the MMU makes it look like it is for you (so it'll only look contiguous to the CPUs, not to external device drivers). If you really need large physically contiguous memory for some weird device, you have a problem: it is poorly supported in Linux because after some time memory fragmentation in a running kernel makes it hard. The best way is to allocate the block early in the boot process via the alloc_bootmem() routine.
Before inventing your own cache of often-used objects consider using a slab cache in include/linux/slab.h