From Gamehiker Wiki
The Nintendo 64 Controller Pak was something of a Memory Card for the Nintendo 64. Rather then save to the cartridge itself, data could be saved to this accessory, which plugged into the back of the Nintendo 64 Controller. It was not a requirement for all Nintendo 64 games, but because of the cost of making the cartridge save in itself, several third parties like Konami and Ubisoft took advantage of it as the only option to save their games. There are no Nintendo developed or published titles that require the Controller Pak to save but it is necessary for optional data such as time trial records.
The Controller Pak has yet to be successfully emulated by the Wii's Virtual Console which could be a contributing factor to why its Nintendo 64 library is small.