This was a problem with ADB. ADB use to verify that the file was an APK and read the entire file into memory before installation. This could cause mallac to return NULL, closing your ADB session (see file_sync_client.c lines 335-341).
This was fixed here.
Remove support for verifying APK before sending across wire, since it was reading the entire APK into memory (!) before sending.
I created a simple hello world project and created a large file (1.8 GB) and added the file to assets. ADB successfully installed the APK on my device.
As long as there is enough free space on the device, the APK should be installable. If you are still getting the error could not allocate buffer for 'APK', then update adb.
DEX limits:
The only size limitation I know of is the size of your dex file. If a dex file (classes.dex in your APK) exceeds 5 MB and needs to be installed on Froyo or Gingerbread then installation will fail. Recent versions of Android use an 8 or 16 MB buffer. You can use multidex to fix this issue. However, you probably won't need to worry about the size of your dex file (more info).