Core Data stinks for bulk inserts - direct SQLLite is much faster

I was handed an iOS app that was using the basic JSON handlers and keeping everything in memory. This was slow and running out of memory for large requests.

I converted the project to use stream based JSON parsing stored to a Core Data database backed by SQLLite. This solved the memory issues allowing all requests to work. There are 5 tables involved with a total of 239,000 records if you requested all the data from the server.

The problem - that request took 5 minutes and 43 seconds to process on an iPad 2. It took 33 seconds on the simulator. Let's state this again - Always test on a device. That kind of time is totally unacceptable.

After sniffing around the net and attempting to speed up Core Data I determined that Core Data sucks at doing bulk inserts. Core Data is really nice at hiding the gut level fun of SQLLite but for large inserts it is not the way to go.

I converted everything to use SQLLite directly and saw a massive speed increase. I can now process the same record set in 1 minute and 23 seconds. That cuts a whopping 4 minutes and 20 seconds off the time. The simulator went form 33 seconds to 12 seconds.

A typical user would rarely if ever request all the records but testing worst case scenario is really the best way to go. The speed improvement is seen across the board in the app with similar percentages of improvement. I am very happy with the results even though it took a couple of days to get it all sorted out.

As for the code there are areas that are more involved than the Core Data version and there are areas that are actually a lot cleaner. Everything is a trade off and there is no clear winner.

I had never used Core Data before. It was a really good learning experience to set it all up using the GUI editor in Xcode and finding out what it took to add the Core Data library and support code to an iOS project that was created without it originally. I found a lot of Core Data really straight forward and easy to use. I will not shy away from it on other projects unless they involve bulk inserts.

Having used SQLLite in the past I had a basic object set to start with and I knew my way around FMDatabase. My previous experience did bulk insert processing and I wrote both iOS and Android versions so I stuck with SQLLite to make it easier to convert code back and forth. I wanted to learn something new this time around so I went with Core Data. Good thing I could quickly fall back on my direct SQLLite experience to solve the speed issues that arose.