Wednesday, April 24, 2013

Using YAJL with Objective C

I have been tasked with getting an existing iOS code base ready to be released to the App Store. After loading the existing project into Xcode and resolving static library link issues I was able to run the program but not log in. I added some #ifdef DEBUG processing to allow me to log into a server without a valid certificate and got to play around in the app on the simulator.

I was quickly able to crash the app and after setting the debugger up to stop on all exceptions I found it was running out of memory. Boy, if you can run the simulator out of memory you know you can easily do that on a device.

Looking through the code I found it was using NSJSONSerialization for all JSON processing. That library is fine and it is provided by Apple so you know it is legal to use but when you have large amounts of data it is not super efficient. If you have used XML in the past and are just getting into JSON this is similar to using DOM vs SAX. DOM is great and easy and hands you a nice tree of data but if all you are doing is plucking things out of that tree to put in another format you are better off using SAX especially if there are memory concerns.

The code was parsing everything into a big NSDictionary based tree with 13,318 base records and tons more child records. This is just not going to work on a memory limited device.

I looked up information on stream based JSON parsers and landed on YAJL which you can find at I won't cover how to configure it for Xcode as all that information is on the base site. It was easy to set up for both Mac OSX (my test program) and iOS.

They show a streaming sample that was too basic to be very helpful. Hopefully I can fill in some of the gaps. The first issue with their sample is they read the WHOLE file into memory via NSData then run it through the parser. Not fully streaming at that point. Second they show no samples on what to do in the callbacks.

The code snippet below will help you parse your JSON by keeping track of where you are in the data. It will keep a stack of named dictionaries and arrays as you parse through the data. You can then create and act on the proper objects as you get map keys and values.

It also reads in the JSON file in chunks instead of tossing the whole things into NSData. In my final iOS code I am using a NSURLConnection and parsing the chunks in the didReceiveData call.

The original code ran out of memory when receiving 38.9 meg of data and took 5 seconds to run. I put in some cheater try / catch blocks to let it "finish" without crashing for timing results. The new code does not run out of memory and runs in 2 seconds. Of course this is all in the simulator on a fast MacBook Pro with a lot of RAM. I don't currently have a device to test on but will get one soon.

I looked around on the web for a long time before I was able to piece together enough knowledge to pull this off so hopefully it will help others get up to speed a lot faster.

Things that go in the header file

// Array used as a stack to keep track of where we are
NSMutableArray *stack; // create this in init method
NSString *mapKey;

// Actual class code

- (id) init
    self = [super init];
    if (self != nil) {
        stack = [[NSMutableArray alloc] init];
    return self;

// Code to parse the file in chunks

- (void) parseFile:(NSString *)jsonFileName
    YAJLParser *parser = [[YAJLParser alloc] initWithParserOptions:YAJLParserOptionsAllowComments];
    parser.delegate = self;
    uint64 offset = 0;
    uint32 chunkSize = 10240;     //Read 10KB chunks.
    NSFileHandle *handle = [NSFileHandle fileHandleForReadingAtPath:jsonFileName];
    NSData *data = [handle readDataOfLength:chunkSize];
    NSLog(@"Starting parse of %@...", jsonFileName);
    [stack removeAllObjects];
    mapKey = nil;
    while ([data length] > 0)
        [parser parse:data];
        if (parser.parserError)
            NSLog(@"Error:\n%@", parser.parserError);
        offset += [data length];
        [handle seekToFileOffset:offset];
        data = [handle readDataOfLength:chunkSize];
    [handle closeFile];
    parser.delegate = nil;

// Stack adds newly named object OR last object on stack
- (void)parserDidStartDictionary:(YAJLParser *)parser
    NSString *dictName = mapKey;
    if (mapKey == nil)
        dictName = (stack.count == 0) ? @"" : [stack lastObject];
    [stack addObject:(dictName)];

    // Create new object here based on dictionary name
    // ready to hold soon to come key + value pairs

// End of dictionary - pop off stack
- (void)parserDidEndDictionary:(YAJLParser *)parser
    mapKey = nil;
    [stack removeLastObject];

    // Save object created in start Dictionary to
    // array or core data here

// New array staring, push name onto stack (use previous if we don't have one)
- (void)parserDidStartArray:(YAJLParser *)parser
    NSString *arrayName = mapKey;
    if (mapKey == nil)
        arrayName = stack.count == 0 ? @"" : [stack lastObject];
    [stack addObject:(arrayName)];

// End of array, pop off stack
- (void)parserDidEndArray:(YAJLParser *)parser
    mapKey = nil;
    [stack removeLastObject];

- (void)parser:(YAJLParser *)parser didMapKey:(NSString *)key
    mapKey = key;
    // Do processing here for each map key
    // Something like
    // if (!inResults && [key isEqualToString:@"results"])
    // {
    //    inResults = true;
    // }

- (void)parser:(YAJLParser *)parser didAdd:(id)value
   // Do processing here for each value (will have a mapKey)
   // Something like

   // if (inResults && [mapKey isEqualToString:@"query"])
   // {
   //   query = ((NSString *)value);
   // }
   // value will be (NSString *), (NSNull *) or (NSNumber *)
   // booleans come as (NSNumber *) 0 = false, 1 = true


Thursday, April 18, 2013

Is there really any such thing as a typical day?

I have been at my new job for a little over a month now. I have had the chance to learn a lot of new things. Generally I have been a front end developer doing the GUI and writing custom controls. Of course that does not mean I don't get to do any middle or back end coding. I use SQL where needed, write business logic handling objects, etc.

Here we are using Spring, Hibernate and Postgres for the database chores. I am finding my way about on all those annotations and have sent off and received requests for new table schemas to the DBA.

In a given day you can find yourself using little bits of each framework in a few hours time. I started the day setting up a chron job to call some existing code that calls two different 3rd party APIs to get some data. One of the vendors uses XML and the other JSon. The objects are created via Spring and we use Hibernate to interact with the database. I was using various internal SDK calls to do most of that with all of the work being done in Java.

Once that was up and running I was asked to look at some existing Objective C iOS code to see how close it was to being app store releasable. First off I was not able to login in as the servers I have access to don't have valid certificates on them and our API is HTTPS based. A browser will let you ignore the suspect certificate after a warning message. I added some #ifdef DEBUG to allow the Connection code to accept any certificate. That got me into the app and allowed to try out rest of the GUI.

Sadly I crashed the app pretty quickly. Did not know initially if it was bad memory management or if it ran out of memory. Turns out it ran out of memory. I added a @try block around things so it could push itself along a bit farther and I was able to run everything.

I then started to research why it ran out of memory. I looked at the server request it was making and tried it in a browser. A huge pile of data came back so I ran used curl to grab the data and write it to a local file. Turns out there was 38.9 meg of JSon coming back.

I wanted to sniff around in the JSon so I asked python to pretty print it and sent that to another file. I then opened it up in Sublime 2 to see what I had. Straight forward JSon, just a lot of it.

Next I wanted to see how the code was handling the JSon. Turns out the developer used the standard NSJSONSearialization calls in iOS. This is like using the DOM in XML in that you get it all in memory and they you double it up or more by converting to your internal object format. It was the easy way to go but not the most optimized way to handle things.

Time to hit Google and see if there is a streaming JSon parser. I found YAJL which I plan on experimenting around with today. I also think we should write the results out into a database instead of holding it all in memory. I have written similar code, it was XML and not JSon, storing to SQLite at a previous position so this should be fairly straight forward.

So was this a typical day? Yes, in that I ended up using a pile of different languages, data formats, command line tools and frameworks / SDKs to get my job done. You really need to branch out and learn new things.

I played around with Python in my spare time at home as I kept hearing about it and now I have used it multiple times. I used it at my previous job for a quick error log parser. I have used it here because some sample 3rd party code was written in it and I needed to convert that to Java. I was able to run the Python code and tweak it until it worked before I converted it to Java. I have used python to pretty print JSon files.

Curl was something new to me here. I have used it a lot with the 3rd party vendors as one did all their sample code it in. Curl makes it very easy to grab server results and shove them into a file. It works great with HTTPS as you can pass the user name and password in the command along with ignoring invalid certificates which seems to be a recurring need.

I have used Java for years so that is second nature. I have not used Spring or Hibernate in the past. I am really starting to see the power of the annotations. With great power comes great responsibility though. I am learning when you need to put code in other objects so it can be properly initialized. Anytime you use a new framework you find new and interesting ways to crash your code.

There is very little iOS knowledge here so I was able to step in and handle that too. A bit of a mind shift to get back onto Objective C and it will take some time to remember the syntax, framework and ins and outs of Xcode. Keeps you on your toes to shift around between languages, IDEs and frameworks. Good thing my Mac has a lot of memory, disk space and a large second monitor to keep all of this open and ready to roll.

Just to add to the mayhem I am working on updating my Android game at home. It does not scale well to tablets so I am converting the graphics to vector based allowing it hopefully work on nearly every device. At the end of most days I am pretty mentally fried.

Thursday, April 11, 2013

Motorola Xoom aint what she used to be

I have had the Motorola Xoom for a couple of years. One of the early Android tablets. It is dual core but boy is it showing its age. Part of this is due to my quad core Note II. Everything on my phone is really snappy. Things on the Xoom used to be snappy but now everything seems slow. Boot up time really stinks.

Button presses to get to home screen or bring up list of running apps are pitiful. I still use the tablet a lot for content reading. That works just as well as it always has although the screen is not super bright nor does it have the great colors of the Note II. I enjoy doing crossword puzzles more on the tablet as you can see more of the clues and all of the grid.

Not ready to get a new tablet just yet though. The Xoom works for its specific purposes. The direct plug-in charger quit working but the base charger I ordered from Amazon works just fine. If that goes out then the tablet will be useless.

Next time I will look in a smaller than 10" range but I don't know if 7" is big enough. Probably something in the middle. At least on the Android side of things there appears to be a nice variety to choose from. The current size / weight can make it a bit tough to hold one handed for extended periods of time. I am usually petting a cat while reading articles or using my other hand to one finger type crossword puzzle answers.

Monday, April 8, 2013

Life is a series of subscriptions

It used to be you had a few recurring monthly payments:
Rent / house payment
Utilities (gas, electricity, water, sewer, land line phone)
Car payment

You probably got your TV over the air. There was no internet and without that no such things as multiplayer games that were fee based.

Now you might have had some self imposed recurring fees like a monthly comic book store stop, new miniatures to paint for DnD sessions or something similar.

Today we have added:
Cable or Satellite TV including DVR rental
Internet provider of some type, bundled or otherwise
Cell phone including data plan
Online gaming subscriptions
App purchases
Netflix, Amazon or Hulu type service

We probably dropped:
Land line phone

It all seems so simple. What is another $20 here and there. Sure, get HBO and add data to all the kids phones and up the internet connection speed. Chuck on $3.50 for cell phone insurance, $14.99 to monitor your credit score, etc. When you look at it all - boy does it add up quickly. I cringe every time I think of adding another monthly fee.

I feel I would really enjoy playing WoW but I don't want to pay for a game then play every month to play it. I worry I would eat up too much time playing the game or feel like I had to play it to get my monies worth that month. I don't need a second job that is a game. I know I am generally pretty cheap. I don't buy new releases, I wait until they are sale usually on steam. I like to get Gold editions or multi-packs. When I play an RPG I generally finish it and I feel I get my monies worth out of it. I really enjoyed the Gothic series, Divine Divinity, the original Diablo (later ones became too multiplayer specific, still enjoyed them but not as much as the first), Might and Magic. Boy my wife was sick of MM6 by the time I finished it. Could only have done that before kids.

I enjoy a good FPS and have played through many of the single player experiences more than once. As weird as some of the Delta Force games were I have enjoyed them over and over through the years. Adventure games are fun to play with others helping you. Monkey Island and the Broken Sword series come to mind.

So far I have never signed up for a monthly fee based game. I have never made in app purchases. I did not buy fancy armor for any of the Elder Scrolls series. I like games that I buy, play at my own pace and play through more than once generally with years between the play sessions.

A lot of that is coming to an end. I play games on my phone that last just minutes a session. That is all the time I end up having. At best I try to carve out time each day to do crossword puzzles for a little mental challenge. I miss having an RPG story to return to a few nights a week spread over multiple months. I was gifted Torch Light 2 which was hack and slash and less RPG which I had fun playing and finished with multiple character types.

My son and I had a blast playing Portal 2 especially the co-op mode. We need to fire that up again for another play through. My wife had to keep the door to the computer room closed as we were laughing to hard for her to get any work done. We both have upgraded computers for this go round and it should be a really good time again. Sure, we will remember some of the tricks of the levels but there is a lot of timing involved and a lot of stupid ways to die. We need more co-op games.

We keep thinking computers will make our lives easier. They do remove some of the grunt work but they also make our lives harder. Keeping the wireless network up to snuff, adding new devices to the home network, print, scan, copy, collate, update drivers, new Android OS, new iOS, change the data plan, fix the NetFlix streaming issue, adjust the clock for DST and reboot you freaking car! Good thing I am in the tech field and can usually deal with the issues in a timely manner.

Wednesday, April 3, 2013

A handy Mac terminal command for getting IP without all the other junk of ifconfig

In a terminal window on a Mac you can type:


This will give you your IP address in the midst of a bunch of other stuff that is useful on occasion but usually not anything you care about.

If you type this:

ifconfig | grep "inet "

You will only see the lines related to your IP address which narrows things down a ton.

If you want to narrow it down a bit more you can do this:

ifconfig | grep "inet 192.168" 

If you are on a 192.168 network (which you probably are if using WiFi) then the top item in the list will be your IP address. Of course you can substitute 192.168 for your network prefix.

As a final helper if you need your IP address a lot you can add the following to your ~/.bashrc file

alias ip="ifconfig | grep 'inet 192.168'"

Reload .bashrc via

source ~/.bashrc

Now every time you type:


and press enter in a terminal window you will get your IP address without having your eyes dance all over the full output of ifconfig to find it.

If you forget what you set the alias to you can always type


This will show the current alias list. I also have

alias ls="ls -G"

This shows my directory information in color.