Hey, I want to be an Android programmer. Tell me what to do.

How many times have you seen a post on various forums about someone wanting to be an Android programmer? Is there a simple answer? Of course not, programming is not easy and there are piles of things you end up doing during any given day.

Wait a minute there! Isn't all I have to do is learn Java and some Android libraries? Oh, how we all wish but there is a lot more going on and that is just the start. Yes, you need to learn Java (or Kotlin) and you need to learn about the SDK as well but there is a ton more.

Today it hit me I had been in the following in just one day:

1. Java writing the main app code
2. Kotlin working on a utility program to convert iOS plist to Android format
3. Groovy because I was editing the gradle build scripts
4. Bash as I updated my Java version and needed to change my .profile 
5. Various Linux terminal commands
6. Various Git commands on command line and in Source Tree
7. Vector Graphics as I was working on a new image I needed
8. XML for the Android layout manager, string tables and color tables
9. PNG images for new icons
10. REST calls
11. JSON parsing for REST call responses
12. Sublime text editor creating and running macros against text files
13. File manipulation in Finder
14. Looking at ObjC in Xcode that I am converting to Android
15. Using IntelliJ, Android Studio and Xcode

It is not just learning Java and an API. I could not get Adobe Illustrator to do exactly what I wanted to I directly edited the SVG file. How many languages and file formats have I been in today already? Piles of them and I was able to quickly do the list above and I am sure I missed stuff.

Obviously Java, Kotlin and Groovy have a similar syntax so it is not like I am switching from assembler to JavaScript mentally but there are differences so you do have to perform a mental switch.

I don't mess with graphics every day but we have been setting up a number of build flavors this week so I have been doing a lot of graphics work along with "normal" programming. Honestly there is no such thing as normal. I shift around all the time. I decide to write the utilities in Kotlin as something new to learn, it was not required by my job. Glad I did, gives me a better appreciation of the language. Really want v1 to move out of beta before I use it for my Android apps.

Developers have to understand more than just the language they are using. You end up using a lot of tools like Git, Paint.NET, Adobe Illustrator, Adobe Photoshop, different IDEs, text editors and various utilities. Plus you get to keep the OS on your machine updated along with IDE, SDK, Language versions and tools.

Never know when the boss will need your help on a spreadsheet, document or power point as well. You have to be able to run at least a minimal feature set in a ton of tools just to keep up with life as a developer.

I am unsure how a new developer even gets started. I have amassed a large quantity of knowledge over the years. I can apply that to Kotlin when I experimented with it this week. I already knew the IDE, the .plist file format, XML, how to use collections, etc. I was picking up a new syntax but when you start from scratch you are tossed into a tank of sharks. The only way is to start small. Just work on one concept then move on. If you try to do an entire game or program with multiple screens you will go nuts before you get very far.

Using YAJL with Objective C

I have been tasked with getting an existing iOS code base ready to be released to the App Store. After loading the existing project into Xcode and resolving static library link issues I was able to run the program but not log in. I added some #ifdef DEBUG processing to allow me to log into a server without a valid certificate and got to play around in the app on the simulator.

I was quickly able to crash the app and after setting the debugger up to stop on all exceptions I found it was running out of memory. Boy, if you can run the simulator out of memory you know you can easily do that on a device.

Looking through the code I found it was using NSJSONSerialization for all JSON processing. That library is fine and it is provided by Apple so you know it is legal to use but when you have large amounts of data it is not super efficient. If you have used XML in the past and are just getting into JSON this is similar to using DOM vs SAX. DOM is great and easy and hands you a nice tree of data but if all you are doing is plucking things out of that tree to put in another format you are better off using SAX especially if there are memory concerns.

The code was parsing everything into a big NSDictionary based tree with 13,318 base records and tons more child records. This is just not going to work on a memory limited device.

I looked up information on stream based JSON parsers and landed on YAJL which you can find at https://github.com/gabriel/yajl-objc. I won't cover how to configure it for Xcode as all that information is on the base site. It was easy to set up for both Mac OSX (my test program) and iOS.

They show a streaming sample that was too basic to be very helpful. Hopefully I can fill in some of the gaps. The first issue with their sample is they read the WHOLE file into memory via NSData then run it through the parser. Not fully streaming at that point. Second they show no samples on what to do in the callbacks.

The code snippet below will help you parse your JSON by keeping track of where you are in the data. It will keep a stack of named dictionaries and arrays as you parse through the data. You can then create and act on the proper objects as you get map keys and values.

It also reads in the JSON file in chunks instead of tossing the whole things into NSData. In my final iOS code I am using a NSURLConnection and parsing the chunks in the didReceiveData call.

The original code ran out of memory when receiving 38.9 meg of data and took 5 seconds to run. I put in some cheater try / catch blocks to let it "finish" without crashing for timing results. The new code does not run out of memory and runs in 2 seconds. Of course this is all in the simulator on a fast MacBook Pro with a lot of RAM. I don't currently have a device to test on but will get one soon.

I looked around on the web for a long time before I was able to piece together enough knowledge to pull this off so hopefully it will help others get up to speed a lot faster.

Things that go in the header file

// Array used as a stack to keep track of where we are
NSMutableArray *stack; // create this in init method
NSString *mapKey;

// Actual class code

- (id) init
{
    self = [super init];
    if (self != nil) {
        stack = [[NSMutableArray alloc] init];
    }
    return self;
}

// Code to parse the file in chunks

- (void) parseFile:(NSString *)jsonFileName
{
    YAJLParser *parser = [[YAJLParser alloc] initWithParserOptions:YAJLParserOptionsAllowComments];
    parser.delegate = self;
    uint64 offset = 0;
    uint32 chunkSize = 10240;     //Read 10KB chunks.
    NSFileHandle *handle = [NSFileHandle fileHandleForReadingAtPath:jsonFileName];
    NSData *data = [handle readDataOfLength:chunkSize];
    
    NSLog(@"Starting parse of %@...", jsonFileName);
    [stack removeAllObjects];
    mapKey = nil;
    while ([data length] > 0)
    {
        [parser parse:data];
        if (parser.parserError)
        {
            NSLog(@"Error:\n%@", parser.parserError);
        }
        offset += [data length];
        
        [handle seekToFileOffset:offset];
        data = [handle readDataOfLength:chunkSize];
    }
    
    [handle closeFile];
    parser.delegate = nil;
}

// Stack adds newly named object OR last object on stack
- (void)parserDidStartDictionary:(YAJLParser *)parser
{
    NSString *dictName = mapKey;
    if (mapKey == nil)
    {
        dictName = (stack.count == 0) ? @"" : [stack lastObject];
    }
    [stack addObject:(dictName)];

    // Create new object here based on dictionary name
    // ready to hold soon to come key + value pairs
}

// End of dictionary - pop off stack
- (void)parserDidEndDictionary:(YAJLParser *)parser
{
    mapKey = nil;
    [stack removeLastObject];

    // Save object created in start Dictionary to
    // array or core data here
}

// New array staring, push name onto stack (use previous if we don't have one)
- (void)parserDidStartArray:(YAJLParser *)parser
{
    NSString *arrayName = mapKey;
    if (mapKey == nil)
    {
        arrayName = stack.count == 0 ? @"" : [stack lastObject];
    }
    [stack addObject:(arrayName)];
}

// End of array, pop off stack
- (void)parserDidEndArray:(YAJLParser *)parser
{
    mapKey = nil;
    [stack removeLastObject];
}

- (void)parser:(YAJLParser *)parser didMapKey:(NSString *)key
{
    mapKey = key;
    // Do processing here for each map key
    // Something like
    // if (!inResults && [key isEqualToString:@"results"])
    // {
    //    inResults = true;
    // }
}

- (void)parser:(YAJLParser *)parser didAdd:(id)value
{
   // Do processing here for each value (will have a mapKey)
   // Something like

   // if (inResults && [mapKey isEqualToString:@"query"])
   // {
   //   query = ((NSString *)value);
   // }
   //
   // value will be (NSString *), (NSNull *) or (NSNumber *)
   // booleans come as (NSNumber *) 0 = false, 1 = true

}

Is there really any such thing as a typical day?

I have been at my new job for a little over a month now. I have had the chance to learn a lot of new things. Generally I have been a front end developer doing the GUI and writing custom controls. Of course that does not mean I don't get to do any middle or back end coding. I use SQL where needed, write business logic handling objects, etc.

Here we are using Spring, Hibernate and Postgres for the database chores. I am finding my way about on all those annotations and have sent off and received requests for new table schemas to the DBA.

In a given day you can find yourself using little bits of each framework in a few hours time. I started the day setting up a chron job to call some existing code that calls two different 3rd party APIs to get some data. One of the vendors uses XML and the other JSon. The objects are created via Spring and we use Hibernate to interact with the database. I was using various internal SDK calls to do most of that with all of the work being done in Java.

Once that was up and running I was asked to look at some existing Objective C iOS code to see how close it was to being app store releasable. First off I was not able to login in as the servers I have access to don't have valid certificates on them and our API is HTTPS based. A browser will let you ignore the suspect certificate after a warning message. I added some #ifdef DEBUG to allow the Connection code to accept any certificate. That got me into the app and allowed to try out rest of the GUI.

Sadly I crashed the app pretty quickly. Did not know initially if it was bad memory management or if it ran out of memory. Turns out it ran out of memory. I added a @try block around things so it could push itself along a bit farther and I was able to run everything.

I then started to research why it ran out of memory. I looked at the server request it was making and tried it in a browser. A huge pile of data came back so I ran used curl to grab the data and write it to a local file. Turns out there was 38.9 meg of JSon coming back.

I wanted to sniff around in the JSon so I asked python to pretty print it and sent that to another file. I then opened it up in Sublime 2 to see what I had. Straight forward JSon, just a lot of it.

Next I wanted to see how the code was handling the JSon. Turns out the developer used the standard NSJSONSearialization calls in iOS. This is like using the DOM in XML in that you get it all in memory and they you double it up or more by converting to your internal object format. It was the easy way to go but not the most optimized way to handle things.

Time to hit Google and see if there is a streaming JSon parser. I found YAJL which I plan on experimenting around with today. I also think we should write the results out into a database instead of holding it all in memory. I have written similar code, it was XML and not JSon, storing to SQLite at a previous position so this should be fairly straight forward.

So was this a typical day? Yes, in that I ended up using a pile of different languages, data formats, command line tools and frameworks / SDKs to get my job done. You really need to branch out and learn new things.

I played around with Python in my spare time at home as I kept hearing about it and now I have used it multiple times. I used it at my previous job for a quick error log parser. I have used it here because some sample 3rd party code was written in it and I needed to convert that to Java. I was able to run the Python code and tweak it until it worked before I converted it to Java. I have used python to pretty print JSon files.

Curl was something new to me here. I have used it a lot with the 3rd party vendors as one did all their sample code it in. Curl makes it very easy to grab server results and shove them into a file. It works great with HTTPS as you can pass the user name and password in the command along with ignoring invalid certificates which seems to be a recurring need.

I have used Java for years so that is second nature. I have not used Spring or Hibernate in the past. I am really starting to see the power of the annotations. With great power comes great responsibility though. I am learning when you need to put code in other objects so it can be properly initialized. Anytime you use a new framework you find new and interesting ways to crash your code.

There is very little iOS knowledge here so I was able to step in and handle that too. A bit of a mind shift to get back onto Objective C and it will take some time to remember the syntax, framework and ins and outs of Xcode. Keeps you on your toes to shift around between languages, IDEs and frameworks. Good thing my Mac has a lot of memory, disk space and a large second monitor to keep all of this open and ready to roll.

Just to add to the mayhem I am working on updating my Android game at home. It does not scale well to tablets so I am converting the graphics to vector based allowing it hopefully work on nearly every device. At the end of most days I am pretty mentally fried.

Life with a Mac a few weeks in

At my new job my development machine is a Mac. Having been a heavy windows user for the last bunch of years but using a Mac on occasion to do some iOS development the change has been pretty easy.

Initial setup took some time as did learning initial command line options. Really everything else is pretty much just like on a PC. You run Eclipse, Sublime Editor, Chrome, Outlook, Finder, Terminal window, Evernote, a music player and various file viewers for images and PDF files. 

Once configured you just have to learn the proper shortcut keys to pull of frequent operations and off you go. This is a MacBook pro and I have a large second monitor. Everything is really fast.

I have been working on parsing XML and JSON files from security firms. Lots and lots of parsing of large files - will not HUGE but big as in 55+meg in size. I tried a number of JSON parsers and finally settled on Jackson as I needed to do streaming parsing and needed the ability to build simple JSON constructs on the fly as one provider uses JSON for request and response while the other two use standard HTTPS: requests with ?name=value params for requests and XML for responses.

Each API has its own set of oddities. Right now I have everything we need parsing and am using pagination to keep the response sizes to something reasonable. Next step will be hooking it all into the production code. Always good to iron out the bugs with simple test applications first.

At this point I really don't even think about being on a Mac. Sure the frame decorations are a bit different than in Windows but Eclipse looks and operates like Eclipse. Chrome is Chrome and sucked over all my settings, bookmarks and extensions. 

If I need a program I can generally find it but there are a lot less free options than on the Windows. Most things cost something or there is one or two free versions. Most programs have already made a number of decisions for you. Unzip something? Put it right where you unzipped it from. Don't ask where you want to put it. Download something? Yep, already gonna do it to the one directory I have chosen. Not a huge deal. I move things around expo facto but there is a lot more hand holding.

I have been kind of surprised at home much time I have spent in Terminal. Using Mac Ports to install software, dealing with CURL and Python etc. Some of this is for testing the API and some has been to get around Finder limitations. Of course I do the same when in Windows. Not that a normal schmuck would do this, as a developer I tend to need to do odd things.

There has been a lot of learning. I have never used CURL, xml_grep or JSON streaming parsing before. I had limited exposure to Python and Tree based JSON parsing. None of the sample code I have gotten for the APIs has been written in Java meaning I must learn CURL, Python etc. to know enough to convert to our target language. I love to learn so it has been a lot of fun for me. Gives you an appreciation for the other languages and utilities.

Working with vendors keeps you on your toes too. Good to have 3 going at same time otherwise you are waiting on one to respond while you have nothing to do which is never good. Keep me busy. Also allows you to spot differences and similarities in the APIs making some coding go faster. If this was split between 3 people that knowledge would not be shared as easily. I have shared many classes between the projects.