Trusting the API is not a good idea

You are programming right along and need to accomplish what appears to be a simple task. You look at the API used by the company and find a one line call to do what you need. Make the call, get the expected results and move on to the next task. All rainbows and unicorns right?

Not so fast there bucko! The key word here is fast. The method I called did do what it needed to do and returned nearly instantly because I hardly had any data in my test database. Once you had a large amount of data it was painfully slow as we found out when attempting to run it at a client site. I had no logging output around the call as it was never an issue for me. It just looked like the program froze when running at the client location.

When something breaks I blame myself. I figure it must be in my code. Took another developer looking over my shoulder to point out "That call might be slow". I had looked past that line over and over again. Of course when I ran the program it sailed right past that line so I never thought about it. I was blinded to it because I was sure I was the one to blame.

We were able to have the client run another command against the database to clear out some old data to get them past that point. We then found the next step, again relying on the API, was also very slow but I had logging in place to watch that progress.

Time to rethink how all of this needed to work. We fixed the initial call to work in a reasonable amount of time and are in the process of gutting the code to write records to a file which we will then import via "copy {tablename} from {outputfile}" x 4 instead of doing inserts in the code. We were able to import 1.2 million records in 1 minute and 46 seconds using the 3 copy calls where it was going to take days to do it in code one insert at a time while parsing JSON streams.

This particular client has a much larger dataset than any other client. We are talking to a 3rd party vendor to get the 8.2 million records that need to be inserted via JSON requests. After these code changes things will run faster and scale better for every client. My part of the code is the JSON to our database format conversion. So far that appears to be working properly at least.

I made the mistake of trusting the existing API calls. I have only been at the company for 4 months meaning I had no background in company lore. I had no ideas what areas had bitten folks in the behind in the past and I figured the every call was optimized. Trust no code especially "this one line can't do much or take long". I should have put logging messages above and below the API call so I could have spotted this in the client run quicker. We knew up front they had a ton of data. We knew we did not have a large sample for testing. This is where extra logging is really helpful.

Lesson learned. I hope I don't make it again.

When you know it can be done is coding is easier?

There are times when you are coding that you are not sure what you writing can be done. Oh, sure most anything can be done if you sit down and look at it logically and have enough hardware and time. What I mean is you have not seen it done before so it feels harder. It is easier to give up and work on something else.

But what if you have seen it done before? A really cool transition effect on a mobile device, a pop-up menu on a double tap of a row, fragments that work differently between a phone and a tablet in a unique manner, etc. You know it can be done, you have seen it running on similar hardware to what you have, now duplicating it is a challenge.

Same thing goes when you have written similar code in the past at maybe another job. You don't have access to that source code but you know you basically wrote the same thing before. Again you have a programming challenge instead of a "how the heck can this even be done" task.

I ran into this again today as I was converting code from straight activities to ActionBarSherlock fragments. Everything I was doing I had solved in some manner in the past. I totally knew it could be done so even when it was not working I knew it was because I had done something stupid. My frustration level was pretty low. I did get stuck on one issue but a lunch break and trying to word a question mentally for a web search triggered a new thought showing where I had messed up.

If you are writing a new piece of functionality and it is not working you are unsure if you totally screwed up or if there is a limitation of the SDK or language you are butting up against. Of course you don't want to blame yourself so you start looking for things that are wrong with the API especially if you are new to this framework. You tend to get frustrated quicker and the frustration will be deeper. "This just can't be done" you tell yourself. "There must be a bug in the SDK" rolls through your mind.

Grabbing at straws comes into play. Move it to another thread, yeah that's it. Change the order you call things, that'll fix it! Try a 3rd party API or another widget, the one I first picked must be a steaming pile even though it was the highest rated one on Stack Overflow. This stupid thing has got to work, ain't nobody got time for that.

Finally you solve the problem. Turns out you were at fault the whole time. You were on the right track but you thought the range was 1.0f to 0.0f when it was 0 to 100. You found an obscure post on the web and you spotted the author passing 75. The API is fine, in fact once you stopped cussing at it there are some really cool features it has that you hadn't gotten a chance to use yet.

Programming is a really weird occupation. You spend a good part of the day dealing with the unknown, trying to pull something off you think can happen but maybe can not. Some days you are writing code nearly from memory just humming right along. You have done something similar and this is nearly a no brainer. Just create some objects and parse some JSON, easy peasy.

Other days you are trying something totally new. You have to integrate a third party pie chart widget. You barely know where to start. You spend some time pouring over documentation so hopefully it works the first time. You find some example code and pull it off the web. Do you start a test app? Do you try to drop it into the existing app? Is all the example code in fragments and you are using activities? Can't I just find an exact example of what I am doing? Of course not, the chance of someone else doing the exact same thing you are doing is really low. Heck if everyone needed it then it probably resides in the base language, one of its core libraries or somewhere in the SDK.

The best we can do it post things we figured out to our blog or answering questions on blogs or Stack Overflow. Let others learn from our experiences. Coding should be about solving specific issues for a company not constantly reinventing everything. Grab as much code as you can from other projects you have done, other websites and the SDK itself.

Why I stopped using Android Studio - for now

I was pretty happy using Android Studio. Loved the way I could see the preview of a layout while working on the layout. I had written a new app from scratch to match the functionality of our current iPhone app using Android Studio. I also like the dark theme they provide. Easy on the eyes.

Then I ran into a problem that made me stop using it. I wanted the app to run on more devices so I turned towards ActionBarSherlock. I found various posts on Stack Overflow on how to make it work. I tried them all but was ending up with some weird gradle errors that I was unable to solve.

Gradle: A problem occurred configuring root project '{my project}'.
> Failed to notify project evaluation listener.
   > Main Manifest missing from /Users/kpeck/AndroidStudioProjects/{my project}/AndroidManifest.xml

I have a manifest file of course. It is not in the directory given. I even tried to copy my the file to that directory, even though it does not belong there, and attempted the build again but it still did not work.

It is possible that starting a project from scratch with ABS in it and then copying in my files may work. I will have to give that a shot at some point.

Having run ActionbarSherlock under Eclipse in the past I knew pretty much what to do and had it up and running in a matter of minutes. Import the project into the workspace, delete the duplicate android-support-v4.lib, set the dependency between ABS and my project and it was working. I spent the rest of the day converting code to ABS format. That was generally pretty easy with a few minor snags around loaders.

At some point I may go back and attempt to get it working in Android Studio but right now I need to move forward on the project. I have other areas that I want to convert to activities hosting fragments. Currently I only converted the one that needed a loader but I have some ideas on how to layout items on a tablet that would benefit from fragments.

Android Studio keeps changing the way you deal with project settings. It was through a dialog but now they want you to manually edit things in the build files. Somethings happen automatically as you import but it does not seem to finish the job. It is still alpha software and they are pushing out changes quite often. I want to get familiar with gradle but I don't have the time to pause right now and do that.

I also found the preview has some issues especially with Relative Layouts. The above / below / left / right layout parameters seem to be suggestions in the preview. They work fine in the emulator or on a device but that gets frustrating. I may see the screen look just right in the preview then a change to the XML in an area unassociated to the Relative Layout will cause it to paint incorrectly, usually overlapping controls. Really need a preview I can trust so I am not chasing down issues that don't exist.

Probably good to wait another release or two of Android Studio. There will be more tutorials on how to make things like ABS work with it and I can skip the headaches. Back to trusty old Eclipse for now. I found a dark theme that is reasonable for it but it only changes the editor pane. That is the largest part of the screen so it is better than nothing.

Cancel SVProgressHUD process by tapping on HUD

I am using NSURLConnection to potentially pull a lot of data from the server based on user search criteria. There are times when the user does a pretty board search and getting the results back can take a couple of minutes. It can be a waste of the user's time and server processing especially when the user realizes how much data they are about to get back. I wanted to add a [Cancel] option. Since I already had this working on the Android version I really needed to make it happen on iOS. I looked on the SVProgressHUD site and there was a rather heated discussion about this and how if I was using the HUD to run super long processes then I was doing it all wrong. 

Obviously I beg to differ on that point. I have a potential long running process based on a user request and I show a progress indicator and allow the user to cancel as needed. The code below shows a simple sample but I actually show "Processing record X of Y" in my code so the user gets a feel for how long the process is going to take. I don't know how many records are coming back in the response until I make the request so I can't give a "Are you sure you want to suck down 10meg of data?" message first.

After reading the SVProgressHUD site I investigated some other HUDs to see if they solved my needs but found them to be harder to use or they had other issues once I hooked them up in code. I sniffed around in the SVProgressHUD source code and discovered a solution - Notification Center messages.

Just before I show the HUD I do the following:

[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(hudTapped:) name:SVProgressHUDDidReceiveTouchEventNotification object:nil];

A sample of showing the HUD which lets the user know how to cancel the operation:

[SVProgressHUD showWithStatus:@"Requesting data...\n(tap to cancel)" maskType:SVProgressHUDMaskTypeBlack];

The method that gets called when HUD is tapped:

- (void)hudTapped:(NSNotification *)notification

{

    NSLog(@"They tapped the HUD");

    // Cancel logic goes here

}

and any time I am done with the HUD, either via tap to cancel or run to completion, I call:

[SVProgressHUD dismiss];

[[NSNotificationCenter defaultCenter] removeObserver:self name:SVProgressHUDDidReceiveTouchEventNotification object:nil];

So far it is working like a champ and appears to be a pretty clean solution. I hope you find uses for my discovery as well. Maybe you want to put in an Easter Egg if they tap it 10 times. Have fun.

Use adb to update APK on device even with emulator running

For my Android development I use the fast HAX based emulator on both my PC and Mac. I do the bulk of my testing on the emulator and push the APK to a real device maybe once a day to to make sure everything still looks good and runs as expected.

If you try adb install {yourapp}.apk it gets mad if it finds multiple devices such as a physical device and the emulator. Plus install gets annoyed if the program is already installed on the device.

You should use the following command

adb -d install -r {yourapp}.apk

The -d option tells it to find a device attached via USB. The -r tells it to upgrade which keeps all the existing app data in place which is handy if you are using SQLite databases or have settings saved such as username and login server. This command can save a lot of time and headaches as you don't have to shut down the emulator to push to a device and you don't have to lose any saved data.

If you have a lock screen enabled you will have to unlock the phone before you can push the update to it.

You can also use adb -devices to make sure it is finding the USB connected device. There are times you have to unplug and replug the USB cable to get things working.

Back as solo mobile developer - not easy

Being the solo mobile developer at a job is not easy. I had just gotten used to having a couple of developers around me at my previous contract position. Everyone had bits and pieces of experience in different areas so you would quickly get answers to questions without doing a Google search. They might have some sample code laying about from another project or just know how to do it.

Sure the big boss here wrote the initial iOS code that I have been cleaning up and adding features against but I rarely get to speak with him and he has no Android background. I have been working on the Android port this week getting stuck on silly things that the guy I worked with at the last place would have handled with ease as he had a lot of background in those areas. Now that the core networking is in place I can start moving forward at a pretty good clip.

A real feeling of isolation when you have developers all around you but they either don't know the language (Objective C) or the framework (Android SDK) thus you can't really bounce any ideas off of them. Sure I can hit them up for general business knowledge or Java questions but not for the meat of the development cycle.

For my Android work I have been using Android Studio. I know it is a 0.1 release but that is really just the Android plug-in side of things. The over all IDE is at version 12 and I have enjoyed using IntelliJ in the past. I like the dracula theme as far as colors go.

The ability to edit layout XML and see the results side by side in the preview window is super handy. I have gotten that to generate crash reports from time to time but they did not kill the IDE or lose any of my work.

What I miss from Eclipse is the new class wizard. It asks for the base class and interfaces and lets you quickly build things out. I have asked about this on the IntelliJ forms and they basically stated you can do all of that via a number of steps. I want a Wizard. I also want the auto add of the Activity to the manifest. I should not have to do all that work manually every time I add a new activity to the project. Since this is a new project that has been happening a lot.

The Log Cat filter seems pretty iffy too. I turn it on to filter out things but it forgets about that between program runs so I have to go back to "No Filters" then back to my filter again. It is silly how much junk the emulator kicks out into Log Cat. Since I am using the HAX high speed emulator I get a ton of extra stuff.

I was happy to get the upgrade to 0.11 this morning. Good to know there is active work on Android Studio and I am sure it will continue to improve. I have been using it at home on my Win7 64 bit PC and my Macbook Pro at work. Both seems to run equally well.

Core Data stinks for bulk inserts - direct SQLLite is much faster

I was handed an iOS app that was using the basic JSON handlers and keeping everything in memory. This was slow and running out of memory for large requests.

I converted the project to use stream based JSON parsing stored to a Core Data database backed by SQLLite. This solved the memory issues allowing all requests to work. There are 5 tables involved with a total of 239,000 records if you requested all the data from the server.

The problem - that request took 5 minutes and 43 seconds to process on an iPad 2. It took 33 seconds on the simulator. Let's state this again - Always test on a device. That kind of time is totally unacceptable.

After sniffing around the net and attempting to speed up Core Data I determined that Core Data sucks at doing bulk inserts. Core Data is really nice at hiding the gut level fun of SQLLite but for large inserts it is not the way to go.

I converted everything to use SQLLite directly and saw a massive speed increase. I can now process the same record set in 1 minute and 23 seconds. That cuts a whopping 4 minutes and 20 seconds off the time. The simulator went form 33 seconds to 12 seconds.

A typical user would rarely if ever request all the records but testing worst case scenario is really the best way to go. The speed improvement is seen across the board in the app with similar percentages of improvement. I am very happy with the results even though it took a couple of days to get it all sorted out.

As for the code there are areas that are more involved than the Core Data version and there are areas that are actually a lot cleaner. Everything is a trade off and there is no clear winner.

I had never used Core Data before. It was a really good learning experience to set it all up using the GUI editor in Xcode and finding out what it took to add the Core Data library and support code to an iOS project that was created without it originally. I found a lot of Core Data really straight forward and easy to use. I will not shy away from it on other projects unless they involve bulk inserts.

Having used SQLLite in the past I had a basic object set to start with and I knew my way around FMDatabase. My previous experience did bulk insert processing and I wrote both iOS and Android versions so I stuck with SQLLite to make it easier to convert code back and forth. I wanted to learn something new this time around so I went with Core Data. Good thing I could quickly fall back on my direct SQLLite experience to solve the speed issues that arose.