Split & Cat Large Files

We still live in a 32-bit world.  This mean if you deal with large file transfers, you will undoubtedly bump up against the 4 GB limitation (2^32) posed by legacy file systems and network infrastructure.  One way of skirting around the 4 GB limitation is to use the Unix commands split and cat to split up large files into smaller chunks.

For example, if you’re trying to save a 5 GB file (e.g. 5gb_largefile.iso) to a USB drive formatted for the FAT32 filesystem (max file size of 4 GB), you can split and save the file as 3 files of 2 GB or less with the following command:

split -b 2048m 5gb_largefile.iso 5gb_largefile.iso_

to yield the following files:

  • 5gb_largefile.iso_xaa (2 GB)
  • 5gb_largefile.iso_xab (2 GB)
  • 5gb_largefile.iso_xac (1 GB)

When you’re ready to utilize the file, you can combine the 3 files back into the original file with the following command:

cat 5gb_largefile.iso_xa* > /tmp/5gb_largefile.iso

In this case, you need to be sure to write the output to a directory (e.g. /tmp on a Unix machine) that does not have the 4 GB limit.

cURL Shell Scripting

A shell script that processes a batch of file URLs and determine the availability of each file:

#!/bin/sh 
# expects "|" delimited file containing 2 columns
# column 1: line number
# column 2: URL
#
# outputs "|" delimited file containing 3 columns
# column 1: line number
# column 2: file URL
# column 3: file available (Yes|No) 
printf "Enter the file name containing the file URLs (input)\n"
printf ":"
read URL_LIST

# make sure input file exist
if [ ! -f $URL_LIST ]; then
 echo "Invalid input file specified."
 exit 1
fi

printf "Enter the file name to capture results (output)\n"
printf ":"
read URL_STATUS

# make sure output file doesn't exist
if [ -f $URL_STATUS ]; then
 echo "Invalid output file specified."
 exit 1
fi

printf "line number|file url|file available\n" >> $URL_STATUS

# loop through input file, row by row
for i in `cat $URL_LIST`; do

# parse for 1st & 2nd column, delimited by "|"
line_no=`echo "$i" | awk -F'|' '{print $1}'`
url=`echo "$i" | awk -F'|' '{print $2}'`

# echo progress to screen
printf "Processing line "
printf $line_no
printf "\n"

# check HTTP header if remote file exists; returns "HTTP/1.1 200 OK"
output=`curl --max-time 6 --silent --head $url | grep 'HTTP/1.1 200 OK' | wc -l`

# output line number & URL to file
printf $line_no >> $URL_STATUS
printf "|" >> $URL_STATUS

printf $url >> $URL_STATUS
printf "|" >> $URL_STATUS

# output line number & URL status to file
if [[ $output -eq 1 ]]; then
 printf "Yes\n" >> $URL_STATUS
else
 printf "No\n" >> $URL_STATUS
fi

done

Certified Scrum Master

I have successfully completed the Scrum Master Accredited Certification Program from the International SCRUM Institute:

Matching Pair 1.1

Matching Pair is an iOS game for all ages.  The objective is to find all matching pairs from a set of playing cards that has been randomly shuffled.  A matching pair consist of two cards that have the same color and rank.  For example, the King of Heart and the King of Diamond are a matching pair, since they are both red cards and they are both Kings.

Version 1.1 will soon be available for download on Apple’s iTunes App Store.  Download version 1.0 now to get version 1.1 as a free upgrade.

Matching Pair 1.1 includes critical bug fixes, usability enhancements and social network integration. Specifically this release:

  • Fixed bugs that periodically caused the application to crash when the user is logged in to Game Center.
  • Integrated with Facebook to allow the user to share game results with friends on Facebook.
  • Added Settings screen to allow the user to change game settings, reset Game Center Achievements and login/logout from Facebook.
  • Improved playability by automatically closing one of the face-up cards (new setting), when the user selects another card with 2 cards already opened.
  • Introduced different levels of gameplay by adding a new ‘Match Color Pair’ setting that allows the user to specify whether a matching pair must have the same color.
  • Added 2 additional Leaderboard categories for games played with the ‘Match Color Pair’ setting turned OFF.

For more information, check out the Matching Pair Product Page.

This slideshow requires JavaScript.

Sean’s 10th Birthday

We celebrated Sean’s 10th Birthday at Petroglyph, a local paint-your-own pottery studio.  It was a fun afternoon for everyone, especially for Sean.  It was his first time painting ceramics.  He decided to paint an ice cream cone, his favorite dessert.  And as a gift, the staff at Petroglyph painted him a custom Angry Bird dinner plate.  Needless to say, he was thrilled when he saw the final products.  Judging by his smiles, it was a memorable and happy birthday!

This slideshow requires JavaScript.

Download Pro Calculator 1.2

The Download Pro Calculator is an iPhone application designed for software professionals and power users looking to estimate the transfer time and the storage media required for large file downloads or transfers.

Version 1.2 is now available for download on Apple’s iTunes App Store.

This release includes the following enhancements and bug fixes:
  • Fixed the vertical scrolling bug for the Connection Speed and Storage Media list.
  • Fixed screen layout bug (on iPad running  iOS 5) when the iAd banner is hidden.
  • Extended maximum value to 1024 for new Connection Speed and Storage Media.
  • Updated the default Connection Speed and Storage Media list for new installations.
  • Added cross promotion screen on startup to promote other applications.
  • Conducted full regression testing on iOS 5 devices.

For more information, check out the Download Pro Calculator product page.

Protected: Happy New Year

This post is password protected. To view it please enter your password below:

RIP Steve

 

Rest in peace Steve Jobs.  I really can’t imagine a world without you!  Thank you for all the innovations!  You have changed the world for the better.

Sept 30, 2011

Today is my big sister Cheryl’s birthday.  She would be 46 years old if it weren’t for Cancer.  Last year, we went to the Afghani House (her favorite restaurant) for her birthday dinner.  She insisted on treating the meal, as if she knew.  Ironically, it’s a gift we won’t be able to reciprocate this year.  One week later, she learn of the fateful news that would change everything.

For a family that at times was too busy to make time for each other, birthday celebrations were that much more special.  It reminded us that we were a family, and it renewed the family bond  that at times was very fragile.  This year, we won’t be celebrating Cheryl’s birthday.  There’s an eerie feeling that permeates through out the family, but no one wants to talk about it.

Perhaps it’s superstition or simply inappropriate to celebrate the birthday of a love one who has passed away.  In any case, she is not forgotten and I just want let her know that we are still very much missing her.

Happy Birthday Cheryl!

Half Dome Postmortem

Eight years ago, a group of friends and I went hiking up to Half Dome (16 miles round trip).  It was a scorching hot Summer day (~90F degrees) for Yosemite.  We didn’t leave camp until around 7 AM.  By the time we reached the base of Half Dome, it was around 1:30 PM and we had run of out water.  Still, some of us managed to climb up to the top.  On the hike down, I borrowed a small bottle of water from another hiker.  Five of us shared it, taking small sips mindful that it was our only water supply.  By the time we returned to the Yosemite valley (6 hours later), we were dehydrated, hungry and exhausted.

Over the Labor Day weekend, some of us returned to Yosemite to go camping and to redeem ourselves on the slopes of Half Dome.  It was a similarly hot day (~85F degrees).  However, we were more mentally prepared for the challenges ahead.  Learning from our prior experience, we left camp by 5 AM.  We reached the base of Half Dome before Noon, and we started the hike down around 2 PM.  We brought ample water.  And we managed to avoid the congestion on the Half Dome cables and the extra strain of hiking/climbing up, under the blistering afternoon sun.  Overall, it was a very successful trip!

Eight years is a long time to recall everything, which is why I’m writing this blog.  This is a postmortem on what went well and how we might improve, should we decide to try again in 8 years.  If you are planning to hike up to Half Dome, I hope you can benefit from our lessons learned and the checklist below:

  • Half Dome PermitReserve in advance and bring a copy of the Half Dome permit for each person planning to climb up to the top of Half Dome.  When we were there, a ranger was at the base of Half Dome checking for permits.  Only about 400 permits are issued each day, although it’s possible to find people with extra permits along the trail or on the following website.  If you’re really desperate, you can try to beat the ranger to the top, since the ranger doesn’t live on Half Dome :-)
  • Leave early – Start the hike early enough to beat the heat during the Summer and the crowd on the Half Dome cables, but not so early that you’re hiking in the dark on the treacherous part of the trail such as the Mist Trail (very slippery when the waterfall is pouring) right before Vernal Falls.  Bring a small flash light for the hike in the dark.  We left camp (Upper Pine campsite near the trail head) around 5 AM and reached the Mist Trail at the onset of the sunrise.  It was perfect timing.
  • Bring protection – Mosquito repellents and sun screen are essential when hiking in the Summer.  Always checkout the weather forecast before the hike and dress accordingly.  Dress in layers and wear a hat and shirt that offers UV protection.  I personally prefer the convertible pants that can double as shorts in hot weather.  On warm days, also wear shirts and socks that wick away the sweat.
  •  Stay hydrated - The NPS website recommends 1 Gallon of water per person if you are hiking up to Half Dome.  That seems fairly accurate.  My friend Eric actually packed a 1 Gallon water bottle in his backpack and drank all of it.  However, unless you have strong legs like Eric or you’re a descendant of the mule, I recommend that you pack less water and bring a water filter.  Be sure to pick up water at the river near the Little Yosemite campsite, since there are few water sources past that point.
  • Eat along the way – For maximum efficiency, you want to eat before you get too hungry.  Energy bars and gels, high in protein and vitamins, are compact and perfect for the hike.  Our friend who works at Clif Bar provided us with some samples that were quite tasty.  At the end of the trip, we tallied up and found that we consumed on the average 1,000+ calories per person.  Your needs will vary, but it doesn’t hurt to bring a little bit more and share.
  • Don’t forget the camera – Yosemite is one of the most beautiful place on Earth.  The John Muir trail and the hike up to Half Dome offers plenty of opportunities to take great pictures.  I brought my iPhone and it took some great pictures (see links to pictures below).  Unless you’re a serious photographer, leave the SLR at the camp.  There’s no need to be weighted down.
  • Wear a climbing harness – About 1/4 of the climbers on the slopes of Half Dome wore a climbing harness when we were there.  I really think it should be a mandate for everyone.  The harness allows you to clip yourself to the cables and prevents serious injuries or even death, should you loose your grip and fall off Half Dome.  We didn’t have the climbing harness, but noted it down as a “must have” for next time.
  • Muscles, shoes & gloves -  To reach the top, you need to pull yourself up the slopes of Half Dome (400 feet ascend at up to 45 degrees) with the help of two cables and wood planks that are laid approximately 10 to 15 feet apart.  You need to be physically fit, with strong legs and lots of upper body strength.  As the cables and the granite rock can be quite slippery, you also need to wear shoes and gloves that have great traction.  Our friends who wore tennis shoes this past weekend had no traction on the granite and had to abandon the climb :-(
  • Be Safe & repectfulRecent news of people falling off Half Dome and the waterfalls along the John Muir trail should serve as a reminder that Yosemite is not an amusement park.  There’s danger lurking everywhere.  If you ignore the warning signs or choose to act stupid, nature will take a bite out of you!  Despite the warnings and recent events, we still saw many people wadding in the water just about 15 – 20 feet from the edge of Vernal Falls.  Pity.

Below are some pictures that I took at Yosemite and along the hike up to Half Dome.  BTW, WordPress sucks for not supporting the Google Picasa slideshow widget, so you will need to click on the picture to see my web album.  Enjoy!

Yosemite, September 2011
Follow

Get every new post delivered to your Inbox.